Fork me on GitHub
#clojure
<
2019-08-01
>
CyberSapiens9704:08:45

anyone here have knowledge in image processing ?

mp04:08:24

Hello, i’m new to clojure and I’m having some trouble. I need to test s3 responses and what data is in the bucket. What’s the recommended way to do this? I’ve come across amazonica but i’m having trouble. Do i need to create fixtures? I’m also not sure if i need to pass in credentials.edn which has AWS secret key and access key or if i make up fake keys. Any help would be greately appreciated

jumar05:08:04

There's also aws-api from Cognitect (https://github.com/cognitect-labs/aws-api) which can be a newer interesting alternative to amazonica

seancorfield05:08:25

@mirkopalancaji Hi and welcome to Clojurians! If you're brand new to Clojure and have questions about the basics, the #beginners channel has a lot of people who've opted in to helping out with new folks. If you've got any questions about navigating Clojurians, feel free to ask in #slack-help There's an #aws channel that may help get you answers to AWS-specific stuff. I looked for an Amazonica channel but I don't see anything specific for that.

seancorfield05:08:52

If Amazonica has a "setup" function, test fixtures are probably the way to go to set up the environment or context for your tests (I'm not familiar with that library).

seancorfield05:08:28

You might also want to look at the Cognitect AWS library since that provides a nice, thin wrapper over pretty much the entire AWS API.

👍 4
seancorfield05:08:46

Ah, I see @jumar suggested that. Cool. Thank you.

mp05:08:06

Thanks for the information @seancorfield @jumar. I’ll definitely take a look and research further 🙂

😎 4
seancorfield05:08:49

There's also a #testing channel if you need to deep dive on testing approaches in general.

clj 4
👎 4
Toby Clemson14:08:53

Hi all, I'm building a dev tool using Clojure and would like to write configuration files for the tool in Clojure (similar to leiningen or riemann I suppose). I want to read those configuration files, which may include multiple forms, in a safe way. The files will have been written by the user of the tool so should be safe in terms of their content, however I'm wondering if I should be doing any validation of the contents before evaluating them. What's the best approach to do this?

Toby Clemson14:08:09

Additionally, how would I go about making certain bindings available for the evaluation of those configuration files? I had a look at how leiningen does it but need to dig deeper to understand it fully.

xi14:08:47

Try something like:

(binding [*ns* *ns*]
  (in-ns sym)
  (require ...)
  (intern *ns* 'foo (fn [] ...))

xi14:08:26

Oh and I use load-string to actually eval Clojure

xi14:08:40

You could also try clojure.tools.reader. If you want an AST, there's clojure.tools.analyzer.

Toby Clemson15:08:44

@ULML9PYJH Thanks. I'm a bit confused by that. What's the purpose of the binding call?

Toby Clemson15:08:57

Hmm, is it so that clojure.core functions continue to resolve after the call to in-ns?

xi16:08:33

https://clojuredocs.org/clojure.core/*ns* > ns is a root var and in-ns calls set!, which only works after someone somewhere calls the binding macro

xi16:08:31

I did have to require-clojure inside the new ns to be able to use any clojure.core stuff, not entirely sure why

Toby Clemson17:08:49

Hmm, from the REPL I can call in-ns outside of a binding. It finds or creates that ns and switches to it.

Toby Clemson17:08:36

However, in the created namespace, there are no aliases of clojure.core vars

quadron15:08:27

does it make sense to treat a collection as a virtual reduction and invoke xform on it directly and get a virtualized collection without recourse to transduce? (xform (virtualize collection)) returns virtualized-collection

hiredman16:08:23

xform in the context of transducers refers to a function that transforms a reducing function, it takes a function of the form (fn [accum item] accum) and returns a new function of the same form, so applying that xform to a value and not a function is GIGO

quadron16:08:15

(virtualize collection) is supposed to return a function that knows about the values of the collection

quadron16:08:41

forget about the previous question. how do I turn ((map str) (eduction (range 10))) into a collection?

ghadi16:08:58

that code not valid -- it's applying a transducer to an eduction

hiredman16:08:19

(map str) returns a function that takes as its argument a function

hiredman16:08:29

an eduction is not a function

hiredman16:08:38

(eduction (map str) (range 10)) is valid for example, because internally it has a reducing function, something like (fn [accum item] ...) which internally it applies (map str) to

hiredman16:08:18

similarly for other things that take transducers, like transduce

hiredman16:08:07

(transduce (map inc) + 0 (range 10)) is more or less (reduce ((map inc) +) 0 (range 10))

hiredman16:08:35

xforms are not transforms of values, they are transforms of reducing functions

hiredman16:08:44

given an xform and a collection, the easiest way to produce a new collection is using into, you just supplied an empty new collection and the input collection is poured in via (xform conj) and you don't have to think too much about it

quadron16:08:20

can you give an example where the zero-arity of a transducer is used?

hiredman17:08:20

I cannot, but if anyone can I think it would be @cgrand (someone, back when transducers were introduced wrote an email trying to figure out why the 0 arity existed and what it was used for, and I don't recall the outcome of that, and it might have been cgrand)

hiredman17:08:54

if recall in most cases the 0 arity of the transducer is basically ignored and the 0 arity of the original reducing function is used

quadron17:08:34

why doesn't transduce transform the zero-arity of rf then?

hiredman17:08:01

it does, but for example, transduce calls (rf) before applying xform to rf

cgrand17:08:17

It’s rather; you can’t rely on the 0 arity to be called, and you can’t do more in it that call the downstream (rf)

seancorfield17:08:54

Is it ever called? That’s been a long-time puzzle from my p.o.v.

cgrand17:08:48

You sent me down a rabbit hole, iirc there was one case in core but so far no luck

seancorfield17:08:03

😆 Thank you for looking! It's definitely one of those things where you know it's the right thing to do (define a zero arity version) but you're never sure why you're doing it 🙂

cgrand17:08:49

I’m even looking at reducers now to see if it’s something we inherited

cgrand17:08:04

I definitely rely on it in xforms

quadron17:08:21

so it's only ever called if downstream calls it?

hiredman17:08:24

it is up to the "reduce" implementation

hiredman17:08:29

so it isn't something you can rely on, which is why the only sensible thing for it to do is forwad the call on unchanged

seancorfield17:08:55

I haven't yet run into a single case where I've seen the zero arity version called but that sort of makes sense.

hiredman17:08:20

transduce never calls it, most other "reduces" don't give you the option to leave out the initial value

hiredman17:08:54

for example into's initial value is the empty collection you must pass in

hiredman17:08:23

for core.async the initial value is the channel's buffer which items are added to

colinkahn17:08:48

Is there a concise way to get the file from a namespace symbol? I have an idea of how to do it via clojure.tools.namespace but it seems kind of involved.

hiredman17:08:55

it is involved

colinkahn17:08:20

Haha, alright, suspicions confirmed

hiredman17:08:38

like, there is no guarantee that a file exists, or that a clojure source file exists

hiredman17:08:10

the file the namespace comes from could be aot generated class files on disk

seancorfield17:08:39

(a single file can contain multiple namespaces too)

hiredman17:08:01

or multiple files can contribute definitions to a single namespace

colinkahn17:08:18

Ah, yeah I’m not so worried about those cases, i’m just looking at the tools available in tools.namespace

andy.fingerhut17:08:46

Clojure's own require uses a private function root-resource to turn a namespace into a Java resource path. You can call private functions from outside of the namespace using syntax like this: (#'clojure.core/root-resource 'my-ns.core) (which returns the string "/my_ns/core"

hiredman17:08:25

all of which are unusual things, but possible

Graham Seyffert17:08:52

Does anyone know if there is a difference, under the hood, between using ->ARecord vs. ARecord. for instance creation? We have some old ARecord. usages lying around and I’ve been seeing some rare errors running on JDK 11 that I suspect are related to records…

Graham Seyffert17:08:21

Naively I assumed that the difference was largely syntactical

hiredman17:08:55

ARecord. is a concrete reference to a constructor for a particular class

hiredman17:08:16

->ARecord is a function call through a re-def-able var

hiredman17:08:02

they behave differently for the purposes of code reloading and aot compilation

Graham Seyffert17:08:02

Mmm that is good to know

hiredman17:08:55

also if you use ARecord. you are extremely likely to forget to require the clojure namespace instead of just importing the java type

Graham Seyffert17:08:00

So the ARecord. syntax isn’t only deprecated, it’s potentially more dangerous?

Alex Miller (Clojure team)18:08:39

using ARecord. is not deprecated. This is normal Java constructor interop and is perfectly fine for arbitrary Java objects (and even "ok" for records). But it is strongly preferred to use the Clojure native means of construction (which also means you don't have to import the internal impl class).

Graham Seyffert18:08:42

> This is normal Java constructor interop and is perfectly fine for arbitrary Java objects Makes sense! I was referring to “deprecated” in the context of records, since there are no ->Class constructors for plain Java classes.

Graham Seyffert18:08:26

In the context of deftypes and defrecords, what implications might using the ARecord. syntax have wrt reflection / classloading?

Alex Miller (Clojure team)18:08:39

or at least not anything obvious to me. if you're using records, then you're in "Clojure world" and reflection is not something you'll encounter - you'll always be driving through Clojure abstractions like IPersistentMap, ILookup, etc or through protocols or interfaces for inline impls. I would recommend never making Java interop method calls on records, but if you did I guess you might have the type for a constructor but not if using the constructor method. Since you should never do that, I wouldn't expect it to be an issue.

Alex Miller (Clojure team)18:08:03

for class loading, you're loading the class either way

Graham Seyffert18:08:20

> I would recommend never making Java interop method calls on records in this case referring to e.g. (.afield record) vs. (:afield record)?

Graham Seyffert18:08:31

consider the case where your type/record implements a java interface, for the purposes of calling that method from Java

Graham Seyffert18:08:02

does that change things in any way?

Alex Miller (Clojure team)18:08:49

all record fields are private, so (.afield record) won't work anyways

Alex Miller (Clojure team)18:08:41

actually, maybe they are public, but you should strongly prefer using normal map keyword lookup (this is special-cased and optimized for records), and it's portable to cljs

Graham Seyffert18:08:54

I ask because I’m seeing a (very infrequent) compiler error during our application warmup wherein we’re loading a Clojure namespace from Java, calling a function that will return a type, and then invoking methods on that type. In the course of requiring these namespaces (through the Clojure api), we get an error during record compilation that it can’t find a particular symbol

Alex Miller (Clojure team)18:08:40

for java interfaces, you'll want to typehint to the interface, but that's likely true regardless of how you constructed it

Graham Seyffert18:08:51

The two occurences of this happen while evaluating an (ARecord. field1 field2) call

Graham Seyffert18:08:56

And only happens on JDK11

Alex Miller (Clojure team)18:08:00

well if you're calling all this stuff from Java, that's a whole different set of concerns

Graham Seyffert18:08:23

Caused by: java.lang.RuntimeException: No such var: a-namespace/a-function

Graham Seyffert18:08:43

We’ve used this pattern for a while and I’ve never seen this happen on JDK 8

Graham Seyffert18:08:51

But does happen rarely on JDK 11

Alex Miller (Clojure team)18:08:18

I feel like we've kind of backed way into this problem description, which probably has nothing to do with records

Graham Seyffert18:08:56

it may not. It’s a theory because I can’t find a way to force this to reproduce

Alex Miller (Clojure team)18:08:59

based on that description, I'd say it's most likely a timing issue and 11 happens to load faster than 8, exposing the issue (which is probably there even in 8)

Alex Miller (Clojure team)18:08:22

ns loads are not atomic and concurrent loads from different threads can cause issues like this

Alex Miller (Clojure team)18:08:35

where two threads are racing in loading the same ns

Alex Miller (Clojure team)18:08:08

any chance you've got multiple threads loading code at the same time?

Graham Seyffert18:08:18

I was suspecting concurrent loads as well initially too.

Graham Seyffert18:08:44

We shouldn’t; the namespaces are required serially

Graham Seyffert18:08:12

And in addition, this happens while we’re issuing warmup requests against our application and we fire those off 1 at a time

Graham Seyffert18:08:42

So I can’t rule it out 100%, but based on how things are set up there should only be 1 thread loading at a time.

Graham Seyffert18:08:39

These namespaces are also loaded during the instantiation of a singleton that follows the LazyHolder pattern

Alex Miller (Clojure team)18:08:02

this is something we've been stepping towards fixing recently

Alex Miller (Clojure team)18:08:46

there is a (private) var in core now called serialized-require that uses a loading lock to avoid stuff like this

Alex Miller (Clojure team)18:08:16

not public yet as we're hoping to get rid of it and just make require have those semantics. but if you were in a position to repro it, you could use that instead of normal require and see if it fixed things.

Graham Seyffert18:08:38

Hmm interesting… I wonder why LazyHolder doesn’t fix this in this context whereas synchronized in the issue you linked does

Graham Seyffert18:08:59

The JVM is supposed to guarantee thread safety there

Alex Miller (Clojure team)18:08:23

it should, but maybe worth validating your code

Graham Seyffert19:08:27

Our code looks like this -

Graham Seyffert19:08:37

private static IFn A_FIELD_1;
    private static IFn A_FIELD_2;

    private static class LazyHolder {
        private static final AClass INSTANCE = new AClass();
    }
    public static AClass instance() {
        return LazyHolder.INSTANCE;
    }

    private AClass() {
        try {
            String namespaceOne = "";
            String namespaceTwo = "";

            // Throws in here
            ClojureUtil.requireNamespaces(namespaceOne, namespaceTwo);

            A_FIELD_1 = var(namespaceOne, "a-function");
            A_FIELD_2 = var(namespaceTwo, "another-function");
        } catch (Exception ex) {
            log.error("Exception while loading Clojure namespaces", ex);
        }
    }

Graham Seyffert19:08:47

In case you’re curious :man-shrugging:

Graham Seyffert19:08:48

Maybe just making that util requireNamespaces function synchronized would work

Graham Seyffert19:08:00

All of our namespace loading from Java goes through that right now

Alex Miller (Clojure team)20:08:24

is this the only class you have doing this?

Alex Miller (Clojure team)20:08:46

btw, looks like A_FIELD_1 and 2 should be final too

hiredman17:08:12

which can break things spectacularly

Graham Seyffert17:08:28

That’s very helpful actually

Graham Seyffert17:08:42

Time to go update those constructors haha

Graham Seyffert17:08:51

I suspect these old constructors are the root of some of our JDK 11 woes

Noah Bogart17:08:56

i have a namespace split across a bunch of files using in-ns. i'm finding that test-refresh doesn't reload the changes properly. has anyone gotten this to work together?

hiredman17:08:28

while that is a thing you can do, and it is something clojure.core does, it is not common practice so most tooling doesn't support it

hiredman17:08:37

so like, just don't do it

Noah Bogart17:08:53

I would have structured this differently if I had been the one to initially design it

Noah Bogart18:08:11

follow-up, then! I have a file that's in part of (in-ns 'game.core) namespace. I have a bunch of functions that are pure, and some ! functions that call the pure versions and apply the results to the state atom. I can't pull the whole file out into a new namespace because of circular dependencies: functions elsewhere in game.core call the ! functions, and the ! functions call other game.core functions (it's a giant mess, lol)

Noah Bogart18:08:19

in situations like this, what's the best method for making this less messy? should I pull the pure functions into a new namespace, and then require them? Should I pull out the stateful functions into the namespace, and have them take the needed functions (`update!`, etc) as arguments?

Noah Bogart18:08:14

if the answer is "who the heck knows", that's valid too! this is a plate of spaghetti, and I'm mostly fine with it, just annoying to test sometimes

lukasz18:08:55

introducing proper namespaces with pure and sideffecting functions sounds like the way to go

lukasz18:08:21

whatever turns spaghetti into ravioli is always a good call, then can be refactored later

😏 4
hiredman18:08:54

while the namespace system precludes static circular dependencies you can have circular dynamic dependencies

Noah Bogart18:08:18

do you have an example?

Noah Bogart18:08:30

or do you just mean using declare?

hiredman18:08:16

(ns a)

(defn f [g]
  (fn [x] (g x)))

(ns b
  (:require [a]))

(def f (a/f inc))

(ns c
  (:require [a b]))

(def f (a/f b/f))

👌 4
Noah Bogart18:08:39

is this an okay way to write this?

(ns game.stuff)
(defn break-sub!
  [update! state side ice sub]
  (update! state side (break-sub ice sub)))

(ns game.core
  (require '[game.stuff :refer [break-sub!]])

(defn update! [blah] (blah blah))

(defn break-sub!
  [state side ice sub]
  (break-sub! update! state side ice sub))

Noah Bogart18:08:14

@hiredman that's really good to know, thank you!

seancorfield18:08:56

That last function is a recursive call to itself -- not what you intend, I suspect?

seancorfield18:08:20

Instead of :refer use :as stuff and then call (stuff/break-sub! ,,,)

Noah Bogart18:08:27

oh yes, that's what I meant, thanks

seancorfield18:08:02

But I'm not sure why you'd bother with a wrapper like that? Why not just call stuff/break-sub! directly?

👎 4
hiredman18:08:02

it is often the case that people are bad at separating the runtime image/structure of their programs from the static structure of their source code

hiredman18:08:46

which is one of the reasons I like to use component, the static structure of the source code is what it is, and the runtime image is defined by component

👍 4
Noah Bogart18:08:28

as an example (this happens all over the place, lol), multiple places in game.core call break-sub!, and break-sub! relies on a function from game.core. I don't want to have to pass the game.core functions into every break-sub! call. I didn't know I could reference a function from another namespace without requiring it, tho

Noah Bogart18:08:56

thanks for the input, i'll have to play around with this some more!

hiredman18:08:23

I would strongly recommend you do pass in the functions

hiredman18:08:50

referencing functions without requiring them as an end run around circular namespaces dependencies is a bad idea

👍 4
Alex Miller (Clojure team)18:08:31

you have to require somewhere to load the code into the Clojure RT (via classloading)

Alex Miller (Clojure team)18:08:03

expecting that to be done implicitly is a bet you will eventually lose

Noah Bogart18:08:15

that makes sense

Alex Miller (Clojure team)18:08:33

for example, sometimes people call clojure.string/whatever without explicitly requiring clojure.string. that works, but only because Clojure bootstrapping happens to load clojure.string. If we optimize bootstrapping to not do that, then this code would break.

schmidt7322:08:24

Is there any good set of rules/emacs commands for indenting clojure code properly?

schmidt7322:08:43

how should we deal with long statements in lets

schmidt7322:08:56

also anyone use evil with something like paredit?

schmidt7322:08:25

here is an example:

Janus Troelsen22:08:09

it doesn't make much sense to me to use evil with paredit, either you are in one mode, or the other, no? i just switch manually

Aleksander09:08:59

I use evil-smartparens with strict mode + evil-cleverparens Works well!

schmidt7323:08:01

@ysangkok thx for the advice