Fork me on GitHub
#clojure
<
2020-11-25
>
seancorfield00:11:03

We have a few "rogue" JAR files at work that aren't available in a repo (and we've had to build ourself), and we just stick them in a lib folder accessible to our code and point :local/root at them.

seancorfield00:11:10

(I just checked and we're down to just one such JAR these days but Cognitect's REBL was also handled this way for quite a while: just download it somewhere and point a :local/root dependency at it)

jmckitrick00:11:19

@noisesmith Ah yes, CI/CD must be considered

jmckitrick00:11:44

@seancorfield This project is based on lein for the forseeable future.

noisesmith00:11:46

right, @seancorfield’s solution is to check the jar into the repo it seems

seancorfield00:11:16

(yeah, or build it on the fly from source and keep that source under git)

seancorfield00:11:43

For dev-only stuff, you don't need to check it in, if it's just in your local deps.edn file tho'...

jmckitrick00:11:04

Yes, it will certainly be Java under source control, but I want non-Clojure devs to be able to build it easily, and Clojure devs to consume it easily.

noisesmith00:11:36

I think for something like that, it makes sense to treat it as its own artifact in a maven repo

seancorfield00:11:37

For the particular "rogue" JAR I mentioned above, it's never going to change so it was a one-off add+commit. Ugly but...

noisesmith00:11:46

I'd treat a shared clojure library the same way

seancorfield00:11:17

Aye, I agree that I would expect a Java lib that needs to be built from source to have the artifacts published to a repo for the company.

seancorfield00:11:27

(since it's going to keep changing)

seancorfield00:11:08

We ran Apache Archiva for a similar reason for a couple of years (but it's pretty flaky and didn't seem well-maintained). If we needed to do that again, we'd probably use S3 buckets since that's a fairly widely support repository provider now for Clojure.

jmckitrick00:11:47

Maybe an S3 bucket would work

noisesmith00:11:53

there are alternatives I haven't tried

zendevil.eth01:11:51

🙋 Roll call! Who else is here?

👍 3
seancorfield04:11:57

(reminder that there are thousands of people here so keep things focused on Clojure)

Setzer2209:11:44

why is (or (not nil) (nil x)) an error? I have an optional parameter in my function, a predicate that may either be a lambda or nil, so I call it like this: (or (not f) (f x)) . But it gives me an error when f is nil. Doesn't or short-circuit the evaluation?

lukas.rychtecky10:11:11

Because nil is not a function, you probably want nil?

delaguardo10:11:36

or does short-circuit the evaluation

user> (def f nil)
;; => #'user/f
user> (or (not f) (f 1))
;; => true

lukas.rychtecky10:11:28

Don’t you want something like (when f (f x) ? Or you can (when (fn? f) (f x))

delaguardo10:11:06

but the forms like this (or (not nil) (nil 1)) will fail because first of all this form needs to be compiled and Compile exception will be thrown because of the form (nil 1) In case of macro it is easy to catch this case

lukas.rychtecky11:11:54

Yeah that’s possible because (nil 1) makes no sense.

rickmoynihan10:11:02

Just spotted LazilyPersistentVector in the clojure implementation, and I’m guessing this is part of the logic that keeps collections smaller than 32 elements as essentially linear arrays, but promotes them when they’re larger than that?

zilti10:11:55

I have a question to reducers. I have a line that uses r/map . So far I had r/foldcat in my code, but while that ran everything in parallel as expected, I didn't get the result I wanted out of it. I see in the reducers reference it says "To produce an output collection, use https://clojure.github.io/clojure/clojure.core-api.html#clojure.core/into". When I wrap the call to r/map in an into will it still be run in parallel?

reborg11:11:07

Nope, just r/fold or r/foldcat (which is an r/fold in disguise) enable parallelism for foldable collections (beyond a certain size)

reborg11:11:33

Wait, if you mean on top of r/foldcat then yes

(require '[clojure.core.reducers :as r])
(def input (r/map inc (into [] (range 1000))))
(into [] (r/foldcat input))
> [1 2 3...

zilti11:11:23

No, not on top of r/foldcat, because since the input to it is a vector of maps from r/map, it doesn't process them correctly (and seems to just merge the maps)

zilti11:11:09

What I did now is write a reducef and wrap it in a (r/fold reducef (r/map ...)).

(defn- reducef
  ([] [])
  ([a b]
   (if (and (vector? a) (vector? b))
     (into a b)
     (conj a b))))

reborg12:11:26

Worth remembering that r/foldcat is not an equivalent mapcat or concat of sort. The “cat” part in r/foldcat is mainly an implementation detail. But it seems to work as advertised in your case?

(require '[clojure.core.reducers :as r])
(def large-map (into {} (map vector (range 1000) (range 1000))))
(def vector-maps (into [] (repeat 1000 large-map)))
(def input (r/map #(assoc % :new :key) vector-maps))
(:new (first (r/foldcat input)))
;; :key
(count (into [] (r/foldcat input)))
;; 1000

lmergen11:11:07

out of curiosity, how do Clojure’s short-lived functions behave with the JVM JIT? assuming I have a reduce in a hot function that uses a short-lived reduction function, I can only assume the JVM “forgets” the previous allocations (and invocations) of these short lived functions, right ?

dominicm13:11:32

What is "short lived"?

dominicm13:11:09

Unless you're using eval, all functions are compiled to a class and have their locals available.

andy.fingerhut14:11:58

Anonymous functions are compiled once, not each time the function around them is called.

👍 3
andy.fingerhut14:11:58

And in general local functions defined within another function are compiled once, whether they are anonymous or have a name, so the "anonymous" part of my previous statement is irrelevant.

andy.fingerhut14:11:18

The JVM JIT should see local Clojure functions as just another class with methods to execute. No different from top level functions.

lmergen15:11:48

for example, something like this:

(defn my-transform [prefix input]
   (reduce (fn [xs x]
              (conj xs (str prefix x)))
           {} input))

(my-transform "prefix" ["a" "b" "c"])
i am wondering what the behavior of the local (fn [xs x]) is for the JIT

lmergen15:11:22

especially since the behavior of this function depends upon an external input

lmergen15:11:41

but as i understand now, this is actually compiled once?

dominicm15:11:59

Compiled once, yep.

andy.fingerhut15:11:09

It is compiled once, and if you look at the JVM byte code, and/or the Java source code decompiled from that, you will see that the class created for the (fn [xs x] ...) function has a constructor that takes prefix as a parameter. So the local function is an instance of a class that is constructed once on each call to my-transform, but the class remains from one call of my-transform to the next. The only thing that is allocated fresh on each call to my-transform is an instance of that class.

andy.fingerhut15:11:58

and if I recall correctly, the only thing in those instances are things like a local variable for each value in the inner function's environment, like prefix in your example, and the constructor does not do anything except assign values to those fields and return, so they should be as light weight as any constructor can be.

lmergen15:11:34

this is a great answer, thanks a lot! this was exactly what i was wondering, so the good news is that all this integrates really well into the JIT of the JVM 🙂

andy.fingerhut17:11:29

Since I have seen a similar question only a couple of weeks ago, and did most of this investigation at that time, I decided to put the results into a short article that you might find interesting: https://github.com/jafingerhut/jafingerhut.github.com/blob/master/notes/clojure-inner-functions.md

lmergen20:11:32

it’s great

wombawomba14:11:52

Let's say I have a macro that I want to conditionally define a few functions depending on its input — how can I break out the 'function-building' part into functions? For instance, let's say I have something like:

(defmacro foo [x]
  `(if (even? ~x)
     (defn ~(symbol (str "foo-" x) [] (inc x)))
     (defn ~(symbol (str "bar-" x)) [] (dec x))))
and I want to turn this into something like
(defn make-foo [x] `(defn ~(symbol (str "foo-" x) [] (inc x))))
(defn make-bar [x] `(defn ~(symbol (str "bar-" x) [] (inc x))))
(defmacro foo [x] `(if (even? ~x) (make-foo ~x) (make-bar ~x)))
...how can I get the result of make-foo and make-bar to be evaluated?

wombawomba14:11:27

actually maybe this is a bad example

wombawomba14:11:27

What I'm actually doing is:

(defmacro foo [x]
  `(let [x# (some-fn x)]
     (if (even? x#)
       (make-foo x#)
       (make-bar x#))))

wombawomba14:11:12

and what ends up happening here is that my defns end up getting returned to the code calling the macro without being evaluated

scknkkrer14:11:46

def  prefix explains that the function define a `var` in the namespace. And, `def*`  calls must be top level calls. Any other usage of them is not right.

wombawomba14:11:53

I'd do ~(make-foo x#) to solve this but that doesn't work because x# isn't available unquoted

scknkkrer14:11:16

I recommend you to consider about your data-flow again.

wombawomba14:11:45

@U3JJH3URY if that's the case, you should probably stop using defn 😉

scknkkrer14:11:08

Sorry to bother you, it’s a common knowledge, I just wanted to inform you.

wombawomba14:11:44

I'm aware that non-top-level defs are typically discouraged, but there are places where it's appropriate to make use of them (like in the definition of defn)

wombawomba14:11:02

just wanted to inform you back 🙂

Ed17:11:44

you're calling functions from a macro that just return lists of symbols. for make-foo and make-bar to be macro expanded into the defn for evaluation, they need to be macros not functions ... does that make sense?

👍 3
Ed17:11:40

but, if you're always passing in literal values like that, you could take the syntax quote out and just return the code from foo?

👍 3
Timur Latypoff14:11:18

Are there any editors (or IDE plugins) that somehow highlight the S-expressions which return values from a function? Or maybe there's an easy heuristic that everybody uses? As a newbie-lisper, I find it really difficult to mentally parse a large nested structure of let/loop/if/do/implicit-do to find all the places where expressions' values are not discarded and they form function's return values.

andy.fingerhut14:11:36

I do not know of any editor/IDE help related to this, but I haven't looked for such a thing, either.

andy.fingerhut14:11:27

I would suggest that another technique that would help with this, and perhaps have other benefits as well, is to make some effort at breaking sub-expressions within a large block of code into separate functions.

andy.fingerhut14:11:06

In general, smaller expressions / functions make it easier to understand a lot of things about them, including what you asked about, and can also make it easier to develop and test the smaller functions, sometimes separately.

👍 3
dominicm14:11:18

The most deeply nested code at any point is usually the final step I guess. But this isn't something I generally find myself searching for in my programs either.

👍 3
Timur Latypoff14:11:52

@U0CMVHBL2 I agree. I was just making some quick-and-dirty java-interop megafunction, and wondered if there's already such a "tail detection" implemented somewhere :)

andy.fingerhut15:11:54

Of the IDEs I know of, I don't have deep knowledge of them, but #cider has probably been around the longest, and #cursive has a full time commercial developer behind it, so those are the two IDEs that I'd guess have the most features, and therefore the most likely to have what you are asking for, if any of them do.

potetm15:11:40

I’ve written some pretty gnarly clojure, and I’ve never had the problem of not being able to easily see the return form.

potetm15:11:09

Curious if you have some sort of example. do blocks should be rare enough that they stand out.

potetm15:11:28

Oh, you know what helps: indentation markers.

potetm15:11:39

together with properly formatted code

potetm15:11:49

(This is Cursive.) You can see the indentation markers on the left. Also, rainbow parens help to easily spot, e.g. where the finally block should go.

Timur Latypoff16:11:23

@U07S8JGF7 yeah, using these already. They help, but really don't give me feeling that I have everything covered. For example, upon careful examination, I see three tail positions in my code: 1. (recur) 2. :error 3. (swap!) But the function spans up a couple of screens more 🙂

Timur Latypoff17:11:42

Aaand I was wrong, these are not tail positions, because they turned out to be inside a doseq 😞

potetm17:11:56

So, some things

potetm17:11:19

This sort of side-effect heavy algorithm is never going to feel good in clojure.

potetm17:11:55

There is an upside to that: It strongly encourages you to isolate side effects (which most developers are poor at).

potetm17:11:16

The downside is: Any algorithm that really needs to be side-effect heavy is just going to feel bad.

potetm18:11:20

IME those cases are relatively rare. It’s usually the case that I need to separate logic from side effects.

potetm18:11:29

But those cases do exist. And they’re just not fun.

👍 3
Jonas Claesson16:11:14

I am writing a generative test case in kaocha, using fdef and a local function, but it seems like I get a null pointer exception when calling spec/exercise-fn. The function exercise-fn is calling is declared with letfn. Could that be the reason for my problems?

seancorfield18:11:20

For exercise-fn to run, your function needs at least :arg specs -- and you aren't writing a spec for test-concept-paging

seancorfield18:11:58

user=> (letfn [(foo [n])] (s/exercise-fn `foo))
Execution error at user/eval24110 (REPL:1).
No :args spec found, can't generate
@U01CJ5A5TAA

seancorfield18:11:35

Where is your fdef?

Jonas Claesson18:11:13

(spec/fdef test-concept-paging :args (spec/cat :query ::concept-paging-query :pagination ::paging) :ret (spec/and boolean? #(= true %)))

seancorfield19:11:21

That would be a global (top-level) function tho', right?

Jonas Claesson19:11:21

But the problem might be that the implementation is local?

seancorfield19:11:36

And in letfn, that's a different function -- it's a local name that shadows any global.

Jonas Claesson19:11:47

Ok, now I see. The top-level fdef doesn't find the local one.

Jonas Claesson19:11:54

test-concept-paging is only defined locally.

Jonas Claesson19:11:17

So then I need to define the fdef at the same local level as test-concept-paging?

seancorfield19:11:08

I'm pretty sure fdef only works on top-level functions.

Jonas Claesson19:11:20

Thanks for the help. I'll see tomorrow, how to do that. But is this the right way to test a function that has some parameters constant and the same, and others that you want to vary?

Jonas Claesson19:11:37

Unfortunately it seems that way. I did a quick test

seancorfield19:11:29

clojure.spec is not a type system 🙂

seancorfield19:11:48

(local functions are generally just considered an implementation detail)

seancorfield19:11:18

In the code you shared, I would expect generate-page-params to have the fdef and be the function under test.

Jonas Claesson19:11:56

Ok, thanks! I'll try to rearrange the functionality tomorrow.

Jonas Claesson16:11:32

(test/deftest ^:integration-generative-concepts-paging generative-test-concepts-paging (test/testing "test main/concepts paging generatively" (let [_ (db/create-version-0) concepts (assert-concepts (rand-int 4)) ;; time consuming operation _ (versions/create-new-version 1)] (letfn [(test-concept-paging [query pagination] ;; Return true if the test passed. (let [params (generate-page-params (count concepts) query pagination)] true))] (let [res (spec/exercise-fn `test-concept-paging 10)] (doall (map #(test/is (= true true)) res)))))))

Jonas Claesson16:11:53

user=> (use 'kaocha.repl) user=> (run 'jobtech-taxonomy-api.test.generative-test/generative-test-concepts-paging)

Jonas Claesson16:11:12

Uncaught exception, not in assertion. Exception: java.lang.NullPointerException: null at clojure.core$apply.invokeStatic (core.clj:665) ... clojure.spec.alpha$exercise_fn$iter__2569__2573$fn__2574.invoke (alpha.clj:1881) ... jobtech_taxonomy_api.test.generative_test$fn__68371.invokeStatic (generative_test.clj:145) jobtech_taxonomy_api.test.generative_test/fn (generative_test.clj:135) jobtech_taxonomy_api.test.test_utils$fixture.invokeStatic (test_utils.clj:48)

Jonas Claesson16:11:47

generative_test.clj:145 is the line with spec/exercise-fn

Jonas Claesson16:11:39

I have written generative tests before, but never using a local function. The reason I have the local function is because I want to share some data across the test. This data takes a long time to generate, and I do not want to regenerate it for each time the tested function is called.

respatialized18:11:31

not quite sure if this is the right channel to ask this question, but I was wondering if anyone has ever done side-by-side pair programming with two REPL clients connected to the same REPL server?

Ed18:11:34

yes ... I found it very confusing when someone else changed a function underneath me ... it was better to have one typing at a time

Ed18:11:09

if you do things like load-file or cider-load-buffer then you can clobber changes that have been loaded in from someone else's machine without thinking about it ... it's a very easy thing to do

borkdude18:11:05

@UFTRLDZEW I've heard some people do this via LiveShare in VSCode

✔️ 3
borkdude18:11:30

Ask in #calva for details.

respatialized18:11:53

@U0P0TMEFJ yeah I figured it could get really weird really quick if you're not careful, but it also seemed like an interesting technique in the distributed-by-default setting we're all in now, which is why I wanted to ask about people's experiences

Bob B18:11:04

I think if you're doing pair programming in the sense that you're communicating about what changes each other are making and you're working toward a common goal, I could see it working, but if multiple people are redefining things in a REPL without communicating it, then it feels like essentially the whole "shared mutable state" problem, where the threads are people instead of program threads

3
paul.legato19:11:14

Yes, I’ve always found shared keyboard control creates mutex issues. One person “driving” (typing) at a time, taking turns, seems to work better for me.

cap10morgan18:11:26

How would one set any of the clojure.core/*config-flag* type vars from Java?

cap10morgan18:11:38

I'm writing a plugin in a Java environment that hands me a pretty restricted classloader and I think I need to set *use-context-classloader* to false before I invoke any clojure code.

dominicm19:11:51

You might see an example in bukkure

cap10morgan22:11:41

thanks! I don't see any instances of use-context-classloader in there from a GitHub search.

dominicm07:11:51

It definitely messes with the class loader of clojure very early

st3fan18:11:25

var: #'clojure.core/assert-args is not public. - hmm this seemed like such a useful macro to use in my own code

st3fan18:11:42

I just copied it over to my own 'core' lib

Russell Mull20:11:23

Does anybody know how to get javadoc to work in cider? I've used it for a decade, and never seen it do anything useful.

Russell Mull20:11:58

oh right on, didn't know that was a thing. Thanks.

noisesmith20:11:56

clojure.java.javadoc is aliased to javadoc in the repl by default, and uses the web browser

Lennart Buit21:11:15

(it can even try to find javadoc using google’s I’m feeling lucky URL, which I found amusing when I stumbled over that)

Alex Miller (Clojure team)21:11:00

(although that is broken right now as google changed the url)