Fork me on GitHub
#clojure
<
2022-11-14
>
respatialized15:11:42

I don’t see a channel for https://github.com/clj-commons/claypoole so I’m asking here: can I control the logging level for the threadpools? I get [WARN] messages that are redundant with another logging framework I use, and I’d like to disable them. But having read through the source, I can’t tell where to configure the logging, because there doesn’t appear to be any logging specific code in the library- is it upstream of Claypoole? Do I need to reconfigure logging for ExecutorService or something else?

kwladyka16:11:09

I didn’t use this tool but you should be able to make a filter for logging framework of java.util.logging if you use native logging and filter there by thread name.

Mathieu Pasquet20:11:57

Evening! Has anyone managed to get http://h2database.com/html/features.html#triggers working with Clojure? There's a pretty concise example of using triggers to keep the updated_at field of a database table up-to-date https://github.com/commandos59/h2database/issues/491#issuecomment-363628341, but I have no idea how you'd translate that to clojure.

Mathieu Pasquet20:11:55

So it may still be interesting to learn about getting triggers to work, but the use-case I had (setting the updated_at column in the database), is actually covered by generated columns: http://www.h2database.com/html/features.html#generated_columns

Ben Lieberman21:11:23

Seeing some strange behavior using comp with some host methods. I am calling an API and then mapping over results like so

(map #(.get %) (api-call))
This works fine and returns a seq of Apache BasicHttpResponse. But if I do (map #(comp (.getEntity %) (.get %)) (api-call)) it throws an exception: No matching field found: getEntity for class org.apache.http.impl.nio.client.FutureWrapper. Before .get is called, the values are BasicFutures but I don't really get where FutureWrapper is coming from

p-himik21:11:38

What exactly do you want to do in that second #(...)? In plain English terms.

Ben Lieberman21:11:26

I want to unwrap the futures, which to my understanding is what .get does

p-himik21:11:01

But that's what you're doing in #(.get %). What are you trying to achieve in #(comp ...)? comp is a higher-order function that composes other functions. You don't pass regular data in there.

Ben Lieberman21:11:17

Right, I could be using this .getEntity method wrong as I have never used the Apache client before today. But the method does exist on BasicHttpResponse: https://www.javadoc.io/doc/org.apache.httpcomponents/httpcore/4.4.5/org/apache/http/message/BasicHttpResponse.html#getEntity()

Ben Lieberman21:11:51

I am under the (maybe erroneous) impression that I can compose these two methods and get the underlying data.

sbocq21:11:30

Maybe you want (map (comp #(.getEntity %) #(.get %)) (api-call)) ?

p-himik21:11:32

That would work. But I'd just do (map #(.. % get getEntity) (api-call)). Or, if you're not a fan of .., then (map #(-> % .get .getEntity) (api-call)).

Ben Lieberman21:11:19

Both of those look better, imo. Thanks @U2FRKM4TW @U051G1K7Y!

👍 1
reefersleep09:11:51

comp is great though, you just have to remember that it’s a function that takes functions and returns a function 😅 Lots of HOF going on there! juxt is also nice.

Kiyanosh Kamdar22:11:05

Just wondering if there is a way to make the uber compiles faster? I’ve tried direct-linking and that hasn’t improved the compile times. Currently the uber build takes +10 minutes. It would be handy if I could just compile the one .class file and update the jar with it. Or something similar.

p-himik22:11:46

If you don't care about loading times, you should be able to create an uberjar without actually compiling anything - in other words, without calling (compile).

p-himik22:11:11

And when you call (compile) again, it should only recompile the changed files. So maybe when making an uberjar, you're also cleaning the already existing .class files?

Darin Douglass22:11:03

do you have a company-required file scanner that runs on your machine? i’ve seen small uberjars take ~4m to build with scanners on and <20s with the scanner disabled

Kiyanosh Kamdar22:11:49

I’ve tried removing the clean step, but that didn’t help.

Kiyanosh Kamdar22:11:01

I don’t think I have a scanner running, but let me double check on that.

Alex Miller (Clojure team)22:11:16

agree that 10+ minutes sounds very high and that this is usually file scanner issues

Alex Miller (Clojure team)22:11:28

what is # of files being compiled?

p-himik22:11:47

Could potentially be some blocking code at the top-level of some ns - after all, a namespace is evaluated when it's compiled, right?

Kiyanosh Kamdar22:11:32

our dependencies are huge. Let me get a count of *.class files in the target folder

Kiyanosh Kamdar22:11:27

around +50k *.class files are getting ubered

Alex Miller (Clojure team)22:11:39

but how many are being compiled?

Kiyanosh Kamdar22:11:56

how can I determine that?

Alex Miller (Clojure team)22:11:48

find target/classes "*.class" | wc -l 
maybe?

Alex Miller (Clojure team)22:11:00

assuming you're compiling into target/classes

Kiyanosh Kamdar22:11:31

there are 4828 class files just related to us

Alex Miller (Clojure team)22:11:44

the clojure CLI uberjar build compiles about 3500 classes in ~13 s on my machine

Kiyanosh Kamdar22:11:20

hmmm I can only imagine its all the dependencies that its pulling in.

Alex Miller (Clojure team)22:11:42

is the slow part the compile or the uber jar? you can wrap time calls around those steps in your build if you're not sure

Kiyanosh Kamdar22:11:59

The compile took about 2 minutes, the uber took the rest

Alex Miller (Clojure team)22:11:36

so uber took 8 minutes?

Kiyanosh Kamdar22:11:07

let me try uber without the compile in the picture

Alex Miller (Clojure team)22:11:50

the clojure CLI uberjar has about 75 libs and final jar has about 15k files size 20MB, takes about 31 sec, just as a comparison point

Alex Miller (Clojure team)22:11:20

certainly uber'ing directly into a jar would be faster and we may end up moving to that (there are complexities with managing conflicting files across jars), but usually when people have had these issues, they had something external with their filesystem or virus scanners affecting things

Kiyanosh Kamdar22:11:45

Ok, cool, let me take a look there. We have an EC2 instance where I can try the uber jar as well. That would tell us if its a scanner

Kiyanosh Kamdar23:11:12

That was it. I turned off the company Virus scanner, and brought it down to 3 minutes without cleaning. Cleaning adds another minute. Still now as fast as your stuff, but it is about 5 times bigger. So maybe that explains the rest.

👍 1
Kiyanosh Kamdar16:11:02

Hey @U064X3EF3 thanks for your help. I’ve got one simple question I hope. In the examples of an Uber build, why do we have the lines below:

(b/copy-dir {:src-dirs ["src" "resources"]
               :target-dir class-dir})

Kiyanosh Kamdar16:11:31

I assumed that the :path in deps.edn would be enough

Alex Miller (Clojure team)16:11:53

this copies the Clojure source into the target where it can be jar'ed. if you don't need/want that, you can take it out!

Alex Miller (Clojure team)16:11:25

you do probably want the resources to end up in the jar though, so maybe just remove "src" there

Kiyanosh Kamdar16:11:27

Thank you! Gotcha. I’m still relatively new to clojure, would there be some runtime need for the src ?

Alex Miller (Clojure team)16:11:56

it depends, but probably not

Alex Miller (Clojure team)16:11:12

if compiled classes exist, then they are preferred