Clojurians Log v2

Actually, I wonder if it might be the old corporate antivirus software interference story once again...

have you tried https://github.com/clojure-goes-fast/clj-async-profiler on it? I think wrapping that in a prof/profile instead of time should tell you where all that time is going

I haven't yet. I'll give that a try. 👍

The resulting uberjar is 29M, so nothing out of ordinary.

yeah I don't see anything that catches my attention there 🤔

I don't see the GC activity on the graph, was there any?

I don't know. I'm not very adept at using clj-async-profiler. 🙂 Here's the HTML version if you want to take a look.

that small thing in yellow on the right shows GC and Hotspot compilers activity, but is very small

Right, thanks. 👍 At least I learned something new. 🙂

Maybe I'll try strace or whatever the macOS equivalent is...

yeah could be a good idea, if you grep out all those pesky futex calls

Unless I'm foiled by System Integrity Protection...

or you can also run the profiler for the lein process and compare the flamegraphs? maybe there is something there. You can download async profiler https://github.com/async-profiler/async-profiler, unzip that into a folder, and then what I do is run ./profiler.sh -e cpu -d SECONDS_TO_SAMPLE -f /tmp/flamegraph.html YOUR_LEIN_PID

so you can run that against any java process and it will give you the flamegraph

Thanks, I'll give that a try if the dtrace path is a dead end. 👍

I can't remember whether Leiningen explodes all the JARs on the classpath into the target/classes folder before building the uberjar -- I suspect it doesn't -- so if you have antivirus software checking every new file created, tools.build.api/uber is going to be badly affected since it explodes JARs and copies everything into the target tree first, then assembles the JAR from all those files... lots of file activity. (depstar originally did that in two passes but I switched it to copy everything directly into the JAR, exploding JARs on the classpath only in memory -- made it run over 2x faster every without virus checking)

That definitely sounds plausible, thanks. 👍🏻

We use https://github.com/tonsky/uberdeps (which “does not unpack intermediate JARs on disk.”) in our tools.deps projects. The poor performance (even on a Mac without antivirus software) of tools.build.api/uber is the only reason that’s preventing us from switching to it.

I see, thanks. I used to use uberdeps before tools.build was a thing, but I think I’ll give it a try to see how it compares. 👍🏻

https://clojurians.slack.com/archives/C02B5GHQWP4/p1640887876383200?thread_ts=1637254945.235200&cid=C02B5GHQWP4 this sounds like the exact same issue.

Cool. Can you get it configured to ignore certain folders? I've generally found antivirus software to be pretty problematic for developers in any situation where code is getting compiled or build systems are shuffling a lot of files around...

I believe there was an opportunity to do something like that at my previous job, but it used a different AV program... I'll have to ask around.

Yeah, I can reproduce the issue with e.g. https://github.com/seancorfield/usermanager-example. I'm on a pretty beefy MacBook Pro M2, and (b/uber (uber-opts {})) takes over a minute, whereas on a much less capable Linux box (with no corporate malware) it takes ~5 seconds.

Thanks for confirming that @flowthing -- Hopefully, you can address the situation with your IT team because I would expect that A/V tool to slow down a lot of software development tasks.

Just one final comment for posterity, in case anyone finds this thread via Slack search or something: I added exclusions for the folders I use for dev via Virus & threat protection settings » Manage settings » Add or Remove Exclusion... in Microsoft Defender (which our corporate policy luckily allowed), and that made a big difference: building an uberjar via clojure.tools.build.api/uber now takes ~45 seconds instead of 5 minutes.