tools-build 2022-05-23 | Slack Archive

ingesol19:05:17

Hi! I’m hitting a performance bottleneck when AOT-compiling my large project with clojure.tools.build.tasks.compile-clj/compile-clj . I narrowed the problem down to clojure.tools.build.util.file/copy-contents, it spends about 5 minutes copying my roughly 35k class files from working directory to target/classes. I suspected some other service might be keeping my M1 Macbook busy, but nothing found so far. Does this sound familiar to anyone? Is the problem as simple as just having way too many class files?

Darin Douglass19:05:11

Do you have any company-prescribed security software scanning your file system? I've had some pretty drastic improvements disabling said software when building / compiling

Darin Douglass19:05:44

Like 5m -> 30s or so

ingesol19:05:53

@U0NCTKEV8 Oh, maybe I misinterpreted the meaning of this part then https://github.com/clojure/tools.build/blob/master/src/main/clojure/clojure/tools/build/tasks/compile_clj.clj#L109. I ran that in a modified version, just logged the time before and after it.

ingesol19:05:18

@U02EA2T7FEH Could be, I should probably look into that.

hiredman19:05:41

No, I just naively assumed it was compiling into the target directory

hiredman19:05:12

Good thing ssds exist I guess

🙂 1

favila19:05:20

just checking: you are using an m1-native jvm binary and not an x86 one?

ingesol19:05:16

@U09R86PA4 That is also an excellent question. I’m using one that was installed by company scripts, it is JDK 17 and it says “Apple” under “Type” in activity monitor, which should indicate it’s m1-native?

favila19:05:14

I think that’s what that means

Alex Miller (Clojure team)20:05:46

it compiles into a tmp dir, then copies back, but could potentially do otherwise. if you want to file a question on https://ask.clojure.org that would help get it into the queue for me to look at

seancorfield20:05:19

Related to that, depstar 1.x used to work exactly as tools.build does right now, but depstar 2.x switched to building the JAR directly from the tmp dir to avoid that extra copy step and it made a big difference to uberjar build times when AOT was used. The separation of compile & copy means you have a cleaner target folder if compilation fails -- it isn't populated with a mix of old and new classes (assuming you didn't clean it beforehand) or even a partial set of new classes.

seancorfield20:05:57

(we've switched to building our uberjars in CI only so we don't care about the additional time -- and CI is much faster than our dev machines anyway)

ingesol21:05:15

@U04V70XH6 We recently chose the approach of cleaning the target folder before every startup, to avoid some issues we saw with hot reloading and stale/duplicate class files. So we are fine with compiling directly to target.

ingesol21:05:22

@U064X3EF3 Thanks, I posted a question.

ingesol06:05:22

Thanks everyone! We ended up forking tools.build to compile directly to target/classes. And @U02EA2T7FEH was right, our problem seems to have been caused by security software scanning file events.

🙌 1

2022-05-23

Channels