Fork me on GitHub
#tools-deps
<
2021-07-17
>
Cam Saul01:07:29

I'm not sure if this is a known issue, but tools.build doesn't work with Potekmin defprotocol+ (which skips redefining the underlying var/interface class if it already exists) -- at least not in cases where another namespace that require the one defining the defprotocol+/etc. come before the one defining it alphabetically. For example, suppose I have a namespace stuff :

(ns stuff
  (:require stuff.protocols))
and a namespace stuff.protocols:
(ns stuff.protocols
  (:require [potemkin :as p]))

(p/defprotocol+ MyProtocol ...)
tools.build/uber emits a compile.clj file that's more or less
(binding [*compile-path* "/tmp/compile-clj9219850936397513089/compile-clj"] ; number is random for each run
  (compile 'stuff)
  (compile 'stuff.protocols))
and then runs it. (compile 'stuff) of course loads stuff.protocols as a side effect, but this should mean stuff.protocols (and thus MyProtocol) should also ends up getting compiled as a side effects (if I understand correctly? Please correct me if I don't). It seems like what is happening is that the subsequent call to (compile 'stuff.protocols) doesn't emit the init code for MyProtocol (since it already exists) and ends up stomping over the previously-compiled version of stuff.protocols. (This seems to be what's happening because the I have two different stuff/protocols$loading__1234__auto____<number>.class files in the end, but from decompiling protocols__init.class it seems like only one is actually getting used.) Now what's interesting is that https://github.com/technomancy/leiningen/blob/4d8ee78018158c05d69b250e7851f9d7c3a44fac/src/leiningen/compile.clj#L154-L163, but this code works with lein uberjar. AFAIK, the only difference is Leiningen does not put the working-compile-dir (e.g. /tmp/compile-clj9219850936397513089/compile-clj) on the classpath for the compilation process. This means the classes underlying MyProtocol won't already exist no matter how many times stuff.protocols gets compiled. The resulting uberjar when compiled with Leiningen only has a single stuff/protocols$loading_1234_auto__<number>.class which supports this theory. I have been able to get tools.build to work with either one of a couple hacks. The first was to tweak clojure.tools.build.tasks.compile-clj/write-compile-script! to include the lines
(require 'potemkin.macros)
(intern 'potemkin.macros 'equivalent? (constantly false))
in the compile.clj script it generates (this effectively disable Potemkin's skip-class-redefinition logic).The second was to tweak compile-clj to remove working-compile-dir from the classpath. What's interesting tho is that even with the latter hack, tools.build still ends up generating significantly more files (it still has the duplicate _loading__number_auto__ classes, for example). With Metabase, this ends up being a significant different in the number of files. Here are the number of classes in the entire uberjar and for compiling a single namespace with both tools.deps and Leiningen:
# with tools.build
$ jar -tf target/uberjar/metabase.jar | wc -l
102171
$ jar -tf target/uberjar/metabase.jar | grep metabase.util.i18n.impl | wc -l
51

# with Leiningen
$ jar -tf target/uberjar/metabase.jar | wc -l
94706
$ jar -tf target/uberjar/metabase.jar | grep metabase.util.i18n.impl | wc -l
33
I haven't worked out why there's still duplicate classes with tools.build yet. I wrote some bespoke compilation code that just finds all my namespaces with tools.namespace.find and compiles them, and the numbers I get are similar to Leiningen's. I'd love to hear any ideas as to why the difference persists. Does *compile-path* actually need to be on the classpath? The documentation for it says > This directory must be in the classpath for 'compile' to work. But clearly it works even if it's not... I'd love to hear any suggestions people have about how to deal with this. I have a feeling since Potemkin is a bit of a controversial library a fix specially for it might not be something I should hold my breath for. I think another way to fix this issue without removing the *compile-path* from the classpath would be to compile things in a topological dependency order (https://github.com/cloverage/cloverage/pull/303) but it seems a little complicated for tools.build... happy to submit a PR/patch if it's welcome tho

Alex Miller (Clojure team)01:07:00

You can specify the namespaces to compile in order with compile-clj - did you try that?

Alex Miller (Clojure team)01:07:50

It’s not expected that the default will work for every library, particularly for issues like this

Cam Saul01:07:23

we have a little over 400 namespaces for Metabase, so it's a bit too much to do by hand. Let me try programmatically determining the appropriate compile order for them and seeing if tools.build works without the hacks if I do that

vemv15:07:06

honestly Potemkin has a quite extensive history of being problematic for various of its functionality

Cam Saul02:07:47

Ok, by passing in the namespaces in topological order (I basically just copied the Cloverage code I wrote last year to do it programmatically) it does actually work without the hacks, and doesn't create any duplicate files. The bad news is my build.clj file is ~200 LoC now (just to replicate lein uberjar -- nothing else like install or jar). Not so bad for me since I've been writing Clojure around 9 years now but I feel like potentially writing that much code to deploy a project has the potential to trip up newcomers to the language

Alex Miller (Clojure team)02:07:22

I don't think newcomers have 400 namespaces :)

😅 3
Alex Miller (Clojure team)02:07:35

but would very much like to make this work oob

dpsutton02:07:54

We probably have tools namespace on the class path. Can’t that topo sort them?

Cam Saul02:07:25

yeah that's what I ended up doing. tools.namespace topo sort needs a dependencies graph, and to build a dependencies graph you have to find all your namespaces (using tools.namespace.find) and then read the ns forms in each of them (using ns.file and ns.parse). It ended up being ~50 LoC to do all that

dpsutton02:07:32

Ah bummer. Although I was thinking that actually might come in handy so you can ensure all the ones that depend on Potemkin can go first

Alex Miller (Clojure team)02:07:59

if you want to contribute that code, that would be cool to have built in

Cam Saul02:07:46

Sure, I'd be happy to get it up to library quality and contribute it. What's the process for tools.build? Open an issue in JIRA and attach a patch? Or open a PR in the GitHub repo?

Alex Miller (Clojure team)04:07:46

Don’t stress about the api, I’ll adjust as needed

hiredman02:07:44

Just generate a dummy file that requires all your namespaces and only compile that file

hiredman02:07:05

imo anything that calls compile more than once is going to break things

hiredman02:07:58

If you do the dummy file you don't need to sort anything

Cam Saul03:07:36

thanks! I'll have to try that

hiredman03:07:35

I haven't run it, so it may need some debugging

hiredman03:07:15

hrmm I see a bug already, the namespace needs ~'

FiVo14:07:38

Hey, I am using the prerelease cli 1.10.3.905 and I am having a dependency that points to a local jar via :local/root . I am getting the error

Error building classpath. Manifest type :jar not loaded when finding deps for ... in coordinate ...
When using the cli version 1.10.3.855 everything runs fine. Just wanted to report.

Alex Miller (Clojure team)21:07:32

thanks, fixed for next release

Eddie19:07:09

Just wanted to +1 the conversation about compilation order. I think I hit a similar issue as @camsaul just yesterday. I have bit more detail https://stackoverflow.com/a/68413578/4297413.

👍 3
Drew Verlee21:07:32

Should -x:deps find-versions be working for Clojure CLI version 1.10.3.855? it says function not found: clojure.tools.cli.api/find-versions, asking because the example is in the docs already 🙂

Alex Miller (Clojure team)21:07:36

no, only working in prerelease

👍 3
Alex Miller (Clojure team)21:07:33

docs should probably guard that, but I'm hoping it's not more than a week or so till things come back together

rafaeldelboni22:07:42

Is there a way in deps.edn to set that the type of my dependency is pom only? Like this one https://mvnrepository.com/artifact/io.zonky.test.postgres/embedded-postgres-binaries-bom/13.2.0

Alex Miller (Clojure team)22:07:49

no, that's not supported in deps.edn

👍 3
henrik4209:07:03

Is it planed? We're using it a lot with our maven build to encapsulate/export version infos, plugins etc.

Alex Miller (Clojure team)12:07:43

You can file a request on http://ask.clojure.org to gather interest…