This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-07-17
Channels
- # announcements (2)
- # babashka (1)
- # beginners (54)
- # calva (19)
- # clj-kondo (7)
- # cljs-dev (6)
- # cljsrn (18)
- # clojure (11)
- # clojure-europe (19)
- # clojurescript (7)
- # conjure (3)
- # cursive (7)
- # datomic (4)
- # figwheel-main (2)
- # fulcro (4)
- # helix (5)
- # honeysql (24)
- # instaparse (2)
- # lsp (20)
- # malli (17)
- # off-topic (6)
- # pathom (2)
- # practicalli (2)
- # shadow-cljs (2)
- # tools-deps (34)
- # vim (7)
I'm not sure if this is a known issue, but tools.build
doesn't work with Potekmin defprotocol+
(which skips redefining the underlying var/interface class if it already exists) -- at least not in cases where another namespace that require the one defining the defprotocol+
/etc. come before the one defining it alphabetically.
For example, suppose I have a namespace stuff
:
(ns stuff
(:require stuff.protocols))
and a namespace stuff.protocols
:
(ns stuff.protocols
(:require [potemkin :as p]))
(p/defprotocol+ MyProtocol ...)
tools.build/uber
emits a compile.clj
file that's more or less
(binding [*compile-path* "/tmp/compile-clj9219850936397513089/compile-clj"] ; number is random for each run
(compile 'stuff)
(compile 'stuff.protocols))
and then runs it. (compile 'stuff)
of course loads stuff.protocols
as a side effect, but this should mean stuff.protocols
(and thus MyProtocol
) should also ends up getting compiled as a side effects (if I understand correctly? Please correct me if I don't). It seems like what is happening is that the subsequent call to (compile 'stuff.protocols)
doesn't emit the init code for MyProtocol
(since it already exists) and ends up stomping over the previously-compiled version of stuff.protocols
. (This seems to be what's happening because the I have two different stuff/protocols$loading__1234__auto____<number>.class
files in the end, but from decompiling protocols__init.class
it seems like only one is actually getting used.)
Now what's interesting is that https://github.com/technomancy/leiningen/blob/4d8ee78018158c05d69b250e7851f9d7c3a44fac/src/leiningen/compile.clj#L154-L163, but this code works with lein uberjar
. AFAIK, the only difference is Leiningen does not put the working-compile-dir
(e.g. /tmp/compile-clj9219850936397513089/compile-clj
) on the classpath for the compilation process. This means the classes underlying MyProtocol
won't already exist no matter how many times stuff.protocols
gets compiled. The resulting uberjar when compiled with Leiningen only has a single stuff/protocols$loading_1234_auto__<number>.class
which supports this theory.
I have been able to get tools.build
to work with either one of a couple hacks. The first was to tweak clojure.tools.build.tasks.compile-clj/write-compile-script!
to include the lines
(require 'potemkin.macros)
(intern 'potemkin.macros 'equivalent? (constantly false))
in the compile.clj
script it generates (this effectively disable Potemkin's skip-class-redefinition logic).The second was to tweak compile-clj
to remove working-compile-dir
from the classpath. What's interesting tho is that even with the latter hack, tools.build
still ends up generating significantly more files (it still has the duplicate _loading__number_auto__
classes, for example). With Metabase, this ends up being a significant different in the number of files. Here are the number of classes in the entire uberjar and for compiling a single namespace with both tools.deps
and Leiningen:
# with tools.build
$ jar -tf target/uberjar/metabase.jar | wc -l
102171
$ jar -tf target/uberjar/metabase.jar | grep metabase.util.i18n.impl | wc -l
51
# with Leiningen
$ jar -tf target/uberjar/metabase.jar | wc -l
94706
$ jar -tf target/uberjar/metabase.jar | grep metabase.util.i18n.impl | wc -l
33
I haven't worked out why there's still duplicate classes with tools.build
yet. I wrote some bespoke compilation code that just finds all my namespaces with tools.namespace.find
and compiles them, and the numbers I get are similar to Leiningen's. I'd love to hear any ideas as to why the difference persists.
Does *compile-path*
actually need to be on the classpath? The documentation for it says
> This directory must be in the classpath for 'compile' to work.
But clearly it works even if it's not...
I'd love to hear any suggestions people have about how to deal with this. I have a feeling since Potemkin is a bit of a controversial library a fix specially for it might not be something I should hold my breath for. I think another way to fix this issue without removing the *compile-path*
from the classpath would be to compile things in a topological dependency order (https://github.com/cloverage/cloverage/pull/303) but it seems a little complicated for tools.build
... happy to submit a PR/patch if it's welcome thoYou can specify the namespaces to compile in order with compile-clj - did you try that?
It’s not expected that the default will work for every library, particularly for issues like this
we have a little over 400 namespaces for Metabase, so it's a bit too much to do by hand. Let me try programmatically determining the appropriate compile order for them and seeing if tools.build works without the hacks if I do that
honestly Potemkin has a quite extensive history of being problematic for various of its functionality
Ok, by passing in the namespaces in topological order (I basically just copied the Cloverage code I wrote last year to do it programmatically) it does actually work without the hacks, and doesn't create any duplicate files. The bad news is my build.clj
file is ~200 LoC now (just to replicate lein uberjar
-- nothing else like install
or jar
). Not so bad for me since I've been writing Clojure around 9 years now but I feel like potentially writing that much code to deploy a project has the potential to trip up newcomers to the language
but would very much like to make this work oob
yeah that's what I ended up doing.
tools.namespace topo sort needs a dependencies graph, and to build a dependencies graph you have to find all your namespaces (using tools.namespace.find) and then read the ns
forms in each of them (using ns.file
and ns.parse
).
It ended up being ~50 LoC to do all that
Ah bummer. Although I was thinking that actually might come in handy so you can ensure all the ones that depend on Potemkin can go first
if you want to contribute that code, that would be cool to have built in
assuming that works :)
Sure, I'd be happy to get it up to library quality and contribute it. What's the process for tools.build? Open an issue in JIRA and attach a patch? Or open a PR in the GitHub repo?
Don’t stress about the api, I’ll adjust as needed
I added a jira for this at https://clojure.atlassian.net/browse/TBUILD-7
Just generate a dummy file that requires all your namespaces and only compile that file
something like https://github.com/hiredman/tools.build/commit/3709c1b9748519a373908528406fccbb86b678fb
Hey, I am using the prerelease cli 1.10.3.905
and I am having a dependency that points to a local jar via :local/root
. I am getting the error
Error building classpath. Manifest type :jar not loaded when finding deps for ... in coordinate ...
When using the cli version 1.10.3.855
everything runs fine. Just wanted to report.thanks, fixed for next release
Just wanted to +1 the conversation about compilation order. I think I hit a similar issue as @camsaul just yesterday. I have bit more detail https://stackoverflow.com/a/68413578/4297413.
Should -x:deps find-versions be working for Clojure CLI version 1.10.3.855? it says function not found: clojure.tools.cli.api/find-versions, asking because the example is in the docs already 🙂
docs should probably guard that, but I'm hoping it's not more than a week or so till things come back together
Is there a way in deps.edn to set that the type of my dependency is pom only? Like this one https://mvnrepository.com/artifact/io.zonky.test.postgres/embedded-postgres-binaries-bom/13.2.0
Is it planed? We're using it a lot with our maven build to encapsulate/export version infos, plugins etc.
You can file a request on http://ask.clojure.org to gather interest…