Clojurians Log v2

i’ll go ahead and open another issue about incompatibility with :aot flag even if not fixed it might help future desperate and confused devs such as myself to figure out the multiple prefix symptom

https://github.com/benedekfazekas/mranderson/issues/89

ok, so i’m using the original repo https://github.com/fullcontact/full.aws on master. here are my steps and results so far: 1. Checkout orig repo full.aws 2. lein do clean, deps, test => ✅ 3. add mranderson 5.3.0 to plugins > lein do clean, deps, test => ✅ 4. add ^:inline-dep to all full.* deps and to jaxb-api > lein do clean, deps, test => ✅ 5. lein inline-deps > RuntimeException: No reader function for tag Inf => 🤦 6. change plugin version to [thomasa/mranderson "0.5.4-SNAPSHOT"] > lein inline-deps > ✅ so 0.5.3 seems broken, while 0.5.4-SNAPSHOT can run inline-deps, at least on my env.

7. lein with-profile +plugin.mranderson/config test =>

Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
fullaws100SNAPSHOT.fullaws100SNAPSHOT.fullaws100SNAPSHOT.clojure.lang.IRecord

(:import (fullaws100SNAPSHOT.fullaws100SNAPSHOT.fullaws100SNAPSHOT.clojure.lang IRecord IType IObj IReference Var)))

8. search & replace: (fullaws100SNAPSHOT\.){2,} => fullaws100SNAPSHOT. the above import looks sane now, but still fails

Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
fullaws100SNAPSHOT.clojure.lang.IRecord

not sure how to move forward from here, i’m stuck

Nice summary @mor.gazith! I'll start with reproducing your steps (probably) tomorrow, and then digging into what is going on.

thank you 🙏

Ok, so problem 1 of n might be related to :aot :all , will recheck that later, but for now, we'll not do that. Problem 2 of n seems to be that MrAnderson is deleting the full directory that it generated under target/srcdeps . It does not take into account that a lib that you are inlining might share the same root namespace as the project you are inlining from. In our latest test case: full.aws is inlining full.json. If I rename nses under src in the full.aws lib from full.* to notfull.* I see ./target/srcdeps/notfull.* generated as expected. I'll raise this as an issue on MrAnderson.

@mor.gazith what if your GitHub user id? If you are ok with it, I'd like to reference it in the issue.

sure, I'm at @marmor7

wow, good catch!

thanks!

https://github.com/benedekfazekas/mranderson/issues/88

@mor.gazith side note: isn't it a bit odd that fullcontact libraries include .class files? That's typically not a good practice for a library, is it?

I'm looking at full.core-1.1.1.jar and it seems to include .class files for its sources and all its deps.

And same for full.json-0.12.0.jar. Could this be contributing to your jackson dependency hell issues?

i have no idea about that… i that related to the :oat thing? maybe jvm is compiling all java imports to class files and adding them to the resulting jar?

is there something in the lib itself or the compile process that causes those class files to get into the jar file?

Yeah it is due to :aot. Folks don't typically aot (compile .clj to .classes) for libraries. They will sometimes aot compile their apps for faster startup.

Ok starting to try things out. I think I'll verify 0.5.4-SNAPSHOT as other folks are using that in production.

I'm not sure that lein test has value? I don't see any tests for full.aws. Maybe it at least loads some nses? I dunno.

Oh ya, I guess that is its value, as I see you get a failure from launching tests.

it'll output 0 tests which means it complied successfully

@mor.gazith what JDK are you using? Probably should match that too.

I think I'm running jdk8 for this one

Ok. Also can you paste your project.clj ? Just to be sure we are using same.

That's a lot of inline deps yeah? Takes a while!

yep, tell me about it 🙂 took over 5 minutes

maybe I'm using it wrong? my aim was to bump the aws SDK version without causing Jackson dependency hell

which I had before hearing about mranderson

Hmm... yeah, are you actually suffering Jackson dependency hell or just trying to be proactive?

Also: I seem to be getting a different result than you:

❯ lein with-profile +plugin.mranderson/config test
Compiling full.aws.core
Syntax error macroexpanding at (core.clj:1:1).
Execution error (FileNotFoundException) at full.aws.core/loading (core.clj:1).
Could not locate full/core/config__init.class, full/core/config.clj or full/core/config.cljc on classpath.

Full report at:
/tmp/clojure-4717255487822232053.edn
Compilation failed: Subprocess failed (exit code: 1)
Error encountered performing task 'test' with profile(s): 'base,system,user,provided,dev,config'
Compilation failed: Subprocess failed (exit code: 1)

I am suffering hell, for over a week before going with the shading approach

I never got that FileNotFound issue

Can you paste the project.clj you tested with above?

for the sake of minimizing the moving parts here, let me try with inlining just one dep

ya that's what I'm trying

just trying full.json because it seems to dep on jackson

which one did you choose?

ok

Seems to be closer to what you were reporting:

❯ lein with-profile +plugin.mranderson/config test
Compiling full.aws.core
Compiling full.aws.dynamo.kvstore
Compiling full.aws.s3
Compiling full.aws.sqs
Compiling mrandersonf3f583ec.camel-snake-kebab.v0v4v0.camel-snake-kebab.core
Compiling mrandersonf3f583ec.camel-snake-kebab.v0v4v0.camel-snake-kebab.extras
Compiling mrandersonf3f583ec.camel-snake-kebab.v0v4v0.camel-snake-kebab.internals.alter-name
Compiling mrandersonf3f583ec.camel-snake-kebab.v0v4v0.camel-snake-kebab.internals.macros
Compiling mrandersonf3f583ec.camel-snake-kebab.v0v4v0.camel-snake-kebab.internals.misc
Compiling mrandersonf3f583ec.camel-snake-kebab.v0v4v0.camel-snake-kebab.internals.string-separator
Compiling mrandersonf3f583ec.camelsnake.v0v10v0.camelsnake.core
Syntax error macroexpanding at (core.clj:18:1).
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:387).
fullaws100SNAPSHOT.fullaws100SNAPSHOT.camelsnake.Converter

Full report at:
/tmp/clojure-6394218391740156539.edn
Compilation failed: Subprocess failed (exit code: 1)
Error encountered performing task 'test' with profile(s): 'base,system,user,provided,dev,config'
Compilation failed: Subprocess failed (exit code: 1)

Compiling mranderson5b5c66f9.javaclasspath.v0v2v3.clojure.java.classpath
Compiling mranderson5b5c66f9.fulljson.v0v12v0.full.json
Syntax error macroexpanding at (factory.clj:1:1).
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
fullaws100SNAPSHOT.com.fasterxml.jackson.core.JsonFactory

we both get ClassNotFoundException but for different classes, i assume because of parallelism going on?

maybe? here's my lein version, what's yours?

❯ lein --version
Leiningen 2.9.10 on Java 1.8.0_352 OpenJDK 64-Bit Server VM

Leiningen 2.8.3 on Java 1.8.0_191 Java HotSpot(TM) 64-Bit Server VM

Hmm... any reason you are running an old version of lein? Can you bump to current?

i’ve tried at some point planting lein script with updated version in the repo and used that, didn’t help

let me update now

ok, i’m at:

Leiningen 2.10.0 on Java 1.8.0_191 Java HotSpot(TM) 64-Bit Server VM

Compiling mranderson5b5c66f9.camelsnake.v0v10v0.camelsnake.core
Syntax error macroexpanding at (core.clj:18:1).
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
fullaws100SNAPSHOT.fullaws100SNAPSHOT.camelsnake.Converter

same result as yours

oh, ok, this is the issue where is duplicates the prefix 2 or more times on many sources

(fullaws100SNAPSHOT\.){2,} => fullaws100SNAPSHOT.

now i’m back at:

Syntax error macroexpanding at (factory.clj:1:1).
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:382).
fullaws100SNAPSHOT.com.fasterxml.jackson.core.JsonFactory

under target/srcdeps/fullaws100SNAPSHOT/com/fasterxml/jackson i don’t have “core” only “dateformat”

i really appreciate you taking the time to investigate by the way 🙏

my pleasure

I am not sure how many problems we are dealing with here yet. I think dealing with one minimal reproducible problem at a time is the only way to proceed. For me, a first exploration might be attempting to create a minimal repo of the weird looking fullaws100SNAPSHOT.fullaws100SNAPSHOT.camelsnake.Converter reference.

sure, i agree

Hmm... I wonder if :aot :all is problematic? Frankly don't know at this point.

yes!

without :aot it seems to compile

hmmm… under com/fasterxml/jackson i still don’t see “core”… wonder why

question, now that the dep full.json is shaded (hopefully correctly) do i need to manually update references to full.json in my code to point to mrandersona1575b6b.full... ?

but without :aot :all are things loading/compiling when launching test on a project without tests?

no, I don't think you should need to update references to full.json , the inlining should take care of renaming the inlined ns and any references from your code to that inlined ns.

but it doesn’t, the code in full.aws/sqs.clj still requires [full.json]

Hmm... I'll run it here and see what I can see.

Well, something seems amiss... I think an sqs.clj should be generated under target/ with the reference changed... I'll have a deeper look (probably) tomorrow.

thanks!