Fork me on GitHub
#datomic
<
2020-10-29
>
Matheus Moreira12:10:00

hello! today i notice a weird interaction between datomic client api and djblue/portal (https://github.com/djblue/portal): when this last one is not on the classpath, then i can obtain a connection to my (local, datomic pro) database; when portal is on the classpath, connecting to the database fails with the following error:

Exception in thread "async-dispatch-1" java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/msgpack/MessagePack
	at com.cognitect.transit.TransitFactory.writer(TransitFactory.java:104)
	at cognitect.transit$writer.invokeStatic(transit.clj:161)
	at cognitect.transit$writer.invoke(transit.clj:139)
	at $marshal.invokeStatic(io.clj:48)
	at $marshal.invoke(io.clj:38)
	at $client_req__GT_http_req.invokeStatic(io.clj:76)
	at $client_req__GT_http_req.invoke(io.clj:73)
	at datomic.client.impl.shared.Client._async_op(shared.clj:398)
	at datomic.client.impl.shared.Client$fn__34578$state_machine__5717__auto____34593$fn__34595.invoke(shared.clj:423)
	at datomic.client.impl.shared.Client$fn__34578$state_machine__5717__auto____34593.invoke(shared.clj:422)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
	at datomic.client.impl.shared.Client$fn__34578.invoke(shared.clj:422)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at clojure.core.async.impl.concurrent$counted_thread_factory$reify__469$fn__470.invoke(concurrent.clj:29)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NoClassDefFoundError: org/msgpack/MessagePack
	at com.cognitect.transit.impl.WriterFactory.getMsgpackInstance(WriterFactory.java:77)
	at com.cognitect.transit.TransitFactory.writer(TransitFactory.java:95)
	... 20 more
Caused by: java.lang.ClassNotFoundException: org.msgpack.MessagePack
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 22 more

Matheus Moreira12:10:45

i noticed that portal has com.cognitect/- transit-cljs, transit-js, transit-clj, and transit-java as dependencies. why would these dependencies interfere with datomic obtaining a connection?

Matheus Moreira12:10:25

(there are other dependencies but i don’t believe they have anything to do with the case: cheshire 5.10.0 and http-kit 2.5.0).

alexmiller12:10:12

a) what are you using to make the classpath b) if clj, what version? (prob best to upgrade to latest if you haven’t) c) please provide the set of project deps that repro this

Matheus Moreira12:10:22

i am using clojure tools (clojure 1.10.1) and i open a repl using cider-jack-in-clj. then i start my system via integrant.repl.

Matheus Moreira12:10:58

this is my deps.edn. when connecting to the repl, i add -A:dev to the clj command.

favila13:10:52

Portal requires transit but excludes msgpack

Matheus Moreira13:10:12

and datomic requires msgpack?

Matheus Moreira13:10:05

if this is the case, is it a classpath resolution conflict/problem, i.e. msgpack should be in the final classpath because datomic requires it, even if portal excludes it?

alexmiller13:10:30

what version of the clojure tools? clj -Sdescribe

alexmiller13:10:06

there were issues with this kind of scenario that were fixed a few versions ago

alexmiller13:10:43

release info here https://clojure.org/releases/tools - latest is 1.10.1.727

favila13:10:38

The client apparently requires msgpack but doesn’t depend on it directly, expecting it via transit

favila13:10:52

Portal uses transit but not the msgpack encoding option, so it doesn’t want to bring it in

favila13:10:57

Datomic client uses transit with msgpack encoding but doesn’t require it directly

favila13:10:04

I wonder what maven would compute in this case

alexmiller13:10:31

I'm stepping away till this afternoon but I'd be happy to look at this in depth when I get back. My recommendation would be to move to latest clj if you haven't already as there have been fixes in this area.

Matheus Moreira14:10:39

thanks, @U064X3EF3 and @U09R86PA4. i’ll update clj tools if mine is not up-to-date.

favila14:10:01

(Sorry if I’m confusing, slack must have barfed on my messages because they’re all out of order and 20 min late)

😂 3
Matheus Moreira14:10:52

@U064X3EF3 fyi i updated clj tools and the error still happens.

alexmiller18:10:57

I do actually see msgpack on the classpath with these deps. Can you run clj -Sforce -Stree -A:dev and grep the output for msgpack? force will force recomputing the classpath - it's possible you are seeing an older cached classpath

alexmiller18:10:17

also if you're still seeing it after that, do clj -Strace -A:dev and attach the trace.edn file it emits here

Matheus Moreira11:11:45

clj -Sforce -Stree -A:dev returns nothing. djblue/portal was commented out in deps.edn, maybe that is why you see it in your output.

Matheus Moreira11:11:33

@U064X3EF3 sorry for the delay in my reply…

joshkh19:10:18

when untruthifying™ a boolean value of a schema attribute, is there an advantage to choosing one method over the other?

[:db/retract :some/ident :db/isComponent true]
vs
[:db/add :some/ident :db/isComponent false]
just curious

Lennart Buit19:10:15

fwiw, they are not equivalent, right? The first removes the fact about being a component altogether, and the second asserts it as false

joshkh21:10:29

yup. in this case the resulting behaviours are the same (:some/ident is no longer a component), but the resulting annotations of the ident are different (false vs missing). i'm just wondering why someone might choose one over the other.

Michael Stokley20:10:38

do folks compose pull patterns? suppose i have an entity and for one use case, i need a set of attrs; for another, i need a non-overlapping different set of attrs. since it's all data, maybe they can be defined separately but then composed so i can make one db call instead of n

steveb8n03:11:02

@michael740 I do this a lot. I use the apply-template fn in clojure core to inject pull vectors into other pull vectors or queries. it’s simple and it works well.

🙏 3
steveb8n03:11:37

even better if you match a transform fn for the results to each pull expr. then you can compose them to process the results as well.

Michael Stokley23:11:46

do you mean apply-template in clojure.template?

steveb8n21:11:40

sorry, yes, that’s the one. in my case, I generate the pull expressions and post query transform fns from a domain model. something like https://github.com/stevebuik/clj-code-gen-hodur

Michael Stokley20:10:28

i probably want to use sets instead of vectors in the initial representation? merge those, then swap the sets for vectors before use with datomic

Michael Stokley20:10:18

handling the vector syntax vs the map syntax might be tricky

kenny20:10:38

We go pull pattern -> ast -> merge -> pull pattern.

kenny20:10:56

The merge is typically very simple since it’s in the ast format.

Michael Stokley20:10:33

this looks cool as all get out

kenny20:10:30

We also use pathom so this is a very natural lib for us to use. You could probably write a smaller version to just do what you need if you don’t want to bring on a new dep.