Fork me on GitHub
#polylith
<
2021-09-29
>
Travis15:09:21

Hi guys, hopefully i can get some help. This could be a dumb problem on our part and I am not an expert in tools deps but we have several components that have dependencies on kafka-streams and jackdaw ( clojure kafka lib ). For what ever reason when we try to evaluate the files using them we get a class not found on the kafka dependency. Every command I run seems to indicate the the dep is being resolved just fine but not when run through the repl.

Travis15:09:29

Not sure what might be going on here

Travis15:09:58

it also seems if we put the deps in the dev alias at the root of the project it works

seancorfield16:09:47

How is your dev alias referencing the components that depend on Kafka etc?

seancorfield16:09:19

And how are those components declaring their dependency on Kafka?

Travis16:09:27

in the component its self we have this

{:paths ["src" "resources"]
 :deps {fundingcircle/jackdaw     {:mvn/version "0.8.0"}
        com.stuartsierra/component {:mvn/version "1.0.0"}
        org.apache.kafka/kafka-clients {:mvn/version "2.8.1"}
        org.apache.kafka/kafka-streams {:mvn/version "2.8.1"}}
 :aliases {:test {:extra-paths ["test"]
                  :extra-deps {}}}}

Travis16:09:43

in the root deps dev alias we add the path to the components under extra-paths and also a reference to it in :extra-deps

Travis16:09:52

also fyi we are using Calva mostly

seancorfield16:09:20

Ah, use :local/root in :extra-deps instead of paths in :extra-paths. That should solve it.

seancorfield16:09:50

Ours at work:

:dev ; Polylith "everything"
  {:extra-deps {;; bases
                poly/admin {:local/root "bases/admin"}
                poly/artifact-uploader-cli {:local/root "bases/artifact-uploader-cli"}
                poly/batch-jobs {:local/root "bases/batch-jobs"}
                poly/frankieee-cli {:local/root "bases/frankieee-cli"}
                poly/preview-web {:local/root "bases/preview-web"}
                ;; components
                poly/affiliate-link {:local/root "components/affiliate-link"}
                poly/artifact-uploader {:local/root "components/artifact-uploader"}
                poly/billing-machine {:local/root "components/billing-machine"}
                poly/billing-sdk {:local/root "components/billing-sdk"}
...

seancorfield16:09:00

Then you only need deps that are not declared in the components, i.e., stuff that would normally go in a projects deps.edn file. For us that's JDBC libraries and logging libraries, and Clojure itself:

;; and 3rd party libs specific to projects/required for pinning:
                mysql/mysql-connector-java {:mvn/version "8.0.22"}
                ;; use log4j 2.x:
                org.apache.logging.log4j/log4j-api {:mvn/version "2.14.1"}
                ;; bridge into log4j:
                org.apache.logging.log4j/log4j-1.2-api {:mvn/version "2.14.1"}
                org.apache.logging.log4j/log4j-jcl {:mvn/version "2.14.1"}
                org.apache.logging.log4j/log4j-jul {:mvn/version "2.14.1"}
                org.apache.logging.log4j/log4j-slf4j-impl {:mvn/version "2.14.1"}
                ;; ensure correct Clojure version for subprocess tests:
                org.clojure/clojure {:mvn/version "1.11.0-alpha2"}}

seancorfield16:09:04

If you update deps in a component, you'll need -Sforce when you start your REPL since transitive deps can go "stale" with :local/root deps (known issue with t.d.a).

seancorfield16:09:07

Here's how I start my REPL at work with Polylith and our legacy code available:

(! 513)-> SOCKET_REPL_PORT=5000 clojure -Sforce -M:rebel:portal:everything:dev:test:runner:build:dev/repl
:everything and :runner are the equivalent of :dev and :test respectively for our legacy code.

seancorfield16:09:56

:build is for our build.clj stuff so I can work on that script (and run tasks) in my editor/REPL. The rest come from my dot-clojure repo.

Travis17:09:56

ok, just to be clear in our main deps.edn file we have this

:aliases  {:dev {:extra-paths [  "components/kstream/src"
                                "components/kstream/resources" .....
and
extra-deps {org.clojure/clojure          {:mvn/version "1.10.3"}
                               org.clojure/tools.deps.alpha {:mvn/version "0.12.1003"}
                               poly/kstream               {:local/root "components/kstream"}

Travis17:09:09

we are putting a reference in both spots

seancorfield17:09:08

No, remove the paths versions.

seancorfield17:09:23

You only need the :local/root deps versions.

seancorfield17:09:43

That way :dev is more like a "project" (which is how it should be). Only Cursive has a problem with :local/root deps and it's just that it doesn't recognize them as source modules. I use VS Code and it's fine with just :local/root deps in :dev.

Travis17:09:47

ok, isn't the other needed if someone is using cursive ? Maybe i am confused on that

seancorfield17:09:52

You still use paths in :test.

seancorfield17:09:10

You have to cater for Cursive users as well?

Travis17:09:23

We have one, lol

seancorfield17:09:08

Ugh! Sorry. Then, yeah, maybe you will need to duplicate the components in as paths as well. You should go register a vote on the Cursive issue about that 🙂

Travis17:09:24

lol, will do

Travis21:09:43

ugh, still no luck on this dep issue

seancorfield00:09:57

Did you use -Sforce when starting your REPL? (or you could just remove the .cpcache folder from the workspace root)

tengstrand03:09:15

If you execute the libscommand @U096GDTLL, does everything looks right for the dev column?

Travis14:09:31

@U04V70XH6 Yeah, i tried Sforce and I did actually delete cpcache as well. @U1G0HH87L Yeah I can see those exact deps with X under the dev column

Travis14:09:33

and we have components with other external libs and those are working just fine. Just something about the ones requiring jackdaw/kafka

Travis14:09:02

Also its finding jackdaw just fine but getting class not found on the kafka-streams java lib from jackdaw

seancorfield14:09:52

@U096GDTLL Ah, if it is just a specific library, I would experiment in a standalone, non-Polylith repo and see if you can repro.

Travis14:09:25

it works fine from a standalone, We are actually port a working app into poly

Travis14:09:28

so far the only way i have managed to get it working in poly is include those specific kafka deps in the dev alias

tengstrand15:09:08

If you specify bricks in ./deps.edn as :extra-paths then you also have to specify all the libraries in the :extra-deps section for the :dev alias. This is how it’s solved in the polylith’s https://github.com/polyfy/polylith/blob/master/deps.edn (because I use Cursive). So maybe everything is as good as it can be for now, and we only have to wait till Cursive adds support for :local/root in the REPL to improve the situation in dev.

seancorfield15:09:33

We use :extra-deps and :local/root in our :dev alias for bricks and we do not need to specify any of the additional library dependencies there. However, it sounds like it is something specific to the Kafka libs since it is only those that @U096GDTLL is having a problem with? Can you share exactly what those dependencies are, and also what class name I could try importing to see whether it works? I'm very curious now about why this is happening and what is special about those libs...

Travis18:09:17

Our component deps look like this

{:paths ["src" "resources"]
 :deps {fundingcircle/jackdaw     {:mvn/version "0.8.0"}
        com.stuartsierra/component {:mvn/version "1.0.0"}
        org.apache.kafka/kafka-clients {:mvn/version "2.8.1"}
        org.apache.kafka/kafka-streams {:mvn/version "2.8.1"}}
 :aliases {:test {:extra-paths ["test"]
                  :extra-deps {}}}}

Travis18:09:20

we add a direct dep on kafka to essentially override the older version in jackdaw ( 2.8.0 ). Problem happens either way.

Travis18:09:08

; Evaluating file: core.clj
; Syntax error (ClassNotFoundException) compiling at (jackdaw/streams/interop.clj:1:1).
; org.apache.kafka.streams.kstream.Serialized
; Evaluation of file core.clj failed: class clojure.lang.Compiler$CompilerException
This is the error message we have been consistently getting. Again I would not rule out something stupid on our part, lol

seancorfield18:09:45

I can repro that in a minimal (non-Polylith) repo.

seancorfield18:09:51

(! 855)-> tree .
.
|____deps.edn
|____subproject
| |____deps.edn

(! 856)-> cat deps.edn 
{:aliases
 {:dev
  {:extra-deps ; this was missing and that caused the error -- after adding this, it worked
   {example/subproject {:local/root "subproject"}}}}}

(! 857)-> cat subproject/deps.edn 
{:paths ["src" "resources"]
 :deps {org.apache.kafka/kafka-clients {:mvn/version "2.8.1"}
        org.apache.kafka/kafka-streams {:mvn/version "2.8.1"}}
 :aliases {:test {:extra-paths ["test"]
                  :extra-deps {}}}}

(! 858)-> clj -A:dev 
Clojure 1.10.3
user=> (import org.apache.kafka.streams.kstream.Serialized)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:440).
org.apache.kafka.streams.kstream.Serialized
user=> 
(edited to add note about why it failed for me)

seancorfield18:09:26

If I run a REPL inside subproject, it works as expected.

Travis18:09:40

interesting, kind of what we are seeing

seancorfield18:09:37

Oh, hang on. I'm doing something stupid.

seancorfield18:09:45

I fixed my :dev alias -- and now it works as expected...

seancorfield18:09:23

{:aliases
 {:dev
  {:extra-deps
   {example/subproject {:local/root "subproject"}}}}}
I was missing the :extra-deps part so it wasn't pulling in any of the subproject stuff.

seancorfield19:09:56

so you have that deps.edn you posted in components/something/ and in your :dev alias in the workspace-level deps.edn you have poly/something {:local/root "components/something"} in :extra-deps in your :dev alias?

seancorfield19:09:14

and then you start your REPL as clj -A:dev (and maybe other options)

Travis19:09:58

just using calva to start the repl

Travis19:09:24

running a command like this

clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version,"0.8.3"},cider/cider-nrepl {:mvn/version,"0.26.0"}}}' -A:dev:+user-command-processor:test -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"

seancorfield19:09:50

Just use a plain REPL, clj, without all the nREPL nonsense in play. See if it works there -- like I tested it above.

seancorfield19:09:28

Using clojure -Stree -A:dev:+user-command-processor:test would also be a useful check, to see if the Kafka dependencies are present.

Travis19:09:52

poly/kstream projects/management/components/kstream
  . fundingcircle/jackdaw 0.8.0
  . org.apache.kafka/kafka-clients 2.8.1
  . org.apache.kafka/kafka-streams 2.8.1
  . com.stuartsierra/component 1.0.0
  . com.taoensso/timbre 5.1.2
  . com.fzakaria/slf4j-timbre 0.3.21

seancorfield19:09:45

And in a plain clj REPL, can you import the class like I did above?

seancorfield19:09:37

That path looks really odd: projects/management/components/kstream

seancorfield19:09:15

I would expect components/kstream to be the relative path there (from the :dev alias :extra-deps).

seancorfield19:09:00

I can understand there being a projects/management folder, with a deps.edn, but that shouldn't be what :dev depends on.

Travis19:09:31

it listed my full path to the component i just stripped it down when i pasted it

Travis19:09:11

poly/kstream /home/camechis/projects/management/components/kstream
  . fundingcircle/jackdaw 0.8.0
  . org.apache.kafka/kafka-clients 2.8.1
  . org.apache.kafka/kafka-streams 2.8.1
  . com.stuartsierra/component 1.0.0
  . com.taoensso/timbre 5.1.2
  . com.fzakaria/slf4j-timbre 0.3.21

Travis19:09:20

I accidently got rid of the beginning path

Travis19:09:31

management is the root of the workspace. I did try what you did above and got the class not found issue

Travis19:09:10

so strange

seancorfield19:09:06

That -Stree output above shows that the dependencies are there so you should be able to import that class...

Travis19:09:23

clj -A:dev
Clojure 1.10.3
user=> (import org.apache.kafka.streams.kstream.Serialized)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:471).
org.apache.kafka.streams.kstream.Serialized

seancorfield20:09:56

and yet clj -Stree -A:dev shows the dependency is present? That is extremely strange.

Travis22:09:36

@U04V70XH6 I appreciate all the help given by the way.

seancorfield23:09:37

NP. I really want to solve this one -- or at least find out why it is happening to you!