How is your dev alias referencing the components that depend on Kafka etc?

And how are those components declaring their dependency on Kafka?

in the component its self we have this

{:paths ["src" "resources"]
 :deps {fundingcircle/jackdaw     {:mvn/version "0.8.0"}
        com.stuartsierra/component {:mvn/version "1.0.0"}
        org.apache.kafka/kafka-clients {:mvn/version "2.8.1"}
        org.apache.kafka/kafka-streams {:mvn/version "2.8.1"}}
 :aliases {:test {:extra-paths ["test"]
                  :extra-deps {}}}}

in the root deps dev alias we add the path to the components under extra-paths and also a reference to it in :extra-deps

also fyi we are using Calva mostly

Ah, use :local/root in :extra-deps instead of paths in :extra-paths. That should solve it.

Ours at work:

:dev ; Polylith "everything"
  {:extra-deps {;; bases
                poly/admin {:local/root "bases/admin"}
                poly/artifact-uploader-cli {:local/root "bases/artifact-uploader-cli"}
                poly/batch-jobs {:local/root "bases/batch-jobs"}
                poly/frankieee-cli {:local/root "bases/frankieee-cli"}
                poly/preview-web {:local/root "bases/preview-web"}
                ;; components
                poly/affiliate-link {:local/root "components/affiliate-link"}
                poly/artifact-uploader {:local/root "components/artifact-uploader"}
                poly/billing-machine {:local/root "components/billing-machine"}
                poly/billing-sdk {:local/root "components/billing-sdk"}
...

Then you only need deps that are not declared in the components, i.e., stuff that would normally go in a projects deps.edn file. For us that's JDBC libraries and logging libraries, and Clojure itself:

;; and 3rd party libs specific to projects/required for pinning:
                mysql/mysql-connector-java {:mvn/version "8.0.22"}
                ;; use log4j 2.x:
                org.apache.logging.log4j/log4j-api {:mvn/version "2.14.1"}
                ;; bridge into log4j:
                org.apache.logging.log4j/log4j-1.2-api {:mvn/version "2.14.1"}
                org.apache.logging.log4j/log4j-jcl {:mvn/version "2.14.1"}
                org.apache.logging.log4j/log4j-jul {:mvn/version "2.14.1"}
                org.apache.logging.log4j/log4j-slf4j-impl {:mvn/version "2.14.1"}
                ;; ensure correct Clojure version for subprocess tests:
                org.clojure/clojure {:mvn/version "1.11.0-alpha2"}}

If you update deps in a component, you'll need -Sforce when you start your REPL since transitive deps can go "stale" with :local/root deps (known issue with t.d.a).

Here's how I start my REPL at work with Polylith and our legacy code available:

(! 513)-> SOCKET_REPL_PORT=5000 clojure -Sforce -M:rebel:portal:everything:dev:test:runner:build:dev/repl

:build is for our build.clj stuff so I can work on that script (and run tasks) in my editor/REPL. The rest come from my dot-clojure repo.

ok, just to be clear in our main deps.edn file we have this

:aliases  {:dev {:extra-paths [  "components/kstream/src"
                                "components/kstream/resources" .....

extra-deps {org.clojure/clojure          {:mvn/version "1.10.3"}
                               org.clojure/tools.deps.alpha {:mvn/version "0.12.1003"}
                               poly/kstream               {:local/root "components/kstream"}

we are putting a reference in both spots

No, remove the paths versions.

You only need the :local/root deps versions.

That way :dev is more like a "project" (which is how it should be). Only Cursive has a problem with :local/root deps and it's just that it doesn't recognize them as source modules. I use VS Code and it's fine with just :local/root deps in :dev.

ok, isn't the other needed if someone is using cursive ? Maybe i am confused on that

You still use paths in :test.

You have to cater for Cursive users as well?

We have one, lol

Ugh! Sorry. Then, yeah, maybe you will need to duplicate the components in as paths as well. You should go register a vote on the Cursive issue about that 🙂

lol, will do

https://github.com/cursive-ide/cursive/issues/2554

ugh, still no luck on this dep issue

Did you use -Sforce when starting your REPL? (or you could just remove the .cpcache folder from the workspace root)

If you execute the libscommand @U096GDTLL, does everything looks right for the dev column?

@U04V70XH6 Yeah, i tried Sforce and I did actually delete cpcache as well. @U1G0HH87L Yeah I can see those exact deps with X under the dev column

and we have components with other external libs and those are working just fine. Just something about the ones requiring jackdaw/kafka

Also its finding jackdaw just fine but getting class not found on the kafka-streams java lib from jackdaw

@U096GDTLL Ah, if it is just a specific library, I would experiment in a standalone, non-Polylith repo and see if you can repro.

it works fine from a standalone, We are actually port a working app into poly

so far the only way i have managed to get it working in poly is include those specific kafka deps in the dev alias

If you specify bricks in ./deps.edn as :extra-paths then you also have to specify all the libraries in the :extra-deps section for the :dev alias. This is how it’s solved in the polylith’s https://github.com/polyfy/polylith/blob/master/deps.edn (because I use Cursive). So maybe everything is as good as it can be for now, and we only have to wait till Cursive adds support for :local/root in the REPL to improve the situation in dev.

We use :extra-deps and :local/root in our :dev alias for bricks and we do not need to specify any of the additional library dependencies there. However, it sounds like it is something specific to the Kafka libs since it is only those that @U096GDTLL is having a problem with? Can you share exactly what those dependencies are, and also what class name I could try importing to see whether it works? I'm very curious now about why this is happening and what is special about those libs...

Our component deps look like this

{:paths ["src" "resources"]
 :deps {fundingcircle/jackdaw     {:mvn/version "0.8.0"}
        com.stuartsierra/component {:mvn/version "1.0.0"}
        org.apache.kafka/kafka-clients {:mvn/version "2.8.1"}
        org.apache.kafka/kafka-streams {:mvn/version "2.8.1"}}
 :aliases {:test {:extra-paths ["test"]
                  :extra-deps {}}}}

we add a direct dep on kafka to essentially override the older version in jackdaw ( 2.8.0 ). Problem happens either way.

; Evaluating file: core.clj
; Syntax error (ClassNotFoundException) compiling at (jackdaw/streams/interop.clj:1:1).
; org.apache.kafka.streams.kstream.Serialized
; Evaluation of file core.clj failed: class clojure.lang.Compiler$CompilerException

I can repro that in a minimal (non-Polylith) repo.

(! 855)-> tree .
.
|____deps.edn
|____subproject
| |____deps.edn

(! 856)-> cat deps.edn 
{:aliases
 {:dev
  {:extra-deps ; this was missing and that caused the error -- after adding this, it worked
   {example/subproject {:local/root "subproject"}}}}}

(! 857)-> cat subproject/deps.edn 
{:paths ["src" "resources"]
 :deps {org.apache.kafka/kafka-clients {:mvn/version "2.8.1"}
        org.apache.kafka/kafka-streams {:mvn/version "2.8.1"}}
 :aliases {:test {:extra-paths ["test"]
                  :extra-deps {}}}}

(! 858)-> clj -A:dev 
Clojure 1.10.3
user=> (import org.apache.kafka.streams.kstream.Serialized)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:440).
org.apache.kafka.streams.kstream.Serialized
user=> 

If I run a REPL inside subproject, it works as expected.

interesting, kind of what we are seeing

Oh, hang on. I'm doing something stupid.

I fixed my :dev alias -- and now it works as expected...

hmmm

{:aliases
 {:dev
  {:extra-deps
   {example/subproject {:local/root "subproject"}}}}}

so you have that deps.edn you posted in components/something/ and in your :dev alias in the workspace-level deps.edn you have poly/something {:local/root "components/something"} in :extra-deps in your :dev alias?

indeed

and then you start your REPL as clj -A:dev (and maybe other options)

just using calva to start the repl

running a command like this

clojure -Sdeps '{:deps {nrepl/nrepl {:mvn/version,"0.8.3"},cider/cider-nrepl {:mvn/version,"0.26.0"}}}' -A:dev:+user-command-processor:test -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"

Just use a plain REPL, clj, without all the nREPL nonsense in play. See if it works there -- like I tested it above.

Using clojure -Stree -A:dev:+user-command-processor:test would also be a useful check, to see if the Kafka dependencies are present.

poly/kstream projects/management/components/kstream
  . fundingcircle/jackdaw 0.8.0
  . org.apache.kafka/kafka-clients 2.8.1
  . org.apache.kafka/kafka-streams 2.8.1
  . com.stuartsierra/component 1.0.0
  . com.taoensso/timbre 5.1.2
  . com.fzakaria/slf4j-timbre 0.3.21

And in a plain clj REPL, can you import the class like I did above?

That path looks really odd: projects/management/components/kstream

I would expect components/kstream to be the relative path there (from the :dev alias :extra-deps).

I can understand there being a projects/management folder, with a deps.edn, but that shouldn't be what :dev depends on.

it listed my full path to the component i just stripped it down when i pasted it

poly/kstream /home/camechis/projects/management/components/kstream
  . fundingcircle/jackdaw 0.8.0
  . org.apache.kafka/kafka-clients 2.8.1
  . org.apache.kafka/kafka-streams 2.8.1
  . com.stuartsierra/component 1.0.0
  . com.taoensso/timbre 5.1.2
  . com.fzakaria/slf4j-timbre 0.3.21

I accidently got rid of the beginning path

management is the root of the workspace. I did try what you did above and got the class not found issue

so strange

That -Stree output above shows that the dependencies are there so you should be able to import that class...

clj -A:dev
Clojure 1.10.3
user=> (import org.apache.kafka.streams.kstream.Serialized)
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:471).
org.apache.kafka.streams.kstream.Serialized

lol

and yet clj -Stree -A:dev shows the dependency is present? That is extremely strange.

indeed

@U04V70XH6 I appreciate all the help given by the way.

NP. I really want to solve this one -- or at least find out why it is happening to you!