Fork me on GitHub
#tools-deps
<
2021-12-02
>
jeff.terrell02:12:36

I think I must be missing something obvious here. I have the "JPam" jar on my classpath:

$ clj -Spath | perl -pe 's/:/\n/g' | grep jpam-1.1
/home/jeff/.m2/repository/net/sf/jpam/jpam/1.1/jpam-1.1.jar
...and in that jar is a net.sf.jpam.Pam class:
$ jar tvf /home/jeff/.m2/repository/net/sf/jpam/jpam/1.1/jpam-1.1.jar | grep Pam.class
  3520 Thu Apr 16 11:30:28 EDT 2009 net/sf/jpam/Pam.class
...yet when I try to use it, I get a NoClassDefFoundError:
$ clj
Clojure 1.10.3
user=> (require '[clj-pam.auth :as auth] :reload-all)
nil
user=> (auth/simple-auth "jeff" "hello")
Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader/loadLibrary (ClassLoader.java:2429).
no jpam in java.library.path: /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
Relevant code (slightly simplified) is:
(ns clj-pam.auth
  (:import net.sf.jpam.Pam))

(defn simple-auth [username password]
  (.authenticateSuccessful (new Pam) username password))
Am I missing something obvious?

Alex Miller (Clojure team)02:12:33

Looks like jpam is a native lib

Alex Miller (Clojure team)02:12:10

That java.library.path is not classpath, that's native libs

jeff.terrell02:12:07

I do see the word "native" in places that suggest you're right, but I don't really know what that means with respect to the JVM finding what it needs to find.

jeff.terrell02:12:04

There is a dependency that has "native" in the name:

$ clj -Spath | perl -pe 's/:/\n/g' | grep native
/home/jeff/.m2/repository/org/clojars/canweriotnow/jpam-native-deps/1.1.1/jpam-native-deps-1.1.1.jar
...and that jar has native-looking things in it:
$ jar -tvf /home/jeff/.m2/repository/org/clojars/canweriotnow/jpam-native-deps/1.1.1/jpam-native-deps-1.1.1.jar
     0 Thu Mar 07 17:32:46 EST 2013 native/
     0 Thu Mar 07 17:32:04 EST 2013 native/linux/
     0 Thu Mar 07 17:36:30 EST 2013 native/linux/x86_64/
 33015 Fri Jun 08 01:19:18 EDT 2007 native/linux/x86_64/libjpam.so
     0 Thu Mar 07 17:36:56 EST 2013 native/linux/x86/
 29446 Sat Jun 09 03:32:24 EDT 2007 native/linux/x86/libjpam.so
     0 Thu Mar 07 17:32:26 EST 2013 native/macosx/
     0 Fri Mar 08 16:58:52 EST 2013 native/macosx/x86_64/
 14884 Fri Mar 08 16:58:52 EST 2013 native/macosx/x86_64/libjpam.jnilib
     0 Thu Mar 07 17:37:26 EST 2013 native/macosx/x86/
 45348 Sat Jun 09 03:50:24 EDT 2007 native/macosx/x86/libjpam.jnilib
How do I get Clojure/the JVM to find these native things?

phronmophobic03:12:21

usually, it can find the right native shared library in the jar based off the OS and architecture. what OS and architecture are you using?

jeff.terrell03:12:00

uname reports x86_64 GNU/Linux, and there is a native/linux/x86_64/libjpam.so file in the JAR. I would think it would be able to find it too, but apparently not?

phronmophobic03:12:59

oh weird. I know it sometimes checks a few places, but for my native library, the path in the jar is linux-x86-64/libMyLib.so . Actually, ignore this. The path I use is from JNA.

phronmophobic03:12:49

I know there are some setups and builds that will do fancy things like finding and extracting the jar for you.

phronmophobic03:12:51

Actually, I think the way to do it is to extract the shared library from the jar to somewhere on your library path.

phronmophobic03:12:58

fwiw, I've been using JNA via https://github.com/Chouser/clojure-jna/ to interface with c libraries and the c interface for many libs is better than the java wrappers.

jeff.terrell16:12:18

Yes, that's the library. Ah, that's very helpful, thanks! I can work with this. Just hadn't come across any native stuff before.

jeff.terrell02:12:13

Maybe I need to go learn more about what exactly "native" means in a Java context...

jeff.terrell03:12:34

Turns out, if I extract the relevant libjpam.so for my architecture and point the java.library.path property to the containing directory, it works. Example:

clj -J-Djava.library.path=.
Seems like there should be a better way. Oh well.

hiredman06:12:27

Many java projects that use native code do that automatically via a static initializer that writes the shared library to a temp file and loads it from there

jeff.terrell16:12:32

That's very helpful, thanks!

Mateusz Milian09:12:01

Hi, a question regarding dependencies: I have some local libs (jars), which I want to include into my classpath. I do not want to add them one by one (is too many). Is there any way to include all jars from the folder using deps.edn? Something like: {:deps {lib/lib {:local/root "local/lib/*}} I know, I can merge jars and create fatJar, but I'm wondering if there is a better option, similar to:

java -cp "local/lib/*" -jar app.jar

hiredman09:12:53

That kind of thing is problematic because clj tries to avoid launching multiple jvms by generating the classpath once, and caching it based on deps.edn contents

hiredman09:12:09

And wild cards are weird there

hiredman09:12:04

Assuming it is a fairly static list of jars, just write a clojure program to list them and generate a deps.edn containing them all

Mateusz Milian09:12:53

ok, thx. This will solve my issue.

pithyless13:12:56

When I've got an error like this: Error building classpath. Unable to compare versions for borkdude/sci (one is from git, another from maven); how can I find the projects that are bringing in the dependencies? Normally, I would use -Stree , but that fails in this case.

Alex Miller (Clojure team)13:12:45

Don't know that I have a good answer other than commenting stuff out in the deps

Alex Miller (Clojure team)13:12:20

You can also just add sci as a top level dep

Alex Miller (Clojure team)13:12:28

That overrides stuff below

Alex Miller (Clojure team)13:12:46

Would at least let you run tree

borkdude13:12:16

@pithyless fwiw SCI moved to babashka/sci now as well (to make matters more complicated ;))

😅 1
pithyless13:12:46

Interesting; adding it as a top-level dep allowed me to override it, but -Stree is no longer returning the other versions so still not sure which one was conflicting

pithyless13:12:00

But it did push me over the wall, thanks!

Joshua Suskalo15:12:38

This new license detector is really cool @alexmiller, thanks for the work on that! Do you think there would be value to adding a complementary command that would list out every dependency for which a license can't be found to make it easier to manually validate them?

Alex Miller (Clojure team)15:12:35

I was debating whether to put something in there (other than nothing) like "unknown" or "not detected" or something, wasn't sure if that was needed or not

Alex Miller (Clojure team)15:12:07

I mean you could :format :edn, slurp, and use filter to list those

Joshua Suskalo15:12:51

That's true, building a tool that does this that's external isn't hard. I just was thinking it might fit well in the existing tool. I think you could also work out a grep expression to do the same thing.

Joshua Suskalo15:12:17

So yeah, it's not necessarily a hard problem, it's just up to if you think it fits into the tooling.

Alex Miller (Clojure team)15:12:34

I guess I'd rather wait and see how big an issue it is

dpsutton15:12:30

we recently built this at metabase. It was right before our switch to deps.edn so it is still based on a classpath rather than a basis map. But it is open source so have at it: https://github.com/metabase/metabase/blob/master/bin/build-mb/src/build/licenses.clj

dpsutton15:12:09

looks in jars for license information. but i may have to revisit it since it doesn’t find it in Clojure’s own jar. But we have a backfill to list licenses for ones not detected. And in CI we have a test that fails if it finds any that it doesn’t not classify

Alex Miller (Clojure team)15:12:11

yeah, I don't think we want to do any of that, thanks. I suspect you're missing a lot of license info by not using a real pom model that would pull that from parent poms (which the deps one will find)

dpsutton15:12:16

Yeah. I was surprised to hear Clojure’s jar had a license information in it as we had to backfill it. So i’ll revisit that next week. I wasn’t suggesting deps.edn include this but for others who needed a bit more from licensing checks

Alex Miller (Clojure team)15:12:03

clojure has a parent pom, that has license data

dpsutton15:12:34

I see. I’ll check out how tools.deps uses those classes and adjust accordingly. I also go digging through for license files in the jar and it would be nice if i could abandon that

Alex Miller (Clojure team)15:12:51

actually, no it's in the published pom. is it not in the pom inside the jar?

dpsutton15:12:06

I think it is in the actual jar. i’m not sure why i miss it navigating through the xml

dpsutton15:12:44

I cheat and look for a pom adjacent to the jar and then dig in the jar itself. So i should see it in the published pom

Alex Miller (Clojure team)15:12:08

yeah, it's in the pom inside the jar

Alex Miller (Clojure team)15:12:21

so you should find it in either of those places

👍 1
Jeff Evans18:12:57

I found this old thread, but just to ask again, has anyone managed to implement something akin to Maven dependencyManagement in deps.edn? The scenario is that we have a number of plugins, each of which might have its own dependencies. But ultimately, these plugins (which are each an uberjar in their own right) are loaded dynamically into the root application, and therefore they all end up in the same classloader. If one plugin depends on a different version of Jackson than another, there could be a clash when both plugins are shoveled into the same classloader. In Maven-land, I would simply have a parent pom.xml that declares dependencyManagement and puts a Jackson version in there, and each child module can inherit from that pom. https://clojurians.slack.com/archives/C6QH853H8/p1595987442250200

mpenet19:12:10

you can achieve this with that tool : https://github.com/exoscale/deps-modules/

mpenet19:12:49

it's updating the deps.edn files from a version file

Jeff Evans19:12:16

thank you! will have a look at it

Alex Miller (Clojure team)18:12:22

there's not really anything akin to that yet. there are various workarounds.

Alex Miller (Clojure team)18:12:49

like generating your deps.edn or overloading parts of it at run time