A coworker approached this from another angle I hadn't thought of. What if you want to publish your library to clojars, but need a dependency which isn't in a maven repo? Do you give up or fork the dep and pray you don't have to go further down the stack? It's not hard to see how this encourages movement away from maven.

> What if you want to publish your library to clojars, but need a dependency which isn't in a maven repo? This is already true in the Java world if you s/clojars/maven central/

The current solution is usually to bring the dependency directly into your project as shaded source code.

(in both Java and Clojure worlds)

Thanks Sean. I didn't know that.

Also, Leiningen has at least one plugin that has supported git dependencies (separate from Maven-style deps) for many years: https://github.com/tobyhede/lein-git-deps

That puts me a little at ease

Seems like you can try and load the class file by its name (without .class and with . instead of /) and then watch for an exception:

$ pwd
/home/p-himik/tmp/java-classpath-test
$ cat Test.java
package aaa;

class Test {
    public static void main(String[] argv) {
        System.out.println("Hello");
    }
}
$ jshell -c .
|  Welcome to JShell -- Version 11.0.14
|  For an introduction type: /help intro

jshell> import java.net.URL;

jshell> URL[] urls = new URL[]{new URL("file:///home/p-himik/tmp/java-classpath-test/")};
urls ==> URL[1] { file:/home/p-himik/tmp/java-classpath-test/ }

jshell> import java.net.URLClassLoader;
  
jshell> URLClassLoader cl = new URLClassLoader(urls);
cl ==> java.net.URLClassLoader@533ddba

jshell> cl.loadClass("Test")
|  Exception java.lang.NoClassDefFoundError: aaa/Test (wrong name: Test)
|        at ClassLoader.defineClass1 (Native Method)
|        at ClassLoader.defineClass (ClassLoader.java:1017)
|        at SecureClassLoader.defineClass (SecureClassLoader.java:174)
|        at BuiltinClassLoader.defineClass (BuiltinClassLoader.java:800)
|        at BuiltinClassLoader.findClassOnClassPathOrNull (BuiltinClassLoader.java:698)
|        at BuiltinClassLoader.loadClassOrNull (BuiltinClassLoader.java:621)
|        at BuiltinClassLoader.loadClass (BuiltinClassLoader.java:579)
|        at ClassLoaders$AppClassLoader.loadClass (ClassLoaders.java:178)
|        at ClassLoader.loadClass (ClassLoader.java:576)
|        at ClassLoader.loadClass (ClassLoader.java:522)
|        at (#5:1)

jshell> 

As you can see, I never mentioned aaa to jshell - but it does know about it.

Forgot one command above - of course, I also ran javac Test.java.

Looking through JDK sources now - maybe there's a reasonable way to get to that aaa/Test without having to parse the exception.

@U2FRKM4TW clj-kondo should not be using any classloader stuff for this

it's supposed to be running in a graalvm native-image so you should just treat this as a random binary file without any relation to the classloader of the environment

Oh... makes sense.

I think ASM works that way

so maybe it's not so bad to use that, it seems to be a small library

found this: https://stackoverflow.com/questions/1649674/resolve-class-name-from-bytecode

@U037U6UBEAW helpful thank you!

you're welcome

excuse me I am a newbie

but here's a crude but working translation:

(defn get-class-name [path]
  (with-open [dis (java.io.DataInputStream. (io/input-stream path))]
    ;; skip first 8 bytes
    (.readLong dis)
    (let [constant-pool-count (dec (.readUnsignedShort dis))
          counter (atom 0)
          classes (atom {})
          strings (atom {})]
      (do
        (while (< @counter constant-pool-count)
          (do
            (case (.read dis)
              1 (swap! strings assoc @counter (.readUTF dis))
              5 (do (.readLong dis) (swap! counter inc))
              6 (do (.readLong dis) (swap! counter inc))
              7 (swap! classes assoc  @counter (.readShort dis))
              8 (.readShort dis)
              (.readInt dis))
            (swap! counter inc)))
        ;; skip access flags
        (.readShort dis)
        (clojure.string/join
         "."
         (butlast
          (clojure.string/split
           (get @strings (- (get @classes (.readUnsignedShort dis)) 2))
           #"/")))))))

@U037U6UBEAW cool! I'm getting a NPE here:

(prn (get @classes (.readUnsignedShort dis)))

a nil I mean

can you send me the class you're running it against?

I just tried it only with a half dozen of classes I found on my laptop

This is clojure.lang.PersistentVector.class

thanks!

I'll likely stick with ASM since I don't want to get a lot of problems with this :)

🙂

replace (get @strings (- (get @classes (.readUnsignedShort dis)) 2))

with (get @strings (- (get @classes (- (.readUnsignedShort dis) 1)) 1))

nice, it returns the package name now

awesome!

(it also works in babashka I noticed :)

cool!

I've not found a more elegant way because of the need to increment the counter inside the loop a couple of times

@U037U6UBEAW Have you accounted for this? https://stackoverflow.com/questions/1649674/resolve-class-name-from-bytecode/1650442#comment125596079_1650442

I swapped your impl to use volatiles now, should be a little faster

That comment makes me want to stick with ASM since I won't have to deal with stuff that I don't really know a lot about (parsing class files)

I think ASM is safer

still cool that you managed to port it. I'm going to keep that example somewhere

a new class format eventually will break that simple parser

I just wanted to practice a little bit 🙂

Double checked against https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html counter need to start from 1 indeed: > A constant_pool index is considered valid if it is greater than zero and less than constant_pool_count and so there's no need to decrement indices for retrieving correct values

also, .skipBytes could be a better semantic choice since we are effectively throwing away those values

The need to skip constant pool entries is also explained here https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.4

> In retrospect, making 8-byte constants take two constant pool entries was a poor choice.

so ends up with

5 (do (.skipBytes dis 8) (swap! counter inc))
6 (do (.skipBytes dis 8) (swap! counter inc))

Have a nice day!

@U037U6UBEAW Thanks a lot!

@U037U6UBEAW I converted the while and mutation to a loop and shared it here: https://gist.github.com/borkdude/d02dc3ff1d03d09351e768964983a46b

looks much better now, thanks 🙂

Sorry @U04V15CAJ but I wasn't happy until having correctly parsed at least all 17k classes in my home... First there was an error in the gist at line 19 where the counter must be incremented by 2 (unfortunately, "8-byte constants take up two entries in the constant_pool table of the class file"), and some tags were not considered (most prominently CONSTANT_MethodHandle_info that consumes 3 bytes and caused some quirks). Also made the counter start from 1 ("constant_pool index is considered valid if it is greater than zero and less than constant_pool_count") so now indices on table are correct and there's no more need to decrease them when retrieving values...

(defn class->package
  "Implementation by Marco Marini."
  [class-file]
  (with-open [dis (java.io.DataInputStream. (io/input-stream class-file))]
    ;; skip first 8 bytes
    (.skipBytes dis 8)
    (let [constant-pool-count (.readUnsignedShort dis)
          [strings classes]
          (loop [counter 1
                 classes {}
                 strings {}]
            (if (< counter constant-pool-count)
              (case (.read dis)
                1 (recur (inc counter) classes (assoc strings counter (.readUTF dis)))
                (5 6) (do (.skipBytes dis 8)
                          (recur (+ 2 counter) classes strings))
                7 (recur (inc counter) (assoc classes counter (.readUnsignedShort dis)) strings)
                (8 16 19 20) (do (.skipBytes dis 2)
                                 (recur (inc counter) classes strings))
                15 (do (.skipBytes dis 3) (recur (inc counter) classes strings))
                (do (.skipBytes dis 4)
                    (recur (inc counter) classes strings)))
              [strings classes]))]
      ;; skip access flags
      (.skipBytes dis 2)
      ;; (prn (get @classes (.readUnsignedShort dis)))
      (str/join
       "."
       (butlast
        (str/split
         (get strings (get classes (.readUnsignedShort dis)))
         #"/"))))))

Thanks again and enjoy the evening 🙂

Awesome, I'll update the gis!

Very early effort, likely to have bitrotted, I seem to recall running into another package that did the same thing but better but I can't recall where

There are snippets like https://gist.github.com/skeeto/3178747 floating around

I also could have sworn I had a gist with some clojure code for parsing class files, but I couldn't find it

@UKFSJSM38 and I are now looking for a way to embed CFR (java decompiler) in clojure-lsp so it will automatically do it for every editor

After reading @U054W022G explanation of ns related function in his book, I now understand that the proper way of writing what I was trying to write is:

(defn foo []
  (create-ns 'bar)
  (intern 'bar 'baz #(println "baz")))

Also note that ns is a macro, not a function. Meaning the ns call gets expanded when you define the function, not when you run the function. There's also in-ns, which might be doing what you're expecting: https://clojuredocs.org/clojure.core/in-ns I thought it was a function, but I can't find its source in clojure.core.

EDIT: see below for a very minimal example (protocol doesn't matter here) Let's create projecta with two files: projecta.protocols

(ns projecta.protocols
  (:refer-clojure :exclude [abs]))

(defprotocol PAbs
  (abs [object])) 

(ns projecta.core
  (:require [projecta.protocols :as p]))

(deftype SomeType [^double v]
  Object
  (toString [_] (str v))
  p/PAbs
  (abs [_] (SomeType. (clojure.core/abs v))))

(ns-unmap *ns* 'abs)
(defn abs [v] (p/abs v))

Let's create projectb with lein project.clj

(defproject projectb "0.1.0-SNAPSHOT"
  :prep-tasks [["compile" "projecta.core"] "javac"]
  :dependencies [[org.clojure/clojure "1.11.0"]
                 [projecta "0.1.0-SNAPSHOT"]])

projecta.core/abs is not defined 😕

$ lein repl
Compiling projecta.core
nREPL server started on port 43847 on host 127.0.0.1 - 
REPL-y 0.5.1, nREPL 0.8.3
Clojure 1.11.0
OpenJDK 64-Bit Server VM 18-ea+36-Ubuntu-1
[...]
user=> (require '[projecta.core :as pa])
nil
user=> (def a (pa/->SomeType -3))
#'user/a
user=> (pa/abs a)
Syntax error compiling at (/tmp/form-init18231071095412228753.clj:1:1).
No such var: pa/abs

When projecta.core is constructed as following:

(ns projecta.core
  (:refer-clojure :exclude [abs])
  (:require [projecta.protocols :as p]))

(deftype SomeType [^double v]
  Object
  (toString [_] (str v))
  p/PAbs
  (abs [_] (SomeType. (clojure.core/abs v))))

(defn abs [v] (p/abs v))

Everything works as intended

$ lein repl
Compiling projecta.core
nREPL server started on port 45563 on host 127.0.0.1 - 
REPL-y 0.5.1, nREPL 0.8.3
Clojure 1.11.0
OpenJDK 64-Bit Server VM 18-ea+36-Ubuntu-1
[...]
user=> (require '[projecta.core :as pa])
nil
user=> (def a (pa/->SomeType -3))
#'user/a
user=> (pa/abs a)
#object[projecta.core.SomeType 0x3131b2ea "3.0"]

Why?

@U064X3EF3 maybe you can help here?

Not going to look at it this weekend but I can check it out on Monday

Thanks! No rush, I just want to understand what is going on here and why this happens. Enjoy the weekend!

Struggling to parse the examples, it looks like everything working as intended

The protocol is defined in the namespace projecta.protocols so that is where the abs function that is part of the protocol is

Ah, I see the example in the middle with the face

No, that is correct too

Maybe protocol can be misleading, I just copied what I have in fastmath and clojure2d.

The difference is that in the first approach I call ns-unmap to remove abs from the namespace. In the second I use refer-clojure

The minimal example:

(ns projecta.core)
(ns-unmap *ns* 'abs)
(defn abs [v] (clojure.core/abs v))

versus

(ns projecta.core
  (:refer-clojure :exclude [abs]))
(defn abs [v] (clojure.core/abs v))

Try printing out the value of *ns* before the unmap

Both work without precompilation.

Sure

But when the namespace is precompiled latter contains abs, former not

I just wonder if the value of *ns* is not what you expect when loading an aot compiled namespace, because compilation is not happening in that case

I kind of expect it should be, but don't recall

After adding (println *ns***)

Compiling projecta.core
#object[clojure.lang.Namespace 0x5d10455d projecta.core]

user=> (require '[projecta.core :as pa])
#object[clojure.lang.Namespace 0x2db705a7 projecta.core]
nil
user=> (pa/abs -4)
Syntax error compiling at (/tmp/form-init1901708797312745991.clj:1:1).
No such var: pa/abs

hmmm... so compiled namespace is different object than namespace after require. Maybe that's why pa/abs is not visible? It belongs to a different namespace.

Oh, you know, I bet it is a constant pool thing

When you aot compile a namespace, all the code is turned into a static init method on the class file that is generated for the ns

Constants (like vars referenced) are lifted out into static fields

So for vars you only pay for var resolution once

But what is happening (my guess) is the def isn't looking up the var again, but reusing the one in the constant pool, which the ns-unmap previous removed from the ns

Try putting the def in an immediately invoked thunk ((fn [] (def ...))) Not sure if that would work around it if I have the issue right, but it might (the fn gets its own constant pool, but I am not sure when those constants would be initialized)

works!

((fn [] (defn abs [v] (clojure.core/abs v))))

Yeah, and if you don't aot compile, top level forms are effectively each compiled in a think like that, so they don't share a constant pool

I need to digest it. I mean, I'm not familiar with the aot path (yet). Thanks a lot anyway.

ok, I've read it again, I think I got it.

Definitely an interesting divergence of behavior between aot and not

Vaguely related to https://clojure.atlassian.net/browse/CLJ-1241

oh, indeed