Fork me on GitHub
#clojure
<
2020-02-03
>
Garrett Hopper03:02:11

I'm trying to use Clojure to build a plugin (distributed as a jar file) for an application. I'm having issues when this application loads the jar. It appears to do with this application messing with the Java classloader causing clojure.lang.RT.load to now be able to find clojure/core__init.class (which is in the jar).

Caused by: .FileNotFoundException: Could not locate clojure/core__init.class, clojure/core.clj or clojure/core.cljc on classpath.
        at clojure.lang.RT.load(RT.java:462)
        at clojure.lang.RT.load(RT.java:424)
        at clojure.lang.RT.<clinit>(RT.java:338)
Is there a way to change the classloader used by the AOT compiled jar?

Garrett Hopper03:02:48

It seems like *use-context-classloader* may have something to do with this, but I'm not sure how to change that within my Leiningen project.

seancorfield03:02:09

@ghopper You said it's an AOT'd JAR? And you're loading it into... another Clojure app?

Garrett Hopper03:02:36

No, it's being loaded into a standard Java app.

Garrett Hopper03:02:41

I saw something here: https://groups.google.com/forum/#!msg/clojure/Aa04E9aJRog/f0CXZCN1z0AJ about calling Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader()); before the Clojure code gets run, but I'm not sure how to add that Java code to the JAR.

seancorfield03:02:53

What sort of API does the Java app expect of the plugins? How are you implementing that in your plugin? Via a Java shim that calls into Clojure or are you trying to do it via gen-class directly in Clojure.

Garrett Hopper03:02:34

It uses an annotation on a Java class that extends another Java class. I'm using gen-class.

Garrett Hopper03:02:03

I suppose adding a Java shim would probably make the whole thing simpler.

Garrett Hopper03:02:33

Then I could do the Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader()); before doing some sort of dynamic loader stuff to pull in Clojure?

seancorfield03:02:03

Yeah, I'd highly recommend that approach, to be honest.

seancorfield03:02:23

And then use the standard Clojure/Java API to call into Clojure from that shim.

Garrett Hopper03:02:34

I should still be able to attach nREPL or something to it and make live updates, right?

seancorfield03:02:41

I think both Stu and Alex have posted examples to GitHub of that sort of thing.

Garrett Hopper03:02:57

Where might I find links to those examples?

Garrett Hopper03:02:36

:thumbsup: Thanks

brandon05:02:05

hi, my understanding is that when code is evaluated through a normal nREPL, any classes that are dynamically defined during evaluation will be stored in memory only, in a DynamicClassLoader. Is there a simple way to get those classes to actually be written to files?

seancorfield05:02:47

The behavior in the REPL is how Clojure code is treated when a library is loaded too: source in the library is compiled when a namespace is loaded, one form at a time -- just like the REPL -- to classes in memory.

seancorfield05:02:31

Only AOT compilation -- a separate, specific step, usually associated with creating an "uberjar" of a complete application -- are the compiled-in-memory classes also written to disk.

seancorfield05:02:28

Nearly all Clojure libraries are distributed as JARs of source code, so the normal mode of operation is compile-on-demand, to memory.

seancorfield05:02:14

Does that answer your question @brandon268?

brandon05:02:47

Hmm, to be more specific, my goal is to open a REPL, type in some forms, and then be able to get the class files of any dynamically-generated classes. Something along the lines of REPL middleware.

brandon05:02:56

The challenge I'm running in to is understanding the process for evaluating code in a REPL, and what customization I can apply to the class loader in that context, if any

seancorfield05:02:13

What problem are you actually trying to solve?

brandon05:02:28

Get the class files and send them to other machines

seancorfield05:02:22

If you type the forms into a file, you can AOT it to get class files but that's not normally how Clojure programs are run or distributed.

didibus05:02:19

Ya, not sure you can call the aot compiler on just a form. I remember it taking resources.

didibus05:02:16

You might be able to have a middleware that writes the sent form to a file, compile it, and then sends you back the compiled .class

emccue05:02:09

@brandon268 is this some apache spark - type usecase?

brandon05:02:00

there's already infrastructure in place to load dynamic class files on remote machines, the main challenge is coercing a Clojure REPL to produce actual class files

brandon05:02:36

so far, I haven't had any luck with re-setting the class loader before evaluating a form

emccue05:02:07

what infrastructure is already in place?

emccue05:02:20

or rather, what was the usecase before

brandon05:02:44

if a class files is placed in a certain directory, it becomes accessible to remote machines

seancorfield05:02:28

I thought I saw a library announcement just recently that lets you use regular Clojure source with Spark?

emccue05:02:40

this is pure evil, so I don't actually suggest it

emccue05:02:53

but you can always dig into the internals of DynamicClassLoader

emccue05:02:09

peek at whatever private fields the definitions are in

emccue05:02:16

public class DynamicClassLoader extends URLClassLoader {
    HashMap<Integer, Object[]> constantVals = new HashMap();
    static ConcurrentHashMap<String, Reference<Class>> classCache = new ConcurrentHashMap();
    static final URL[] EMPTY_URLS = new URL[0];
    static final ReferenceQueue rq = new ReferenceQueue();

seancorfield05:02:18

Oh, I was thinking of this (for Apache Beam): https://github.com/atdixon/thurber

brandon05:02:46

I've created a "wrapper" class loader that takes a DynamicClassLoader and takes any new class, also dumps it to a file. But Clojure doesn't let me override the class loader, or I'm doing it wrong

brandon05:02:23

I'm probably just doing it wrong 😛

didibus05:02:54

You tried calling setContextClassLoader ?

didibus05:02:58

On the thread?

brandon05:02:38

(binding [*use-context-classloader* true]
  (let [original-cl (.getContextClassLoader (Thread/currentThread))]
    (try
      (.setContextClassLoader
        (Thread/currentThread)
        (FileOutputClassLoader.
          original-cl
          "./repl-classes"))
      ;; arbitrary code here
      (finally
        (.setContextClassLoader (Thread/currentThread) original-cl)))))

didibus05:02:36

That didn't work?

didibus05:02:47

You might alao need to bind clojure.lang.Compiler/LOADER

brandon05:02:32

both? :thinking_face:

didibus05:02:41

Normally I think the DynamicClassLoader is bound to that variable. And that's the one used to compile repl forms. The context class loader is used before, to load a namespace and compile Clojure core I think

didibus05:02:03

I think only clojure.lang.Compiler/LOADER needs to be set to a DynamicClassLoader

didibus05:02:31

For repl forms. But I would try all combination until it works :p

Matt Butler14:02:26

Hey, is there a channel where people talk about using redis from clojure?

bortexz14:02:55

This clojure client could be of help: https://github.com/ptaoussanis/carmine

Matt Butler14:02:48

:thumbsup: using this client, wondered if there was an associated channel :)

bortexz14:02:20

Ops, ok! I missread your question 😅 thought you were asking for resources to use redis from clojure

seancorfield19:02:04

@mbutler Feel free to ask Clojure/Redis questions here but ask folks to follow-up in a thread, rather than in the main channel.

seancorfield19:02:01

We use Redis at work heavily. We looked at Carmine but it didn't support some stuff we needed (and still doesn't) so we went with Redisson but that is a very heavyweight dependency and we got tired of it dragging in 10MB of dependencies(!) as well as depending on some libraries that were very, very slow to become compatible with Java 9+, so we recently switched to Jedis which is fast and lightweight, but didn't have the sort of connection pooling that Redisson had, so we wrote our own with core.async.

lukasz19:02:58

@U04V70XH6 apart from the connection pool, what else you were missing from Carmine?

lukasz19:02:07

We used Jedis in a couple of places and it wasn't great

seancorfield19:02:00

I think when we got started, we needed support for Redis Cluster but I think we gave up on that at some point. @U0NCTKEV8 may have more insight since he's the one dealing with Redis at that level, more than myself.

seancorfield19:02:04

We've also moved away from the taoensso libraries -- we used nippy and timbre and we've gotten rid of both of those. Too many codependencies and a lot of weird utility code that we didn't need (and therefore didn't want loaded into our systems).

hiredman19:02:55

our history with redis client libraries is actually very twisted, and mixed across different subprojects

hiredman19:02:32

the first thing I recall was a move from carmine to jedis, which was, if I recall, an attempt to simplify things and cut down dependencies, but then we had a new project that came in using redisson (another library the subproject used had integrations with redisson, but we didn't end up using the integrations), so we had both jedis and redisson for a bit, and then in a bid to unify things got rid of jedis in favor of redisson, and then to try and simplify things (in some part to prepare for upgrading jdk versions) we switch everything to jedis, and ended up needing a custom connection pool because jedis's connection pool stuff didn't make sense with somethings we were doing with redis pub/sub

lukasz20:02:28

Looks like we're not the only ones with 2+ Redis clients in our codebase

Matt Butler19:02:17

Wasn't the direction that had intended this thread to go, but it's been a really good read. Those whove moved off carmine, in what format are you encoding your entries?

Matt Butler19:02:42

Also would people be interested in a #redis channel? Seems like enough usage and clojure related discussion to justify one. Anyone know the process for getting a channel?

hiredman19:02:42

you click the (+) next to Channels in the ui and create one

Matt Butler19:02:02

Haha, I had imagined I needed to be an admin, probably should have checked

Matt Butler19:02:42

Well #redis now exists

seancorfield19:02:55

@U0NCTKEV8 how would you describe how we are encoding values in Redis these days? EDN?

hiredman19:02:17

yes, edn, I think we are kind of a special case in that we are using redis's pub/sub much more then we are using the key/value store. I don't think we even have metrics on how much we are doing key/value operations, but our metrics for pubsub have alerting attached to them

Matt Butler19:02:44

My question was actually around using carmine and the new redis streams API. The xreadgroup blocks forever and is unresponsive to thread.interrupt. the only option appears to be using the client unblock command

Matt Butler19:02:08

But in carmine reliably knowing what "client" you are using for any one op is harder than youd hope.

Matt Butler19:02:47

I've gone with setting a client name and querying redis for client list and doing some string matching. But it's all feels a bit unsatisfactory especially as this is only really there to support reloaded repl workflow in dev (killing the jvm is the only shutdown option needed in prod)

hiredman19:02:51

the last time I looked at carmine it had a kind of global queue of operations, which I really did not care for

Matt Butler19:02:34

Yeah it's very opaque to you, kinda happens automagically as part of the wcar macro

Matt Butler14:02:54

Had a quick search, couldn't find anything but I might have missed it.

grounded_sage14:02:33

How do I track down where this exception is from?

Exception in thread "async-thread-macro-2" java.lang.NullPointerException
        at clojure.core.async$thread_call$fn__14495.invoke(async.clj:484)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

ghadi16:02:57

@grounded_sage in some cases the JVM elides stack traces on NullPointers

ghadi16:02:16

There is a JVM flag to tell it not to do that

ghadi16:02:50

-XX:-OmitStackTraceInFastThrow

kenny16:02:04

I always start all my REPLs with this flag. Any reason not to also enable it in production? It doesn't seem like it would actually have a significant performance impact.

jumpnbrownweasel16:02:54

For Java apps in general I think the only reason not to disable it in production is when the app is using exceptions as common "return values", i.e., not for exceptional conditions. In that case, the creation of the stack trace could have noticeable overhead. But this seems like an odd case to me.

jumpnbrownweasel16:02:22

But in the exception you posted, there is a stack trace. When stack traces are omitted, what I see is just the message. So I don't see how disabling the JVM flag would help in that case.

ghadi16:02:40

no there are missing parts in the stack trace that was posted

jumpnbrownweasel16:02:06

ok, I haven't see that happen before, but I haven't seen everything. :-)

ghadi16:02:18

core.async/thread ....????..... NPE

ghadi16:02:34

core.async thread doesn't itself throw NPEs

jumpnbrownweasel16:02:27

Can't any thread throw NPE when there is a bug?

ghadi16:02:25

yes, but async/thread doesn't throw it, the user code does

ghadi16:02:30

(async/thread ((fn foo [x] (x 42)) nil))

ghadi16:02:43

the foo ^ function should be visible in the trace

ghadi16:02:51

between the NPE and the thread

jumpnbrownweasel16:02:41

i see what you mean

jumpnbrownweasel16:02:22

Just curious about this. https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async.clj Line 484 is the call to (f) . Could that cause NPE if f is nil?

(defn thread-call
  "Executes f in another thread, returning immediately to the calling
  thread. Returns a channel which will receive the result of calling
  f when completed, then close."
  [f]
  (let [c (chan 1)]
    (let [binds (clojure.lang.Var/getThreadBindingFrame)]
      (.execute thread-macro-executor
                (fn []
                  (clojure.lang.Var/resetThreadBindingFrame binds)
                  (try
                    (let [ret (f)]
                      (when-not (nil? ret)
                        (>!! c ret)))
                    (finally
                      (close! c))))))
    c))

ghadi16:02:37

no - there's only one caller of that function, and it passes things

jumpnbrownweasel17:02:22

thread-call is a public API fn, and could be called by the app with nil.

jumpnbrownweasel17:02:33

But I guess it is only called from one place in the async thread. Still, maybe worth checking for calls to that fn in the app.

ghadi17:02:12

yup good eyes, it looks like someone called it with a nil

ghadi17:02:29

I too quickly assumed

ghadi17:02:46

usually when you see a short trace with an NPE, some elision is happening

jumpnbrownweasel17:02:02

I wish the default for -XX:-OmitStackTraceInFastThrow were switched, it causes so many problems. But I guess the JVM team is just as concerned with backward compatibility as you folks are with the Clojure API. :-)

ghadi17:02:51

they regularly change the -XX options - it's not a contract

ghadi17:02:00

other JVMs don't have this option

ghadi17:02:04

it's a hotspot thing IIRC

grounded_sage22:02:30

Man this is too deep for me at the moment 😅 though I think I tracked down the bad piece of code which was a loop that never returned.. just halted.

grounded_sage21:02:25

How do I set this JVM flag in the repl @U050ECB92 @U083D6HK9?

jumpnbrownweasel21:02:27

What we concluded above is that the stack trace is accurate and something (seems to be in your code) is calling thread-call and pasing nil, causing the NPE.

grounded_sage21:02:23

Thanks @UBRMX7MT7. From what I have been learning from the videos I have been watching lately. Take out the async when you can… I just removed that thread-call

jumpnbrownweasel21:02:34

ok! Any sort of multi-threaded anything is more difficult to debug and get right, that's for sure.

seancorfield19:02:04

@mbutler Feel free to ask Clojure/Redis questions here but ask folks to follow-up in a thread, rather than in the main channel.

Kevin Byrne22:02:52

I could use some help with reflection. I have a Java .jar that I'm wanting to digest at runtime, and so far I've been able to find it and load the class. However, when I attempt to invoke a method, I'm running into issues.

Kevin Byrne22:02:34

When I call (.getMethods my-class) I'm getting back reasonable info: #object["[Ljava.lang.reflect.Method;" 0x6e321cee "[Ljava.lang.reflect.Method;@6e321cee"]

Kevin Byrne22:02:24

And I can getName from the method `

(.getName (first (.getMethods my-class)))

Kevin Byrne22:02:31

which gives me what I'm expecting

noisesmith22:02:41

why are you reflecting? is the exact set of methods available unknowable at compile time?

Kevin Byrne22:02:12

But if I try to invoke the method (.invoke my-method)

Kevin Byrne22:02:31

I get No matching field found: invoke for class java.lang.reflect.Method

Kevin Byrne22:02:59

The methods are not knowable at compile time

noisesmith22:02:11

you need to pass the object as the first arg

noisesmith22:02:43

so the shortest possible call is (.invoke my-method my-instance) or (.invoke my-method my-class)

noisesmith22:02:33

> If the underlying method is static, then the specified `obj` argument is ignored. It may be null. TIL - there still needs to be a placeholder there

noisesmith22:02:00

also any args need to be packed into an object-array

Kevin Byrne22:02:07

So the method I'm trying to invoke is just a public static void main(String[] args)

noisesmith22:02:37

then you can pass nil for the object, then an object-array where the first item is a string-array (optionally empty)

Kevin Byrne22:02:46

So I would that look like (.invoke my-method my-class nil)?

noisesmith22:02:32

it would be (.invoke my-method (or my-class nil) (object-array (into-array String [])))

noisesmith22:02:42

it might work with an empty last arg, not sure

noisesmith22:02:25

but strictly speaking, main there has exactly one arg which needs to be an array of string

noisesmith22:02:47

and the .invoke method takes all args in an object array

Kevin Byrne22:02:52

(.invoke my-method (object-array (into-array String []))) returned "No matching method invoke found taking 1 args for class java.lang.reflect.Method"

noisesmith22:02:08

right, you missed the mandatory first arg

noisesmith22:02:17

which can be nil or your class in this case

Kevin Byrne22:02:59

(.invoke my-method my-class (object-array (into-array String []))) says "wrong number of arguments"

Kevin Byrne22:02:14

same with (.invoke my-method nil (object-array (into-array String [])))

Kevin Byrne22:02:16

More specifically `Execution error (IllegalArgumentException) at jdk.internal.reflect.NativeMethodAccessorImpl/invoke0 (NativeMethodAccessorImpl.java:-2). wrong number of arguments`

noisesmith22:02:20

wrong number off arguments to who? invoke, or the underlying method?

hiredman22:02:03

that is the underlying method

noisesmith22:02:06

oh - I think you want (object-array [(into-array String [])])

noisesmith22:02:08

that's my bad

Kevin Byrne22:02:29

hah! That was it

noisesmith22:02:49

so the semantics here are that you want one empty array of String

noisesmith22:02:09

where before you had zero objects (coerced from an empty array of String)

noisesmith22:02:32

gotta stack your empties correctly :D

Kevin Byrne22:02:43

heh, well thank you sir.

Kevin Byrne22:02:59

ok, I think I understand better now. In the method definition, it's looking for (Object obj, Object... args), so the Object... is the outer (object-array), and the first item in it was String[] created by calling (into-array String []).

noisesmith22:02:32

right, exactly

noisesmith22:02:32

it's easier to get confused by because main already acts like varargs by taking arguments in an array, so we end up needing the array of arrays

noisesmith22:02:16

and object-array takes a collection to coerce, so it's easy to leave off a layer and get the wrong coercion for nested arrays

👍 4