What do you mean by "fails to initialise"?

I’ve made sure the Clojure runtime is packaged in the jar file, and that the classes are on the class Path

have you called require on the namespaces prior to trying to use them?

A little error pops up (in Bitwig) saying “could not load BitwigExample.class” or something similar

With just the Clojure.core plus test I don’t think I need to require anything yet

I’ve also tried adding a static main method which calls Clojure to the BitwigExample Java class. So I can test it from the command line. This works fine

> “could not load BitwigExample.class” or something similar Hard to guess what the reason might be. It would be helpful if Bitwig provides any logs or prints larger error context to stdout.

Do you get extra information if you run Bitwig on command line? E. on Linux .deb install 5.2b7 startup says:

$ bitwig-studio
[2024-6-10 19:18:7.462 info] About to start the following process:  /opt/bitwig-studio/bin/show-splash-gtk /opt/bitwig-studio/resources/splash-bitwig-studio.png
[2024-6-10 19:18:7.463 info] Child process launched with PID 295514
[2024-6-10 19:18:7.463 info] Backing up log file "/home/user/.BitwigStudio/log/BitwigStudio.log" to "/home/user/.BitwigStudio/log/BitwigStudio-previous-run.log"
[2024-6-10 19:18:7.463 info] About to start the following process:  /opt/bitwig-studio/bin/BitwigStudio -cp /opt/bitwig-studio/bin/bitwig.jar:/opt/bitwig-studio/lib/cp:/opt/bitwig-studio/bin/libs.jar -Dorg.sqlite.lib.path=/opt/bitwig-studio/lib/bitwig-studio -XX:+UseZGC -XX:+ZGenerational -Xms300m -Xmx3g -Djava.io.tmpdir=/tmp/bitwig-user -DinstallationRoot=/opt/bitwig-studio -DsplashPid=295514 -Djava.awt.headless=true -XX:ErrorFile=/home/user/.BitwigStudio/bitwig-studio-jvm-crash.log com.bitwig.flt.app.BitwigStudioMain
[2024-6-10 19:18:7.463 info] Redirecting stdout to /home/user/.BitwigStudio/log/BitwigStudio.log
[2024-6-10 19:18:7.463 info] Redirecting stderr to /home/user/.BitwigStudio/log/BitwigStudio.log
[2024-6-10 19:18:7.463 info] Child process launched with PID 295515

thanks @U06C3BLAH98, i found a log file that gives this

[2024-06-10 17:31:26.362 Console BitwigControlJavaTest error] java.lang.NoClassDefFoundError: Could not initialize class clojure.java.api.Clojure
    at com.jahenderson777.BitwigControlJavaTestExtension.init(BitwigControlJavaTestExtension.java:23)
    at ZWA.YQx(SourceFile:24)
    at OLF.FqI(SourceFile:60)
    at Jez.GCp(SourceFile:298)
    at Jez.GCp(SourceFile:211)
    at Jez.<init>(SourceFile:168)
    at jjo.GCp(SourceFile:380)
    at jjo.fzi(SourceFile:523)
    at jjo.m_(SourceFile:334)
    at opc.run(SourceFile:91)
    at jjo.run(SourceFile:238)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "Control Surface Session"]
    at clojure.java.api.Clojure.<clinit>(Clojure.java:97)
    ... 11 more

The most recent discussion here that mentions that exception in the context of that class: https://clojurians.slack.com/archives/C03S1KBA2/p1574457737462700 Not a lot of detail in there but maybe the link at the bottom of the thread can be helpful.

thanks @U2FRKM4TW , very helpful , seems i might be able to change the class loader to fix the issue

yes, adding Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader()); before I start using Clojure seems to overcome my error, thanks!

It might not be the correct solution because class loaders are not trivial. But it might be, and seems that it's at the very least a solution. I myself don't really know.

Would be interested to see example code and/or results for this. Using the DrivenByMoss (Java) controller extension myself and have played a bit with the Javascript controller API eg. https://github.com/jasalt/daw-livestream-helper. This ReactJS controller script framework thing looked interesting lately https://github.com/joslarson/react-bitwig, but it would be nice to have better handle for the Java side of it also.

I’ll keep you updated @U06C3BLAH98, so far I’ve just established that I can start Clojure from within a Bitwig extension, start an nrepl server, connect from vscode and re-evaluate a function. The DrivenByMoss Java project is a great resource

We're on the latest core.async version at work, 1.6.681, and we no longer run into this so I assumed it was fixed in the latest version. @U0NCTKEV8 - any thoughts?

@U0DTSCAUU Are you using lein or the Clojure CLI?

the latest version rolled back the change that triggered ASYNC-249

Right, that's what I thought...

(ASYNC-249 is really a Clojure compiler bug that is being exposed)

I’m on the Clojure CLI

Something is definitely very wrong with my setup. Simply adding core.async 1.6.681 to my otherwise working deps.edn triggers the error!

"triggers" how? what action are you doing where you see it? if you are using any AOT compiled code, you will likely need to clean and recompile it

do you have a user.clj? override deps?

@U064X3EF3 yes that was the problem! Cleaning up some old AOT classes did the trick. Thank you so much!

The usual way to compartmentalize your resources in any Java-based app is to put them in appropriate packages, just like your code. So instead of serving your files from the public package at the root of your resources folder, you'd put them in a properly named package. You could use the classic reverse-domain structure if you want to be explicit, but your project name should be sufficient if you want less nesting.

Ok, so I'll serve files from resources/project/public . Thanks!

what’s your wrap-resources line look like?

[wrap-resource "public"]

(Update: I did have an ~old version of cel-java (0.2). I upgraded to 0.5.2 and we seem to be back)

if you are able to re-break it consistently, i’d be interested if you renamed (def cel-program …) to something else like (def repl-cel-program …) and see if that solves it

it seems like the var introduced in the catch is maybe getting confused in the reflection for .eval?

@U0C5DE6RK As a data point, we've been running Alpha 12 in production for two weeks with no issues, so I'm very interesting in uncovering a resolution to your issue above...

@U0C5DE6RK is there anything else in your catch branch? are you using cel-program there?

clojure.lang.Var$Unbound is the sentinel value that vars that have been created but not bound return when deref'ed

class clojure.lang.Var$Unbound cannot be cast to class java.lang.ClassLoader means somewhere you are taking the value of a var that doesn't have a value and trying to use it as a classloader

yes. and cel-program is a var introduced in the catch, but also the name of the destructured local that is used in interop. my wonder is if these two are getting confused when reflecting?

what makes you think they are?

@U11BV7MTK He didn't have the def there in the original code that failed...

the error is somewhere upstream of eval-program!, something is grabbing var that isn't bound and passing it in

oh i confused the two. i also thought i say cel-program in the stacktrace around the error symbol. So i’m definitely not on the right track

so you can start by asserting that the values being passed in are all what you expect them to be

The question is: what is cel-program? And can @U0C5DE6RK confirm the .eval was line 272 where the error originated? (looks right since 269 would line up with the defn and .eval is three lines below that.

wild guess, I would check your virtual pmap to make sure it propagates bindings

user=> (declare foo)
#'user/foo
user=> @#'foo 
#object[clojure.lang.Var$Unbound 0x2234078 "Unbound: #'user/foo"]
user=> (declare ^:dynamic foo)
#'user/foo
user=> (.run (Thread. (fn [] (prn @#'foo))))
#object[clojure.lang.Var$Unbound 0x2234078 "Unbound: #'user/foo"]
nil
user=> 
user=> (binding [foo 1] (.start (Thread. (fn []  (prn @#'foo)))) @#'foo)
1
#object[clojure.lang.Var$Unbound 0x2234078 "Unbound: #'user/foo"]

My recommendation would be to revert things back to your original, working Clojure 1.11 version (incl. the older cel-java version), and then change only the Clojure version to 1.12 Alpha 12 and restart your program. i.e., make small, carefully controlled changes one at a time to ensure you have identified the step that breaks things -- and then as soon as you have a breaking version, debug it without adding any defs or changing any versions.

Wow, clojure for the win -- thank you for jumping in on this with me team!

I reverted back to the first breaking change: 1.12.0-alpha12 + old version of cel 0.2. > can @U0C5DE6RK confirm the .eval was line 272 Yes indeed, .eval was on 272 @U04V70XH6! I think @U0NCTKEV8 is right and I don't propagate bindings. Going to try that real quick and update you!

If that fixes things, I suspect you were just getting lucky before in terms of the behavior...

Update: I went ahead and made changes, and I still got this error in 1.12.0-alpha12, after ensure we had the correct binding frame. Here's something interesting I noticed though: if I typehint the cel program, then I no longer see errors in 1.12.0-alpha12:

(defn eval-program!
  [{:keys [cel-program etype action]} bindings]
  (.eval ^dev.cel.runtime.CelRuntime$Program cel-program bindings))

java.util.concurrent.ExecutionException: java.lang.ClassCastException: class clojure.lang.Var$Unbound cannot be cast to class java.lang.ClassLoader (clojure.lang.Var$Unbound is in unnamed module of loader '
app'; java.lang.ClassLoader is in module java.base of loader 'bootstrap')
 at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    java.util.concurrent.FutureTask.get (FutureTask.java:191)

class clojure.lang.Var$Unbound cannot be cast to class java.lan
g.ClassLoader (clojure.lang.Var$Unbound is in unnamed module of loader 'app'; java.lang.ClassLoader is in module java.base of loader 'bootstrap') clojure.error.class=java.lang.ClassCastException clojure.error.line=27
2 clojure.error.phase=:execution clojure.error.source=cel.clj clojure.error.symbol=instant.db.cel/eval-program! exception.escaped=true exception.message=java.lang.ClassCastException: class clojure.lang.Var$Unbound ca
nnot be cast to class java.lang.ClassLoader (clojure.lang.Var$Unbound is in unnamed module of loader 'app'; java.lang.ClassLoader is in module java.base of loader 'bootstrap') exception.stacktrace=java.util.concurren
t.ExecutionException: java.lang.ClassCastException: class clojure.lang.Var$Unbound cannot be cast to class java.lang.ClassLoader (clojure.lang.Var$Unbound is in unnamed module of loader 'app'; java.lang.ClassLoader i
s in module java.base of loader 'bootstrap')
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at instant.util.async$deref_future.invokeStatic(async.clj:83)
        at instant.util.async$deref_future.invoke(async.clj:81)

Is this code up on GitHub where we can see the whole call chain? And perhaps try it ourselves?

Unfortunately the code isn't open source yet. I was able to solve this by downgrading to 1.12.0-alpha5. Will make a note to see if I can cook up a more minimal repro.

are you saying that alpha 5 does not exhibit this issue, but alpha 6 does? (this is very useful if you can reliably reproduce it in this manner)

my assumption is that this symptom is timing related and none of these version differences matter, you're just happening to see it on one or the other by chance

(I do worry Alex might be right. I wrote a quick script to run the test suite 100 times, and saw at least on alpha-5, there was no longer any test failures. There's a lot of changes in this PR though, and 100 times is still not that much; I'll see if I can really nail it down)

the stack traces you've shared don't match your code at all, the stacktraces are the result of an interop call trying to pass an Unbound where a Classloader is required

there were some changes with var interning in 1.12.0-alpha1 (and none afterwards), although I don't actually have any reason to believe those changes affected what you're seeing anyways.

I agree with @U0NCTKEV8 that I don't think you're actually working the problem you've observed

there is why I suggested starting to add asserts to make sure what you expect is being passed where you expect it is

like, some of the stacktraces show clojure.lang.Reflector, that can only be a reflective interop call

the original stack trace and code posted at the top here were invoking (.eval cel-program bindings). That was a reflective call. When trying to emit the args to the reflective call, it was getting a ClassCastException where it expected a Classloader but was getting an Unbound var. I'm inferring that bindings (or maybe cel-program, but I think that would give you a different error) is unbound.

if it's only sometimes unbound (as it seems to be), then I suspect concurrency and racing (supported by the pmap type stuff in the stack trace), maybe from loading that is racing

Really appreciate the deep dive team! I am going to try isolating this as much as I can for the next few hours, and if I can come up with something will update you. Will also make sure to include everything in one gist, so all the stacktraces and code match up.

a good trick for forcing these kinds of things is introducing artificial delays via Thread/sleep to make a race repeatable

Alright, I have a repro! https://github.com/stopachka/alpha12-cel-error/tree/main If I run alpha5, things work:

clj -M:use-5:run-m 
# => 1000

clj -M:use-12:run-m 
Execution error (ClassCastException) at stopachka.cel/eval-program! (cel.clj:66).
class clojure.lang.Var$Unbound cannot be cast to class java.lang.ClassLoader (clojure.lang.Var$Unbound is in unnamed module of loader 'app'; java.lang.ClassLoader is in module java.base of loader 'bootstrap')

https://github.com/stopachka/alpha12-cel-error/blob/main/src/stopachka/async.clj#L8 don't do this, use bound-fn or bound-fn*

Nice, updated the repro to use bound-fn* (Noob aside: is there a reason why in core we use binding-conveyer-fn for future-call, rather than bound-fn* ?)

What Java version are you using?

Are programs mutable? seems like you are using the same program in every future

you can completely eliminate the pmap stuff and trigger it with just a @(future ...)

a single future call

and in fact it triggers for me without a future at all

Hey @U064X3EF3 --

openjdk 22 2024-03-19
OpenJDK Runtime Environment Corretto-22.0.0.36.2 (build 22+36-FR)
OpenJDK 64-Bit Server VM Corretto-22.0.0.36.2 (build 22+36-FR, mixed mode, sharing)

eval is overloaded - if you type hint the bindings to ^Map does that still fail or give you a different message?

adding a java.util.Map type hint on bindings makes the exception stop happening

this is in new code

it must not be deterministic in selecting the overload

the boxArg there has support for using a dynamic proxy to adapt (in reflection) an IFn to an FI

yeah, it would not surprise me if that is not deterministic

one of the other overloads is CelVariableResolver which is an FI

@U04V70XH6 so we don't see a problems on the new alpha because we have warn-on-reflection set everywhere

the problem here is that it tries to use Compiler.LOADER, but it's unbound

which explains why it would appear to work fine at the repl

for some value of fine

it should fail at some point regardless, but would probably have been more obvious without the dynamic proxy error

I can take it from here, I think there are a couple things to address

thanks @U0C5DE6RK for sticking through to the repro

these are real bugs to fix before GA

sends high five Thank you all for diving deep with me on this!

there is code here checking that the incoming arg is an IFn - but you passed a map, which is an IFn so it looked ok to adapt

Ooooo tricky

so if not for the classloader issue, it would have adapted the map into a CellVariableResolver, with signature (CellVariableResolver, CellVariableResolver) -> CellVariableResolver which the dynamic proxy would have happily adapted into an IFn.invoke(Object, Object), which maps support

and it would have, I assume, run the program but without the bindings (since you called a different .eval than you meant to)

which might actually have worked sometimes

note that (set! *warn-on-reflection* true) would have tipped you off here, and is a good practice to set below your ns definition on any namespace where you do Java interop

And this is why our CI pipeline at work deliberately fails the build if someone tries to add a new src file that doesn't have (set! *warn-on-reflection* true) at the top 🙂

(and also fails the build if it spots new reflection warnings in the code)

@U064X3EF3 So is this due to expanding the Method Value stuff in Alpha 10 to fall back to reflective calls?

not method values, this is FI adapting which may occur either at compile time via invokedynamic or (if reflective) at runtime via dynamic proxy