Clojurians Log v2

@pieterbreed Oh I meant to say this the other day: Fusebox works with cljs because there's a completely different implementation for it. It's very much tied to the host VM.

From the conversation I figured as much 🙂

I have a working fix in SCI/bb that now actually uses type hints when doing the interop (thus you get 3 instead of nil consistently), but I also made one breaking change: when providing a type hint that can't be resolved, it's an error (like in Clojure). Now I ran into a failure with the meander library which is tested in CI.

275:                :counter/value ?value
276:                :as ?element}
277:               ^& ?rest-data}}]
278:     (let [element* (update ?element :counter/value inc)
279:           data* (conj ?rest-data element*)
                     ^--- Unable to resolve classname: &
280:           env* (assoc env :data data*)]
281:       [`(nth ~?symbol ~?value nil) env*])

what would ^& even do? is it parsed as just a symbol? does that work in clojure?

you can attach any tag to data, but SCI may be wrong about preserving this tag somewhere in a macroexpansion or so https://github.com/noprompt/meander/blob/74de6b1f651441092cc12d1c9012ef7086033040/src/meander/substitute/epsilon.cljc#L425

no time for this today, hopefully will find out tomorrow

Some more debugging done, if the native image fails it seems to be picking a different method overload than when it succeeds:

Native failing:

ctxClass: class java.util.concurrent.ThreadPoolExecutor
meth: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.lang.Runnable)
boxedArgs [Ljava.lang.Object;@7d8d4a71
retTYpe: class java.lang.Object

Native working:

ctxClass: class java.util.concurrent.ThreadPoolExecutor
meth: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.util.concurrent.Callable)
boxedArgs [Ljava.lang.Object;@3729817d
retTYpe: class java.lang.Object

It seems the reflector (which bb + SCI uses to implement interop) tries the methods in different order for the failing native image: Failing:

Trying method: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.lang.Runnable)
Trying method: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.util.concurrent.Callable)
found method: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.lang.Runnable)

Trying method: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.util.concurrent.Callable)
Trying method: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.lang.Runnable)
found method: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.util.concurrent.Callable)

Research to be continued

More info:

user=> (.isAssignableFrom Runnable Callable)
false
user=> (.isAssignableFrom Callable Runnable)
false

so it just takes the first one which fits. but if it takes the Runnable method, then the result from .get will be nil

Would it be fixed if I type hint it?

I didn’t intend to have any reflection at all.

it's already type hinted, but since the reflector doesn't use type hints this won't help. it's not your problem, it's something I have to think about fixing in bb/SCI

bb / SCI always uses reflection to implement interop since it's an interpreter, not a compiler

I think I'll just have to make a more sophisticated reflector which passes the information from the type hints along

instead of deriving them from the concrete arguments

which is hopefully doable!

made this issue: https://github.com/babashka/sci/issues/959

yes of course, we could fix this

let me check

I have this example working now:

(require '[com.potetm.fusebox.bulkhead :as bh])

(def bulkhead
  (bh/init {::bh/concurrency 2
            ::bh/wait-timeout-ms 100}))

(defn run []
  (prn :dude))

(bh/with-bulkhead bulkhead
  (run))

awesome! Mine broke already on the require 🙂

yeah, mine too, but a small fix made everything work :)

PR incoming

Thank you 🙏🏽

once I've merged https://github.com/babashka/babashka/pull/1810 and master CI finishes you can run bb with:

bash <(curl )  --dev-build --dir /tmp

merged it. will take a few minutes before master finishes but then you'll be able to use it. which OS are you on?

linux

ok, please test the --dev-build once CI finishes and if you need a proper other than the dev-build release let me know

ok, I'm about to step out, will let you know over the weekend if that's ok

sure, no hurry

ok, gave it a quick whirl. I think there are more classes missing: The basic all-in-one test is this:

(require '[com.potetm.fusebox.bulkhead :as bh]
         '[com.potetm.fusebox.bulwark :as bw]
         '[com.potetm.fusebox.circuit-breaker :as cb]
         '[com.potetm.fusebox.fallback :as fallback]
         '[com.potetm.fusebox.rate-limit :as rl]
         '[com.potetm.fusebox.retry :as retry]
         '[com.potetm.fusebox.timeout :as to])

user> (require '[com.potetm.fusebox.retry :as retry])
java.lang.Exception: Unable to resolve classname: java.util.concurrent.ThreadLocalRandom user com/potetm/fusebox/retry.clj:8:3

user> (require 'com.potetm.fusebox.circuit-breaker)
java.lang.Exception: Unable to resolve classname: java.util.concurrent.locks.ReentrantLock user com/potetm/fusebox/circuit_breaker.clj:7:3

user> (require 'com.potetm.fusebox.memoize)
java.lang.Exception: Unable to resolve classname: java.util.concurrent.ConcurrentHashMap user com/potetm/fusebox/memoize.clj:5:3

I'm wondering if there's a more direct way to see all the classes that the lib wants to load... or would that be looking at the source code?

yeah, I'll take a look at fixing these too

pushed fixes to main now, wait for CI to finish and then try again

I also added ConcurrentHashMap for the memoize namespace

After that I ran the test suite:

Ran 9 tests containing 182 assertions.
1 failures, 1 errors.

one test is failing in the bulwark namespace and another one in the timeout namespace

Ah when fixing something else I got all the tests working

but this requires a little change in fusebox (replacing some low level thing with a thing that's already in clojure: bound-fn*)

Looking at this as well. I should be able to push a patch for it later today.

Wow. Thank you. 🙏🏽

With some changes I discussed with @potetm all works on the JVM, but when compiled native I'm still getting:

FAIL in (bulwark-test) (/Users/borkdude/dev/fusebox/test/com/potetm/fusebox/bulwark_test.clj:12)
it works
expected: (= [:something :dangerous] (bw/bulwark spec (Thread/sleep 100) [:something :dangerous]))
  actual: (not (= [:something :dangerous] nil))

Could it be there is some exception being suppressed?

I bet it has to do with some reflection issue that can be solved but I'm only seeing nil :)

(bw/bulwark nil #_ spec
               (Thread/sleep 100)
               [:something :dangerous])

right that goes into the fallthrough

I would expect either an exception or the value or :yes! from the fallback

maybe inline the bh/bulwark call and start commenting out specific utilities?

`(fallback/with-fallback ~spec
     (retry/with-retry ~spec
       (cb/with-circuit-breaker ~spec
         (bh/with-bulkhead ~spec
           (rl/with-rate-limit ~spec
             (to/with-timeout ~spec
               ~@body))))))

I've got a repro here:

(require '[com.potetm.fusebox.bulkhead :as bh]
         '[com.potetm.fusebox.bulwark :as bw]
         '[com.potetm.fusebox.circuit-breaker :as cb]
         '[com.potetm.fusebox.fallback :as fallback]
         '[com.potetm.fusebox.rate-limit :as rl]
         '[com.potetm.fusebox.retry :as retry]
         '[com.potetm.fusebox.timeout :as to])

(def spec (merge (retry/init {::retry/retry? (fn [c dur ex]
                                                    (prn :dur dur :ex ex)
                                                    (< c 10))
                                   ::retry/delay (constantly 10)})
                      (to/init {::to/timeout-ms 500})
                      (fallback/init {::fallback/fallback (fn [ex]
                                                            (prn :ex ex)
                                                            :yes!)})
                      (cb/init {::cb/next-state (partial cb/next-state:default
                                                         {:fail-pct 0.5
                                                          :slow-pct 0.5
                                                          :wait-for-count 3
                                                          :open->half-open-after-ms 100})
                                ::cb/hist-size 10
                                ::cb/half-open-tries 3
                                ::cb/slow-call-ms 100})
                      (rl/init {::rl/bucket-size 10
                                ::rl/period-ms 1000
                                ::rl/wait-timeout-ms 100})
                      (bh/init {::bh/concurrency 5
                                ::bh/wait-timeout-ms 100})))

(prn (bw/bulwark spec (Thread/sleep 100) [:works]))

This also returns nil:

(prn (to/with-timeout spec
       (Thread/sleep 100) [:works]))

When I uncomment:

(to/init {::to/timeout-ms 500})

wat

I'll add some debugging in the timeout logic

init only adds the ::interrupt true, which is only used if it's gonna throw

but the timeout test suite passes?

Aha!

(.get fut
                     to
                     TimeUnit/MILLISECONDS)

huh, I'll probably add a test to the test suite for this then

(don't tell twitter)

it's not your fault, it's a bug in bb

no dont tell them I added a test. they'll get the wrong idea.

hehe sure

(kidding)

of course ;) This does print the correct result in bb native:

(import '(java.util.concurrent FutureTask Executors))

;; Create a callable (a function that returns a value)
(defn my-task []
  (println "Running in a virtual thread!")
  (Thread/sleep 10)
  "Task result")

;; Wrap the callable in a FutureTask
(def future-task (FutureTask. ^Callable (reify java.util.concurrent.Callable
                                          (call [_] (my-task)))))

;; Create a virtual thread per task executor
(def executor (. Executors newVirtualThreadPerTaskExecutor))

;; Submit the FutureTask to the executor (this starts it)
(.submit executor future-task)

;; You can now block to get the result
(prn "Result:" (.get future-task 100 java.util.concurrent.TimeUnit/MILLISECONDS))

that would be weird as well.

does putting a delay around the executor make a difference?

also, I wonder what using a native threadpool does

dereferencing the futuretask with (deref fut to ::o=noes) also returns nil

got something here, when I replace the timeout executor with:

(defonce ^:private
  timeout-threadpool
  (delay (. Executors newVirtualThreadPerTaskExecutor) #_(Executors/newCachedThreadPool (let [tc (AtomicLong. -1)]
                                          (reify ThreadFactory
                                            (newThread [this r]
                                              (doto (Thread. r)
                                                #_(.setName (str "fusebox-thread-"
                                                               (.incrementAndGet tc)))
                                                #_(.setDaemon true))))))))

try Executors/newSingleThreadExecutor ?

I gotta run to coach a soccer game. I'll be back later in the day to push my fixes for this.

sure, enjoy!

and thanks

both:

- (. Executors newVirtualThreadPerTaskExecutor)
- (Executors/newSingleThreadExecutor)

Hmm, this whole thing does the right thing too:

(import '(java.util.concurrent FutureTask Executors))

;; Create a callable (a function that returns a value)
(defn my-task []
  (Thread/sleep 10)
  "Task result")

;; Wrap the callable in a FutureTask
(def future-task (FutureTask. ^Callable my-task))

;; Create a virtual thread per task executor
(def executor (Executors/newCachedThreadPool
               (reify java.util.concurrent.ThreadFactory
                 (newThread [_ r]
                   (let [tc (java.util.concurrent.atomic.AtomicLong. -1)]
                     (doto (Thread. r)
                       (.setName (str "fusebox-thread-"
                                      (.incrementAndGet tc)))
                       (.setDaemon true)))))))

;; Submit the FutureTask to the executor (this starts it)
(.submit executor future-task)

;; You can now block to get the result
(prn "Result:" (.get future-task 100 java.util.concurrent.TimeUnit/MILLISECONDS))

;; Shutdown the executor afterwards
(.shutdown executor)

in bb native. I'll have to park this for the moment as well, I'll continue later

Got a small repro here now:

(import '(java.util.concurrent Executors ThreadFactory ExecutorService))

(defonce ^:private
  timeout-threadpool
  (delay (. Executors newVirtualThreadPerTaskExecutor)
         (Executors/newSingleThreadExecutor)
         (Executors/newCachedThreadPool
          (reify ThreadFactory
            (newThread [_this r]
              (Thread. r))))))

(defn timeout* [{to ::timeout-ms} f]
  (let [fut (.submit ^ExecutorService @timeout-threadpool
                     ^Callable f)
        v (.get fut
                to
                java.util.concurrent.TimeUnit/MILLISECONDS)]
    (prn :v v ) ;; nil ;;;
    v))

(prn (timeout* {::timeout-ms 100} (fn [] 3)))

Smaller repro:

(import '(java.util.concurrent Executors ExecutorService))

(prn
 (let [fut (.submit ^ExecutorService (Executors/newCachedThreadPool)
                    ^Callable (fn [] 3))]
   (.get fut)))

At least I nailed it down to 4 lines now...

well, whaddayouknow, I recompiled bb and it works..

Ran 9 tests containing 197 assertions.
0 failures, 0 errors.

@potetm Here's my patch for fusebox with a test runner for bb as well. You can invoke the test runner with bb test:bb

if you want to apply this patch, I could also add a test run in the Github workflow once bb with the fixes is released

to be sure, I added this test to the bb test suite:

(deftest cached-thread-pool
  (is (= 3 (bb nil "(import '(java.util.concurrent Executors ExecutorService))
                    (let [fut (.submit ^ExecutorService (Executors/newCachedThreadPool)
                       ^Callable (fn [] 3))]
   (.get fut))"))))

Seems to work on CI. Not sure what was the weird hiccup

So steps to finish this up: • apply patch to fusebox • I'll add fusebox tests (unchanged) to bb CI (I test a boatload of libs in bb CI on every commit) • perhaps add the bb test runner to Github actions on fusebox if you're up for it (you can always bug me with problems or just turn it off if you find it gets in the way)

pushed to github

I'll let you add a bb test runner, and after that's working, I'll carve a new release.

On Windows I see this failure:

Testing com.potetm.fusebox.timeout-test
FAIL in (timeout-test) (/C:/Users/runneradmin/.gitlibs/libs/com.potetm/fusebox/b61df08b11e4960b5dea2ccfb4532a717c71cb2a/test/com/potetm/fusebox/timeout_test.clj:16)
base case - no sleeping
expected: (< t 15)
  actual: (not (< 15 15))
FAIL in (timeout-test) (/C:/Users/runneradmin/.gitlibs/libs/com.potetm/fusebox/b61df08b11e4960b5dea2ccfb4532a717c71cb2a/test/com/potetm/fusebox/timeout_test.clj:16)
no interrupt
expected: (< t 15)
  actual: (not (< 15 15))

lol ugh, yes, all of timeout is flaky

just pushed an updated to increase the timeout times and tolerances for those tests

I'm still running into some of these in other CI (mac) now:

FAIL in (timeout-test) (/Users/distiller/.gitlibs/libs/com.potetm/fusebox/2f42391868c82c193628bec8922f8735ae3cac66/test/com/potetm/fusebox/timeout_test.clj:16)
base case
expected: (< t 25)
  actual: (not (< 58 25))

I can just skip testing the timeout namespace for now

I still randomly get the nil result now and then on CI:

FAIL in (cached-thread-pool) (interop_test.clj:247)
expected: (= 3 (bb nil "(import '(java.util.concurrent Executors ExecutorService))\n                    (let [fut (.submit ^ExecutorService (Executors/newCachedThreadPool)\n                       ^Callable (fn [] 3))]\n   (.get fut))"))
  actual: (not (= 3 nil))

wild

I'm debugging. I see that meander produces something like this:

(clojure.core/let [debug (quote {:tag &}) ?rest-data T3__3911] nil)))

but clojure doesn't stumble over this somehow. it does when I try to write a similar macro..

user=> (defn my-let-body [x] `(let [~(:symbol x) ~(:value x)] (inc ~(:symbol x))))
#'user/my-let-body
user=> (defmacro foo [] (my-let-body {:symbol 'x :value 3}))
#'user/foo
user=> (foo)
4
user=> (defmacro foo [] (my-let-body {:symbol (with-meta 'x {:tag '&}) :value 3}))
#'user/foo
user=> (foo)
Syntax error (UnsupportedOperationException) compiling let* at (REPL:1:1).
Can't type hint a local with a primitive initializer

Unfortunately I'm no meander expert..

This is really interesting. When I override the tag myself like this, it does crash:

~(with-meta (:symbol ir) {:tag '&})

debug2 (quote [& clojure.lang.Symbol] this is what the tag is. it is a symbol & but this one doesn't cause problems....?

So this works in clj:

~(vary-meta (:symbol ir) assoc :tag (:tag (meta (:symbol ir))))

~(vary-meta (:symbol ir) assoc :tag '&)

I found that clojure does accept this:

user=> (let* [y nil ^& x y])
nil

yeah but it doesn't actually attach the metadata in that case

So this also works:

user=> (defmacro dude [] `(let [y# nil ~(with-meta (symbol "x") {:tag '&})  y#])
)
#'user/dude
user=> (dude)

(let* [y nil ^& x y]
  (meta x))
=> nil

What's the rule here?

(meta x) is runtime metadata of x, this isn't about type hints

where would you expect the metadata to show up then?

I am talking about the tag of x. The metadata doesn't show up unless you inspect it (the binding symbol) in a macro which is what I'm doing in meander right now since it generates this let in a macro.

user=> (let [^String x (identity "dude") ^& y x] (.length y))
Syntax error (IllegalArgumentException) compiling . at (REPL:1:43).
Unable to resolve classname: &

Look for more discussion here: https://clojurians.slack.com/archives/C03S1KBA2/p1745409821307419?thread_ts=1745408863.163389&cid=C03S1KBA2

Looks like I have a fix (which is yet to be optimized but looks like it works):

$ bb /tmp/repro.clj
Trying method: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.lang.Runnable)
param classes: interface java.lang.Runnable
isCongruent: false
typesMatch: 0
Trying method: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.util.concurrent.Callable)
param classes: interface java.util.concurrent.Callable
isCongruent: true
typesMatch: 0
found method: public java.util.concurrent.Future java.util.concurrent.AbstractExecutorService.submit(java.util.concurrent.Callable)
3 3 3 nil

This isn’t a command, it’s a bash program so you should execute it with bash

Ahh

AFK but will provide an example later

This is ringing a bell. bash -c?

Yep that’s it

Use that as a the first string and then your program as the second

Got it. Thanks!