Is it possible to make the smallest repro possible, preferably even without the awyeah-api lib?

If it's an issue with awyeah-api I suggest creating an issue there, but if you can demonstrate it's a bb issue, please make an issue with bb

That's the kicker; it wasn't obvious to me from the stack trace whether the exception originated from the library or not. It doesn't make it easier that it only fails for a few elements of the collection I'm processing either. Anyway, the reason I started in this channel was because I wasn't able to reproduce the error on JVM, so I thought maybe you could deduce something from the stack trace

Feel free to post the stacktrace (in a gist), perhaps it will ring a bell.

https://gist.github.com/joakimen/81f4f2e6ffe13be7b85ee61e97355d0f

This only happens for a few elements of the collection. I have 47 queues, and when fetching attributes for each queue in parallel, like 1-10 of them fail with that error. On JVM, none of them produce that error

ok, this totally rings a bell, it's a reflection configuration issue

let me fix that right away

oh no it wasn't what I thought

I've seen a similar error in httpkit with native-image. this was an issue with SimpleDateFormat or something. I migrated that to java.time and the error went away, let's check if awyeah is using this old stuff

https://github.com/http-kit/http-kit/commit/b45725fb64c1025c305632a6d6d869ccc447b4a5 here I got the same bug

check this issue: https://github.com/http-kit/http-kit/issues/543 are you providing a virtual thread pool or something?

The error seems to happen around clojure.data.xml$parse so perhaps it's possible to make a repro using that

aaah maybe awyeah-api is using core.async?

yeah, gotcha. go blocks in core.async in bb are using virtual threads

let me try to repro this

I'm trying to repro it like this:

$ bb -e '(clojure.core.async/<!! (clojure.core.async/go (clojure.data.xml/parse (java.io.ByteArrayInputStream. (.getBytes "<a></a>")))))'
#xml/element{:tag :a}

Perhaps it would help you if executed a dummy XML read like the above in your program before doing anything with awyeah

A workaround for your problem would be I think:

(alter-var-root #'clojure.core.async/go (constantly @#'clojure.core.async/thread))

this reverts using virtual threads for go blocks to normal threads

Ah it seems the native image 21.0.0.1 fixes this problem

let me upgrade so you can test the dev version

Sorry, was eating! I got this error using the Java Executors stuff with a fixed thread pool like this

(ns sqs.lib.repro
  (:require [com.grzm.awyeah.client.api :as aws])
  (:import [java.util.concurrent ExecutorService Executors Future]))

(defn jxmap
  ([f coll] (jxmap f coll 100))
  ([f coll concurrency]
   (let [executor  (Executors/newFixedThreadPool concurrency)
         tasks (mapv #(fn [] (f %)) coll)]
     (->> (.invokeAll ^ExecutorService executor tasks)
          (map #(.get ^Future %))))))

(def attributes (jxmap #(aws/invoke sqs-client {:op :GetQueueAttributes
                                                    :request {:QueueUrl %
                                                              :AttributeNames ["All"]}}) queues 100))

yeah, it's related to bb's core.async implementation using vthreads

try this as a workaround:

(alter-var-root #'clojure.core.async/go (constantly @#'clojure.core.async/thread))

before actually requiring/loading awyeah

And that was used in the awyeah-lib, or?

yeah

you can use this:

BABASHKA_PRELOADS="(alter-var-root #'clojure.core.async/go (constantly @#'clojure.core.async/thread))"

I'll upgrade native image to 21.0.0.1 and then you can also try it without that patch

I tried it now by evaluating (alter-var-root #'clojure.core.async/go (constantly @#'clojure.core.async/thread)) , then evaluating the async code. I still get the same problem here, is there a difference between evaluating it in the repl, and providing the env var?

you should ensure to NOT load awyeah before you evaluate this

the env var is a safer bet

right ok, sec

That seems to have done the trick!

of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures

I wouldn't have caught that in a million years

Should this be an issue for awyeah then?

no, it's a bb issue, feel free to post an issue, I'm working on a fix (by upgrading graalvm). Also please post the workaround in the issue so people can find it later

Will do

which OS are you on?

macos 14.1

m1?

Yep!

alright, try this:

bash <(curl ) --dev-build --dir /tmp

https://github.com/babashka/babashka/issues/1639

Testing

Looking good :star-struck:

of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures
of 47 queues, there were 0 failures

(System/getenv "BABASHKA_PRELOADS") ;; => nil

nice :)

What was the fix? So many branches. Saw the bump of graalvm here https://github.com/babashka/babashka/commits/21_0_0_1, but was that all you needed, and how did you know?

yes, that branch fixes it

I didn't know, but I suspected that the graalvm folks might have fixed this bug

Hah

That was a good bet

Seriously, amazing catch, thanks so much, I've had this problem for weeks and thought I was just getting increasingly bad at clojure

hmm, actually, this is weird. the version you installed wasn't the version from my branch but from the master branch. So I'm puzzled why it works for you

it's coming from here: https://github.com/babashka/babashka-dev-builds/releases/tag/v1.3.186-SNAPSHOT

that release is from 3 days ago

I haven't upgraded homebrew-bb for those days, and just did it during lunchtime now. Will try again with the homebrew-version and see

no, this has nothing to do with homebrew

but the dev-builds releases only update from the master branch

anyway I can repro it locally using an example I posted in the issue and I can confirm that it works with 21.0.1 which I have installed locally

but it also mysteriously works with the build from 3 days ago

It could be the result of disabling some optimization

Anyway glad it works with the dev build

Yeah 😄 Can expect this to be part of v1.3.186 later then?

For sure

thanks for sponsoring :)

If anyone in this community deserves it, it's you ⭐

1.3.186 was just released

thanks for sharing, I'll ask in the native-image channel on native-image if they have heard of this

You can follow this issue: https://github.com/oracle/graal/issues/4854

What is aws here?

cognitect aws api

is this in a JVM?

yup

oo, I totally forgot that bb.curl was primarily for bb, not jvm clj

AFAIK the stream should not be consumed if you don't read it

I think babashka.http-client is a better option if you're on the JVM

the interface is mostly the same

sounds good, I'll give that a spin. thanks!

that works a bit better, much appreciated!