Nice! But a question about the example, you would normally not call Thread/sleep in a go block?

probably not, but this is just for force bb to go out of threads :)

haha i see. Good job!

if you have a better example, please provide one :)

I haven’t played with core.async for a long time so would be a challenge. Just remembered that you can’t block in a go block 😅

yeah, this is true

but for bb this is no longer the case 😂

I was mostly wondering if this would work in Clojure hehe. Nice it works in bb 👏

yeah, gotta find a better example that currently works in clojure but fails in bb

in the book I had this example: https://book.babashka.org/#core_async but it still works for me at high numbers or it takes ages in both clj and bb

I’ll think about it too

Is core.async/timeout implemented? Something like (<! (timeout 1000)) maybe?

right

let's try

by replacing (Thread/sleep 1000) with (async/<! (async/timeout 1000)) it works better in clj, still crashes with the old bb but in the new bb I'm getting:

java.lang.AssertionError: Assert failed: No more than 1024 pending takes are allowed on a single channel.

I think I could just replace timeout with a virtual thread implementation too

it might be good to find out why this error is happening though

Yeah I recognize that error from the JVM. I think 1024 is the maximum of the channel’s queue so you are going over the limit of the go blocks internal channel?

Maybe use (dotimes [i 1000] (<!! ). The blocking one

I hope 1024 is still enough to prove your point

Sorry not (<!! ..) but (>!! ..) to insert in a channel

But I guess this completely changes the example hmmm

I don't get why this error gets triggered

with the bb example

Not sure exactly about the bb implementation, but in jvm. The go never blocks. So you are doing N=100000 puts on an internal go block channel (I believe) and the maximum is 1024

I cannot explain it by reading the core.async code (I tried), but I can easily reproduce it in bb

(def c (async/chan))
(dotimes [i 1025] (async/go (async/>! c i)))
Exception in thread "async-thread-macro-1025" java.lang.AssertionError: Assert failed: No more than 1024 pending puts are allowed on a single channel. Consider using a windowed buffer.
(< (.size puts) impl/MAX-QUEUE-SIZE)
	at clojure.core.async.impl.channels.ManyToManyChannel.put_BANG_(channels.clj:157)
	at clojure.core.async$fn__17563.invokeStatic(async.clj:174)
	at clojure.core.async$fn__17563.invoke(async.clj:166)
	at sci.lang.Var.invoke(lang.cljc:202)
	at sci.impl.analyzer$return_call$reify__4482.eval(analyzer.cljc:1402)

I think one problem is that 1024 is still ok for normal threads. So you run into the limits of core.async itself. Starting a million threads is an issue in my bb though:

(require '[clojure.core.async :as async])

(def n (* 1000 1024))

(println n "Threads")

(def *cnt (atom 0))

(def start (System/currentTimeMillis))

(def threads (doall (for [i (range n)]
  (let [t (Thread. (fn [] (swap! *cnt inc)))]
    (.start t)
    t))))

(println "Elapsed" (- (System/currentTimeMillis) start))
(println "cnt" @*cnt)

(System/exit 0)

time bb async.clj
1024000 Threads
Elapsed 48743
cnt 1024000
bb async.clj  34.36s user 47.61s system 167% cpu 48.860 total

So this is a small repro:

(require '[clojure.core.async :as async])

(def n 2000)

(let [begin (System/currentTimeMillis)
      state (atom 0)]
  (dotimes [_ n]
    (async/go
      (async/<!! (async/timeout 1))
      (swap! state inc))
    #_(async/<! (async/timeout 1000)) #_(async/go (async/<! (async/timeout 1000))
              #_(swap! state inc)))
  (while (< @state n))
  (println "Spawned" @state "go blocks and finished in" (- (System/currentTimeMillis) begin) "ms"))

when changing async/go to async/thread the error goes away. also the error only occurs in the native image, not in a JVM that runs bb

async/go creates a virtual thread

(go (<!! …) should be (go (<! …)) maybe. Otherwise it is similar to Thread/sleep

yeah, in this case (bb) it doesn't matter since they both map to the blocking option

I’m not sure what a good example is 😅 But you clearly did something special 🙂

I was also thinking maybe that 1024 queue size is related to the thread limit in core.async itself. With virtual threads maybe https://github.com/clojure/core.async/search?q=1024 can be increased to something higher (https://github.com/clojure/core.async/search?q=1024)

There is something odd with the timeout channel:

(def hashes (atom #{}))

(let [begin (System/currentTimeMillis)
      state (atom 0)]
  (dotimes [_ n]
    (async/go
      (let [chan (async/timeout 1)]
        (swap! hashes conj chan)
        (async/<! chan))
      (swap! state inc)))
  (while (< @state n))
  (println "Spawned" @state "go blocks and finished in" (- (System/currentTimeMillis) begin) "ms")
  (prn (count @hashes)))

It is as if they re-used:

$ clojure -J--enable-preview /tmp/go.bb
WARNING: Implicit use of clojure.main with options is deprecated, use -M
Spawned 2000 go blocks and finished in 41 ms
36

so that would explain multiple processes reading the same channel at the same time

I would assume the timeout channel being unique each time, but that doesn't seem to be the case

hmm yeah indeed

I almost feared there was a concurrency issue with SCI and virtual threads, but I read the source here:

(or (when (and me (< (.getKey me) (+ timeout TIMEOUT_RESOLUTION_MS)))
          (.channel ^TimeoutQueueEntry (.getValue me)))
        (let [timeout-channel (channels/chan nil)
              timeout-entry (TimeoutQueueEntry. timeout-channel timeout)]
          (.put timeouts-map timeout timeout-entry)
          (.put timeouts-queue timeout-entry)
          timeout-channel))

it buffers the same timeout channels if they are within 10ms difference...

ah ok

so perhaps this behavior can also be reproduced with normal JVM + clojure... let's see

Yep, JVM repro:

(require '[clojure.core.async :as async])

(def ^java.util.concurrent.Executor virtual-executor
  (java.util.concurrent.Executors/newVirtualThreadPerTaskExecutor))

(def n 200000)

(let [chans (atom #{})
      begin (System/currentTimeMillis)
      state (atom 0)]
  (dotimes [_ n]
    (.execute virtual-executor
              (fn []
                (let [chan (async/timeout 1)]
                  (swap! chans conj chan)
                  (async/<!! chan))
                (swap! state inc))))
  (while (< @state n))
  (println "Spawned" @state "go blocks and finished in" (- (System/currentTimeMillis) begin) "ms")
  (prn (count @chans)))

Linked discussion in #clojure channel: https://clojurians.slack.com/archives/C03S1KBA2/p1678466471709759

Now that I identified the issue I'm more confident that I can continue with virtual threads. I'll just make a virtual thread version of timeout

Ah look: https://groups.google.com/g/clojure/c/OSwKOKEupGc

hmm weird not sure if I understand how the number of puts is related to the caching of the timeout channel :thinking_face:

number of puts?

oh wait now it’s number of takes. Never mind

yeah, multiple threads are trying to read from the same channel

Ah ok I was misunderstanding the issue the whole time 🙈

sorry, was thinking it was about puts

no worries

(defn timeout [ms]
  (if virtual-executor
    (let [chan (async/chan nil)]
      (.submit virtual-executor (fn []
                                  (Thread/sleep ms)
                                  (async/close! chan)))
      chan)
    (async/timeout ms)))

Maybe I'll end up with a virtual-enabled fork of core.async :P

Yeah I think this pending take is an issue for when you have many threads.. I was trying something myself, but ran into it again

(require '[clojure.core.async :as a])

(def n 100000)

(def *cnt (atom 0))

(def ch (a/chan 100000000))

(dotimes [i n]
  (a/thread
    (loop []
      (let [v (a/<!! ch)]
        (swap! *cnt inc)
        (Thread/sleep 10)))))

(dotimes [i n]
  (a/>!! ch n))

(Thread/sleep 1000)

(println "COUNTED" @*cnt " and n " n)

yes, this is expected, since you read from the same channel

but I didn't expect calls like (async/timeout 1000) returning identical channels (re-used channels) sometimes

Yeah true. It feels like it’s impossible to proof your improvement this way hehe

$ bb /tmp/go.bb
Spawned 200000 go blocks and finished in 5572 ms
200000

Nice! 🚀

Did you already find an upper limit of the number of virtual threads?

Spawned 2000000 go blocks and finished in 7857 ms

$ bb /tmp/go.bb
Spawned 20000000 go blocks and finished in 160668 ms

hm no

this however does work:

user=> (babashka.cli/parse-args [":remote"])
{:opts {:remote true}}

I'll give it a shot 🙂

(don't worry about getting back to the computer, this isn't time sensitive for me at all)

Better example:

user=> (babashka.cli/parse-args [":remote" ":always"])
{:opts {:remote :always}}

it looks like the regex in auto-coerce only matches if everything after the colon is letters/digits: <https://github.com/babashka/cli/blob/52e7eaa3755043a0db7638133d96e4e7e55c2eba/src/babashka/cli.cljc#L97>

user=> (cli/auto-coerce ":abc")
:abc ; keyword
user=> (cli/auto-coerce ":abc-def")
":abc-def" ; string

matches what I'm seeing ☝️

aaah right

so the regex has to be adapted

why did I do this regex again...?

there was a reason for it. maybe a test will fail if you remove it

Yup:

so perhaps you're avoiding things with spaces in them? ./mycli --title ":1 This is a title."

yeah, that was it I think

also numerics as the first character aren't valid

but it's mostly about spaces. feel free to change the regex

also it should accept:

user=> (babashka.cli/parse-args [":remote" ":a/b"])
{:opts {:remote ":a/b"}}

Should :a/b/c be allowed?

that's not valid but we can let that slide

maybe just checking for whitespace makes sense

Some options:

;; single slash:
;; 
;;   [a-zA-Z][a-zA-Z0-9]+(/[a-zA-Z0-9-]+)?
;;
;; multiple slashes:
;; 
;;   [a-zA-Z][a-zA-Z0-9/]+
;;
;; no whitespace:
;;
;;   (not (str/includes? s " "))

I feel like being a bit defensive on the auto-coercion and only supporting things that really are wanted to be keywords is a good thing.

I think also a tab and newline etc, regex has \s for that

yeah

let's do this: it has to start with an alphabetic character, and then followed by alphanumerics, _, - or /

and we won't do validation about the number of /

also a literal . must be supported

Sounds good to me -- I'll put together a PR.

🙏

Let's say I have test.clj and test_test.clj (which is the corresponding test) Then if I (throw (Exception. "boom")) in the deftest in test_test.clj, I get a wrapped error However, if I throw one level down in test.cj, the error is not wrapped

That works, but what about doing this at a deeper level, in bb's clojure.test implementation?

and if that works, another layer deeper, in clojure stacktrace

Yeah that's the next step, I just wanted to make it work reliably

Which it isn't yet

cool

As I sometimes get unwrapped errors like

#error {
 :cause "boom"
 :via
 [{:type java.lang.Exception
   :message "boom"
   :at [java.lang.reflect.Constructor newInstanceWithCaller "Constructor.java" 500]}]
 :trace
 [[java.lang.reflect.Constructor newInstanceWithCaller "Constructor.java" 500]
  [java.lang.reflect.Constructor newInstance "Constructor.java" 484]
  [sci.impl.Reflector invokeConstructor "Reflector.java" 310]
  [sci.impl.interop$invoke_constructor invokeStatic "interop.cljc" 78]
  [sci.impl.analyzer$analyze_new$reify__4304 eval "analyzer.cljc" 1084]
  [sci.impl.analyzer$analyze_throw$reify__4264 eval "analyzer.cljc" 968]
  [sci.impl.analyzer$return_do$reify__3927 eval "analyzer.cljc" 124]
  [sci.impl.fns$fun$arity_1__1166 invoke "fns.cljc" 107]
  [sci.lang.Var invoke "lang.cljc" 200]

Basically I'm using it in my work project to iron out errors before (hopefully) moving it to clojure.stacktrace

However, if I throw one level down in test.cj, the error is not wrapped

(try ... (catch ^:sci/error Exception e))

this way SCI is signaled to produce an ex-info with the stacktrace

Ah-ha

The reason it doesn't always do this is that not each exception can carry metadata

That's interesting. Bad news for me because I'm not controlling the try/catch here. I think it's here https://github.com/babashka/babashka/blob/master/src/babashka/impl/clojure/test.clj#L540

This is only the case for code that is executed within SCI, i.e. interpreted code

the code you are pointing at isn't run inside SCI but as regular clojure code

oh but because it's a macro, it is run inside SCI

I stand corrected by myself ;)

we can of course change that

Could also be this https://github.com/babashka/babashka/blob/master/src/babashka/impl/clojure/test.clj#L720

But I think it's try-expr

you can fix this in userland by overriding the defmethod or using alter-var-root

(for now I mean, as part of the experiment)

Which defmethod do you mean? I'm not following

you can use alter-var-root on try-expr

on a macro?! wow

yeah of course :)

but hmm, not sure if this will be seen in SCI though

yes, I think so because try-expr is called through other macro-expansions

20: (alter-var-root clojure.test/try-expr (constantly try-expr))
    ^--- Can't take value of a macro: #'clojure.test/try-expr

(constantly @#'try-expr)

(alter-var-root #'clojure.test/try-expr (constantly @#'try-expr))

yes

Ok this runs

(defmacro try-expr
  "Used by the 'is' macro to catch unexpected exceptions.
  You don't call this."
  {:added "1.1"}
  [msg form]
  `(try ~(clojure.test/assert-expr msg form)
        (catch ^:sci/error Exception t#
          (clojure.test/do-report {:type :error, :message ~msg,
                                   :expected '~form, :actual t#}))))

(alter-var-root #'clojure.test/try-expr (constantly @#'try-expr))

But I'm still seeing an unwrapped exception

Doesn't see to make a difference

The exception is caught in my monkey-patched try-expr but it's not wrapped

{:type java.lang.Exception
   :message "boom"
   :at [java.lang.reflect.Constructor newInstanceWithCaller "Constructor.java" 500]}

this is because in macro you should probably write: ~(with-meta 'Exception {:sci/error true}) or so

user=> (meta ^:sci/error 'Exception)
nil

user=> (meta (with-meta 'Exception {:sci/error true}))
#:sci{:error true}

That worked! I'm not a macro pro

Added it to the gist https://gist.github.com/pesterhazy/ec1dedd5c83ae47d981488f743042ca8#file-core-clj-L11

I think there are similar problems with thrown? and thrown-with-msg? https://github.com/babashka/babashka/blob/master/src/babashka/impl/clojure/test.clj#L507

Would it make sense as a next step to upstream those changes into babashka.impl.clojure.test?

yes

OK, I have to pick up my daughter now. Maybe I'll have some time next week In the meantime the gist is there if people want to try it

I thought it already did

apparently not

it does false

want a PR?

yeah, I think so

PR: https://github.com/babashka/cli/pull/61

merged!

FZF fan here -- but I haven't tried streaming FZF input yet! As far as I'm aware, process can take stdin as an eager string, or as a lazy stream. So I believe streaming from a process command to find or ls should work.

Something like this perhaps: https://github.com/babashka/process#piping-output

This is working for me:

(require '[babashka.process :refer [shell process]])

(defn demo-3 []
  (-> (process "ls")
      (select-keys [:out])
      (shell "fzf")))

Got this one working with your tip (fd is a find-replacement)

#!/usr/bin/env bb
(ns file.fzf
  (:require [babashka.process :as p]
            [clojure.string :as str]))

(let [find (:out (p/process "fd . '/'"))
      fzf @(p/process ["fzf" "-m"] {:in find
                                    :out :string
                                    :err :inherit})]
  (->> fzf :out str/split-lines (mapv println)))

Nice.

awesome, thanks!

Didn’t even cross my mind that you could pass a not-yet-deref’d process as the input of another process, then deref the second without deref’ing the first :melting_face:

All while having access to Clojure data structures 😁

I think it should be possible to create a lazy sequence in clojure too and pass that to process or shell, but I haven't tried that either.

You might also be interested in babashka.fs/glob if you're doing stuff like this:

user=> (prn (type (babashka.fs/glob "." "**/*.clj")))
clojure.lang.PersistentVector
nil
user=> (prn (babashka.fs/glob "." "**/*.clj"))
[#object[sun.nio.fs.UnixPath 0x1a7235f7 "src/teodorlu/shed/libclonecd.clj"] #object[sun.nio.fs.UnixPath 0xffd8e2c "src/teodorlu/shed/month.clj"] #object[sun.nio.fs.UnixPath 0x65282c8b "src/teodorlu/shed/update_repos.clj"] #object[sun.nio.fs.UnixPath 0x1d7cd503 "src/teodorlu/shed/path_lines.clj"] #object[sun.nio.fs.UnixPath 0x74f11d65 "src/teodorlu/shed/install.clj"] #object[sun.nio.fs.UnixPath 0x60e2d902 "src/teodorlu/shed/quick_clone.clj"] #object[sun.nio.fs.UnixPath 0x28e4e6cc "src/teodorlu/shed/libquickcd.clj"] #object[sun.nio.fs.UnixPath 0x651fbf8a "src/teodorlu/shed/fzf_stream_demo.clj"] #object[sun.nio.fs.UnixPath 0xf9394f5 "src/teodorlu/shed/explore.clj"] #object[sun.nio.fs.UnixPath 0x7429b4c9 "src/teodorlu/shed/browsetxt.clj"] #object[sun.nio.fs.UnixPath 0x1a6156d4 "src/teodorlu/shed/ukenummer.clj"]]

Glob was my initial input ( files (->> (fs/glob "." "**") ) but it blocks, so wasn’t suited for this use-case (searching tons of files incrementally)

Gotcha 👍

Perhaps fs/glob and fs/match could have a callback based API as well. e.g. fs/glob-async or so

if that would help anything

That would be cool, personally I’d love to shell out less

Feel free to post an issue about it and then I'll think about it a little

in babashka/fs

No

The only option is to either source some generated bash stuff from bb or start a new shell in a different directory

#C029PTWD3HR does support it (node.js)

Gotcha 👍 I'm trying to make an interactive babashka script that can change shell directories, which I asked about a bit earlier. I tried using a shell function that executes some of the babashka script's stdout, but ran into problems with my own user interface. Because I was messing with stdout, I got problems when I tried to shell out to less. So I've decided to try to (process/exec (System/getenv SHELL)) instead. For my case, I can probably just as well keep track of the current directory, and pass :dir *current-directory into the places where that's required.

I ended up doing something like this:

(babashka.process/exec "zsh" "-c" (str "cd " (fs/absolutize loc) "; exec zsh"))

Yes: https://github.com/babashka/bbin

bbin is the recommended and easiest way for bb projects, but you can also do it using brew and other package managers, this just more work. Examples of projects that have been distributed like that are #C0400TZENE7 itself and #C03KCV7TM6F

Thanks.