@john none of what you are asking is possible with :advanced builds. and shadow-cljs is not meant to run in any kind of production environment, so what you are attempting is not a supported use case of shadow-cljs (and neither figwheel I'd say)

building workers on demand is really not all that useful if you ask me but I'm also not really sure which problem you are actually trying to solve

I mean you could just create a simple mechanism that loads the same file that the initial page loaded

but then have it do different things depending on if its on a worker context or not

Yeah, a shim for the :advanced mode case is pretty easy

Well, except with separate builds or modules, then you still need correlate each of the names

Optimizations :none is the hardest for me to wrap my head around

(defn ^:export some-worker-fn [maybe args]
  (do-whatever))


(defn spawn [target & args]
  (let [fn-name (munge target)
        worker (js/Worker. (str script-base "?" fn-name))]
    (.postMessage worker (pr-str args))))


(spawn `some-worker-fn :foo :bar)


(defn init-browser-side [])

(defn init-worker-side []
  (let [target (subs 1 js/location.search)
        target-fn (js/goog.getObjectByName target)]
    (js/self.addEventListener "message"
      (fn [e]
        (let [args (reader/read-string (.-data e))]
          (apply target-fn args)
          )))
    ))

(defn init []
  (if (exists? js/document)
    (init-browser-side)
    (init-worker-side)))

something like that maybe

anything that requires thinking about :none will probably break in :advanced, so you probably should not be doing it

That's helpful to think about, thanks for putting that script together

you probably want to add a lot of stuff for better message passing and proper worker termination though 😉

Yeah, I've already got a lot of the machinery built out in in.mesh. You can see it working here: https://github.com/johnmn3/in.mesh/blob/master/figwheel/src/in_mesh/figwheel/root.cljs

Haven't tested it in advanced yet, but the prior work on tau.alpha worked in advanced mode as well

I was using an importScripts shim with tau though. With this I'm trying to lean into the build tool's webworker facilities

The other major piece I'm working on is getting blocking semantics in web workers by calling sync-xhr on a service worker 🙂 similar to this hack https://sleep-sw.glitch.me/

It works on a main thread window page but I can't get the worker's xhr to be caught by the service worker atm

why though? I mean you still absolutely need to go async at some point. so why not just build for async in the first place?

Just a simpler programming model - the illusion of locality. None of it's actually local - stuff is spread out all across the circuit board, but languages give us the illusion of locality because the UX of async everything sucks

core.async is part of the way there, but you always end up bifurcating your code into what's inside/outside a go block, or async/await

its still an illusion though? you are just trading one set of trade-offs for another?

I mean, if it's on the same computer, among workers in the same memory, I don't see the point of giving up the illusion

Across a network, sure

But event then...

well same memory is the key here right? that part seems more relevant than all the worker coordination and stuff? I'm not convinced that one can actually build such a thing on SABs

especially since you need to start locking and stuff which sort of defeats the purpose of all the CLJ(S) datastructures in the first place?

SAB backed persistent datastructures can work just as well as they do on the JVM. There's nothing in principle different, save for the garbage collection you'd have to handle

I'd like to be proven wrong on that but I have my doubts

But that's not what in.mesh is about. in.mesh is just trying to be the underlying coordination mechanism that tries to make workers feel like threads

but I haven't seen any threaded code like you example from above

By threading I just mean spawn thing then do work on thing like you would a thread. Similar to what you showed above, but with more binding conveyance... And once there's blocking semantics in there, it'll be even more like threads

but blocking is the thing you can't do? or don't want to do?

I do want to. Got it working in tau years ago but then moved on to sabs, which make it easy to just do js/Atomics.wait

but sabs are restricted in ways that make deployment hard, so I'm trying to get the old way working, with sync-xhr and a service worker

You basically proxy calls through the service worker and the service worker just hangs the response until another work response with the result

Old example from tau:

tau.alpha.ex=> (let [res (xget "")
          #_=>       iss (-> @res js/JSON.parse (js->clj :keywordize-keys true))
          #_=>       loc (:iss_position iss)]
          #_=>   (println "International Space Station, Current Location:" loc))
International Space Station, Current Location: {:longitude -175.7632, :latitude -10.2605}
nil

xget is impled with a yield and there's no async rewriting. It's top level blocking

ok, now this request takes 10 seconds for some reason. and you are blocking the main thread for the entire time?

no, just a worker in a pool

so, there is still one async invocation somewhere that doesn't block?

Not in the core.async sense of rewriting code. Calling deref just parks the worker until the response is in, then it gets woken up

(defn handle-response [event]
  (let [xhr (.-target event)]
    (if (= (.-readyState xhr) 4)
      (if (= (.-status xhr) 200)
        (let [res (.-responseText xhr)]
          (when res
            (yield res)))
        (yield (.-statusText xhr))))))

(defn xget [url]
  (later [url]
    (let [axhr (js/XMLHttpRequest.)]
      (.open axhr "GET" url)
      (set! (.-onreadystatechange axhr) handle-response)
      (.send axhr nil))))

So you can convert async stuff back into sync

Only works in workers, not the main thread

But that's all using atomics and wait. I had it working with the service worker hack first, but I forgot how to do it 😆

This glitch has the service worker hack, but I can't get it to work in the manner I'm spawning these threads in in.mesh https://sleep-sw.glitch.me/

They also use the hack to do this: https://github.com/BuilderIO/partytown

yes, I'm aware of that hack. its neat but unusable in practice

Depends on the use case. Def slower than native threads, obviously

also I don't think you are making things any easier by trying to make things look sync when they aren't

All sync just looks sync

It's about usability

well, yes. but for that you kinda need to write your own VM. if that is your intent thats fine and interesting. faking it into something that wasn't designed for it later is where I have my doubts

but again the interesting part is the shared memory, so the SABs. solving the coordination is less interesting (to me, personally)

Agreed. I'm going for an on-ramp. Where you can get the semantics out of the box, but if you want higher performance, you have to enable SABs, which requires COOP/COEP... and there's issues with using that from 3rd party libs

Not too familiar with all that minutia, but SABs are kinda hamstrung right now

It'll probably smooth over, after some years

If you're delivering the application and can control the headers, and can control for some other gotchas, you can ship with SABs

Anyway, I appreciate your thoughts (and challenge ;))... Willing to pay a bounty though to get sync-xhr from workers to service workers working in in.mesh though, if anybody is interested 🙂

the code is essentially:

(defn- fetch-event [resolve]
  (js/console.log "[ServiceWorker] Fetch")
  (js/setTimeout
   resolve 1000
   (js/Response. "<h1>Hello, world!</h1>"
                 #js {"headers"
                      #js {"content-type" "text/html; charset=utf-8"
                           "Cache-Control" "max-age=31556952,immutable"}})))
(.addEventListener js/self "install" #(js/self.skipWaiting))
(.addEventListener js/self "activate" #(.waitUntil % (js/self.clients.claim)))
(.addEventListener js/self "fetch" #(.respondWith % (js/Promise. fetch-event)))

So, instead of a timeout, you'd just park the promise, and when the response is ready, the other worker working on it posts up the response and then sw then resolves the promise

And of course you want the sw to be fully async and never do long running operations on there

Anyway, thanks for the advice. I'll let you know if I make any progress

that js/setTimeout call seems wrong?

mmm, I might have messed it up when I cleaned it up to copy and paste

Yeah, the resolve is supposed to be outside the setTimeout

Maybe more like:

(defn- fetch-event [resolve]
  (js/console.log "[ServiceWorker] Fetch")
  (js/setTimeout
   #(resolve
     (js/Response. "<h1>Hello, world!</h1>"
                   #js {"headers"
                        #js {"content-type" "text/html; charset=utf-8"
                             "Cache-Control" "max-age=31556952,immutable"}})) 
   1000))

Here we go... I accidentally deleted the .then chaining:

(defn- fetch-event []
  (js/console.log "[ServiceWorker] Fetch")
  (-> (js/Promise. #(js/setTimeout % 1000))
      (.then #(js/Response.
               "<h1>Hello, world!</h1>"
               #js {"headers"
                    #js {"content-type" "text/html; charset=utf-8"
                         "Cache-Control" "max-age=31556952,immutable"}}))))
(.addEventListener js/self "install" #(js/self.skipWaiting))
(.addEventListener js/self "activate" #(.waitUntil % (js/self.clients.claim)))
(.addEventListener js/self "fetch" #(.respondWith % (fetch-event)))

But it's still broken from the worker - somehow the worker isn't eligible as client to the service worker, in how I'm instantiating them

I mean do you ensure its installed and ready before the worker actually runs?

yeah, tried that

If I wanted to develop remotely using shadow/watch is it possible. I’ve got a http-kit server on the same ec2 I’m running shadow watch, it’s serving my shadow js files through an nginx reverse proxy. Everything is working well enough, except the watch part. I’m imagining this is the root cause error, because it’s repeated frequently. ReferenceError: shadow$bridge is not defined at eval (shadow.js.shim.module$aws_amplify.js? [sm]:3:11) at eval (<anonymous>) at goog.globalEval (live.js:472:11) at env.evalLoad (live.js:1533:12) at live.js:1834:12 reportError @ live.js:1425 env.evalLoad @ live.js:1535 (anonymous) @ live.js:1834 Any ideas would be cool. If it’s not possible, that’s cool too. I’d appreciate a heads up on the not possible part, so I don’t waste a bunch of time. CHEERS and CLJS! Edit: After a little bit of time, I get a connection error. I’ve got a 404 on bundle.js.