Fork me on GitHub

@john none of what you are asking is possible with :advanced builds. and shadow-cljs is not meant to run in any kind of production environment, so what you are attempting is not a supported use case of shadow-cljs (and neither figwheel I'd say)


building workers on demand is really not all that useful if you ask me but I'm also not really sure which problem you are actually trying to solve


I mean you could just create a simple mechanism that loads the same file that the initial page loaded


but then have it do different things depending on if its on a worker context or not


Yeah, a shim for the :advanced mode case is pretty easy


Well, except with separate builds or modules, then you still need correlate each of the names


Optimizations :none is the hardest for me to wrap my head around


(defn ^:export some-worker-fn [maybe args]

(defn spawn [target & args]
  (let [fn-name (munge target)
        worker (js/Worker. (str script-base "?" fn-name))]
    (.postMessage worker (pr-str args))))

(spawn `some-worker-fn :foo :bar)

(defn init-browser-side [])

(defn init-worker-side []
  (let [target (subs 1 js/
        target-fn (js/goog.getObjectByName target)]
    (js/self.addEventListener "message"
      (fn [e]
        (let [args (reader/read-string (.-data e))]
          (apply target-fn args)

(defn init []
  (if (exists? js/document)


something like that maybe


anything that requires thinking about :none will probably break in :advanced, so you probably should not be doing it


That's helpful to think about, thanks for putting that script together


you probably want to add a lot of stuff for better message passing and proper worker termination though 😉


Yeah, I've already got a lot of the machinery built out in in.mesh. You can see it working here:


Haven't tested it in advanced yet, but the prior work on tau.alpha worked in advanced mode as well


I was using an importScripts shim with tau though. With this I'm trying to lean into the build tool's webworker facilities


The other major piece I'm working on is getting blocking semantics in web workers by calling sync-xhr on a service worker 🙂 similar to this hack


It works on a main thread window page but I can't get the worker's xhr to be caught by the service worker atm


why though? I mean you still absolutely need to go async at some point. so why not just build for async in the first place?


Just a simpler programming model - the illusion of locality. None of it's actually local - stuff is spread out all across the circuit board, but languages give us the illusion of locality because the UX of async everything sucks


core.async is part of the way there, but you always end up bifurcating your code into what's inside/outside a go block, or async/await


its still an illusion though? you are just trading one set of trade-offs for another?


I mean, if it's on the same computer, among workers in the same memory, I don't see the point of giving up the illusion


Across a network, sure


But event then...


well same memory is the key here right? that part seems more relevant than all the worker coordination and stuff? I'm not convinced that one can actually build such a thing on SABs


especially since you need to start locking and stuff which sort of defeats the purpose of all the CLJ(S) datastructures in the first place?


SAB backed persistent datastructures can work just as well as they do on the JVM. There's nothing in principle different, save for the garbage collection you'd have to handle


I'd like to be proven wrong on that but I have my doubts


But that's not what in.mesh is about. in.mesh is just trying to be the underlying coordination mechanism that tries to make workers feel like threads


but I haven't seen any threaded code like you example from above


By threading I just mean spawn thing then do work on thing like you would a thread. Similar to what you showed above, but with more binding conveyance... And once there's blocking semantics in there, it'll be even more like threads


but blocking is the thing you can't do? or don't want to do?


I do want to. Got it working in tau years ago but then moved on to sabs, which make it easy to just do js/Atomics.wait


but sabs are restricted in ways that make deployment hard, so I'm trying to get the old way working, with sync-xhr and a service worker


You basically proxy calls through the service worker and the service worker just hangs the response until another work response with the result


Old example from tau:

tau.alpha.ex=> (let [res (xget "")
          #_=>       iss (-> @res js/JSON.parse (js->clj :keywordize-keys true))
          #_=>       loc (:iss_position iss)]
          #_=>   (println "International Space Station, Current Location:" loc))
International Space Station, Current Location: {:longitude -175.7632, :latitude -10.2605}


xget is impled with a yield and there's no async rewriting. It's top level blocking


ok, now this request takes 10 seconds for some reason. and you are blocking the main thread for the entire time?


no, just a worker in a pool


so, there is still one async invocation somewhere that doesn't block?


Not in the core.async sense of rewriting code. Calling deref just parks the worker until the response is in, then it gets woken up


(defn handle-response [event]
  (let [xhr (.-target event)]
    (if (= (.-readyState xhr) 4)
      (if (= (.-status xhr) 200)
        (let [res (.-responseText xhr)]
          (when res
            (yield res)))
        (yield (.-statusText xhr))))))

(defn xget [url]
  (later [url]
    (let [axhr (js/XMLHttpRequest.)]
      (.open axhr "GET" url)
      (set! (.-onreadystatechange axhr) handle-response)
      (.send axhr nil))))
later sends the work to a worker on the pool. yield puts the result in a place. Derefing the later blocks until the message is done


So you can convert async stuff back into sync


Only works in workers, not the main thread


But that's all using atomics and wait. I had it working with the service worker hack first, but I forgot how to do it 😆


This glitch has the service worker hack, but I can't get it to work in the manner I'm spawning these threads in in.mesh


They also use the hack to do this:


yes, I'm aware of that hack. its neat but unusable in practice


Depends on the use case. Def slower than native threads, obviously


also I don't think you are making things any easier by trying to make things look sync when they aren't


All sync just looks sync


It's about usability


well, yes. but for that you kinda need to write your own VM. if that is your intent thats fine and interesting. faking it into something that wasn't designed for it later is where I have my doubts


but again the interesting part is the shared memory, so the SABs. solving the coordination is less interesting (to me, personally)


Agreed. I'm going for an on-ramp. Where you can get the semantics out of the box, but if you want higher performance, you have to enable SABs, which requires COOP/COEP... and there's issues with using that from 3rd party libs


Not too familiar with all that minutia, but SABs are kinda hamstrung right now


It'll probably smooth over, after some years


If you're delivering the application and can control the headers, and can control for some other gotchas, you can ship with SABs


Anyway, I appreciate your thoughts (and challenge ;))... Willing to pay a bounty though to get sync-xhr from workers to service workers working in in.mesh though, if anybody is interested 🙂


the code is essentially:

(defn- fetch-event [resolve]
  (js/console.log "[ServiceWorker] Fetch")
   resolve 1000
   (js/Response. "<h1>Hello, world!</h1>"
                 #js {"headers"
                      #js {"content-type" "text/html; charset=utf-8"
                           "Cache-Control" "max-age=31556952,immutable"}})))
(.addEventListener js/self "install" #(js/self.skipWaiting))
(.addEventListener js/self "activate" #(.waitUntil % (js/self.clients.claim)))
(.addEventListener js/self "fetch" #(.respondWith % (js/Promise. fetch-event)))
And that'll work when you hit a page scoped to the service worker from a browser tab. But when I hit it from a worker, it's just returning immediately, with the actual resource


So, instead of a timeout, you'd just park the promise, and when the response is ready, the other worker working on it posts up the response and then sw then resolves the promise


And of course you want the sw to be fully async and never do long running operations on there


Anyway, thanks for the advice. I'll let you know if I make any progress


that js/setTimeout call seems wrong?


mmm, I might have messed it up when I cleaned it up to copy and paste


Yeah, the resolve is supposed to be outside the setTimeout


Maybe more like:

(defn- fetch-event [resolve]
  (js/console.log "[ServiceWorker] Fetch")
     (js/Response. "<h1>Hello, world!</h1>"
                   #js {"headers"
                        #js {"content-type" "text/html; charset=utf-8"
                             "Cache-Control" "max-age=31556952,immutable"}})) 


Here we go... I accidentally deleted the .then chaining:

(defn- fetch-event []
  (js/console.log "[ServiceWorker] Fetch")
  (-> (js/Promise. #(js/setTimeout % 1000))
      (.then #(js/Response.
               "<h1>Hello, world!</h1>"
               #js {"headers"
                    #js {"content-type" "text/html; charset=utf-8"
                         "Cache-Control" "max-age=31556952,immutable"}}))))
(.addEventListener js/self "install" #(js/self.skipWaiting))
(.addEventListener js/self "activate" #(.waitUntil % (js/self.clients.claim)))
(.addEventListener js/self "fetch" #(.respondWith % (fetch-event)))


But it's still broken from the worker - somehow the worker isn't eligible as client to the service worker, in how I'm instantiating them


I mean do you ensure its installed and ready before the worker actually runs?


yeah, tried that

Patrick Brown20:07:01

If I wanted to develop remotely using shadow/watch is it possible. I’ve got a http-kit server on the same ec2 I’m running shadow watch, it’s serving my shadow js files through an nginx reverse proxy. Everything is working well enough, except the watch part. I’m imagining this is the root cause error, because it’s repeated frequently. ReferenceError: shadow$bridge is not defined at eval (shadow.js.shim.module$aws_amplify.js? [sm]:3:11) at eval (<anonymous>) at goog.globalEval (live.js:472:11) at env.evalLoad (live.js:1533:12) at live.js:1834:12 reportError @ live.js:1425 env.evalLoad @ live.js:1535 (anonymous) @ live.js:1834 Any ideas would be cool. If it’s not possible, that’s cool too. I’d appreciate a heads up on the not possible part, so I don’t waste a bunch of time. CHEERS and CLJS! Edit: After a little bit of time, I get a connection error. I’ve got a 404 on bundle.js.