Clojurians Log v2

How would you add async validations to clojure.spec? Caches can indeed be added, but not in a way I'd like - I cannot just discard a whole validator there when it stops being useful to me, I'd have to painstakingly invalidate caches myself. > I don’t quite understand “in-progress validations are discarded” If I have an in-flight async validation that uses data at epoch 1 and the next epoch 2 arrives before the validation is done, I want to abort the epoch 1 validation and start a new one, with new data. > Are you going to run long-running validation in ClojureScript on a single thread? Yes.

> The graph of validators part – I think it does pretty well. And this is also the most trivial part. :)

What’s your favorite way to make any function in JavaScript/ClojureScript async?

I don't need to make just the top-level validator async - that would be pointless, it would not let me render during validation. I need for every node in the graph to be async.

> Are you going to run long-running validation in ClojureScript on a single thread? > Yes. I would be careful if performance is a requirement. Long running CPU-intensive stuff in one thread typically doesn’t work very well (in terms of smooth rendering).

> I need for every node in the graph to be async. I see where you’re going with this… I can imagine a way coding that in core.async … but I don’t have a good feeling about how well it will perform (for big amounts of data, it will use a bunch of CPU regardless…)

Like, for example, suppose you have

(s/def ::x (s/req-un ::a ::b))

(defn validate-x [v]
  (js/Promise. (fn [resolve _]
    (resolve (s/valid? ::x v)))))

Yes…

Even if this existed or it can be made, my feeling is that it won’t perform well since you won’t have control of what’s really getting scheduled to run.

core.async does not support cancelling mid-flight. I can do some insane juggling with control channels per every node, but... nah, thanks. :)

Yeah exactly

I don’t think there’s a smarter/better way to do it (and again, my gut feeling is that it won’t be good – perhaps acceptable but not great)

I would look into web workers?

It can be made reasonably good if a validator that has a lot of steps puts some js/requestAnimationFrame here and there. Stuff can be CPU intensive, it can take 100% of a single core, and it can still allow the app to be responsive.

That way you won’t impact the rendering… but you do need to shuffle the data over - not sure what amount of data you’re expecting.

Yes… I’ve done stuff like.

It works, but very annoying…

> I would look into web workers? I know of web workers. Unfortunately, that's not enough - I need specifics. Serializing 10 MB of data on any change doesn't make sense. Sending data bit by bit - but then how do I cancel stuff? How do I route the data? How do I route the results? And so on and so forth. Web workers only bring a potential to replace one kind of async stuff with another, that's it. They don't magically solve everything.

10MB on every change is a lot, possibly. But I probably needs to be benchmarked to tell…

I don’t see a better solution than core.async for this… Otherwise … promises that do the same stuff? Those are even harder to control/intercept in bulk probably…

At least with core async you can have a control channel that pauses/resumes execution when a UI interaction/render happens…

I personally would try to throw together a quick web workers PoC before embarking on the core.async complex-control-loop though…

Might be surprising how fast 10MB can be transferred nowadays between threads… it’s not that much…

That way you can either rule out web workers or be pleasantly surprised… That other solution we’re talking about feels like a hornet’s nest of perpetual callback hell/core.async complexity 😅…

Also, I dont know what the problem specifically is, but you can possibly avoid this whole thing if you can somehow apply the validators incrementally rather than in bulk…

But perhaps that’s not possible/practical, you know best.

Misc. library to explore: https://github.com/GoogleChromeLabs/comlink Rather than looking for “validation library”, perhaps “async execution library” is a better/more likely fit.

Just ran this “search” https://grok.com/share/c2hhcmQtMg%3D%3D_7df87bcc-a527-4385-8264-01653e3b19c6

Surely you know that I know about low-level stuff already. :) As I said, I need specifics. If there's some prior art, I'd like to see it. If not, I'll try designing something myself. The initial "mind-palacing" suggests that I won't use spec, core.async or webworkers, but we'll see. > At least with core async you can have a control channel that pauses/resumes execution when a UI interaction/render happens… No you don't. Not in CLJS. core.async does not bring a second thread. If something does something CPU intensive, you will still have to wait for it to finish. You still have to split large tasks into 16 ms steps or so. > Might be surprising how fast 10MB can be transferred nowadays between threads… it’s not that much… It takes hundreds of ms to deserialize Transit data of that size. I imagine serialization would be similar. So it's not "not much", unfortunately. > if you can somehow apply the validators incrementally rather than in bulk… And that's exactly what I want. That's why I'm talking about a graph of validators where each step is async, each step is independent from its parents, each node has its own result and cache.

> core.async does not bring a second thread. of course…

Another existing library… https://nicolas-van.github.io/modern-async/modern-async/2.0.4/ Potentially as a building block, i.e. has a queue abstraction, ability to `cancelAllPending()` etc.

(no experience with it… found by LLM search)

> Surely you know that I know about high-level stuff already. 🙂 💯😜

does that even exist in javascript?

No idea. :)

With the risk of suggesting something absurdly obvious… clojure.spec? Not async or cached (I believe), but I guess that can be added. I don’t quite understand “in-progress validations are discarded” – sounds a bit complex. Are you going to run long-running validation in ClojureScript on a single thread? (or perhaps in a web worker, etc?) The graph of validators part – I think it does pretty well.

Web Worker?

I think you'd break up your data in segments, you said it's a graph? Validate each node one by one, and then validate each edge?

Have you read the previous discussion? :)

I’ve made something similar (?). A series of expensive computations where each step feeds the next step with data and also triggers a re-render. Each step/computation is expressed as a reducing function. A state machine keeps track of what step we are in and in the beginning of each step creates a ”worker” with the reducing function and the input coll as arguments. The worker is then executed and runs for about 10 ms before dispatching a ”completed [output]” or ”not completed [progress]” event to the state machine. If the computation is completed, the state machine progresses to the next step. Each step declares the app-db path to its input data and a re-frame interceptor watches these paths for changes. If any data changes, the state machine receives an event to go back to the respective step and re-run the computations from there.

And obviously, if the computation is not finished, the state machine updates its state with the progress and runs the worker again.

Nice! Is the code available publicly, by any chance?

No, not right now. It’s quite tied to my application and not as general as I would like - but I’ve been aiming to generalize it but haven’t had time. I can see if I can get around to that sometime next week (if of interest)

Would be great, thanks!

I don’t know of a thing like this, but it feels like something that could be built on top of Pathom.

Potentially, maybe. At least you’ll get a graph that can evaluate arbitrary code, express dependencies, and can be executed async.