Clojurians Log v2
missionary 2024-10-21
hey, i am wondering, why was it reasonable to use the blocking <!! operator from core.async here https://github.com/leonoel/missionary/wiki/Creating-flows#coreasync-channel-flow ? wouldn't it be nice to provide glue code in form of callbacks? is it to ensure backpressure?
is there an example somewhere of how to read all available data out of a buffer to process it in a batch? i can do this with core.async like this https://github.com/replikativ/datahike/blob/main/src/datahike/writer.cljc#L97. i still want to have proper backpressure
If I understand correctly the missionary equivalent would be the pattern discussed here, with a zero timeout https://clojurians.slack.com/archives/CL85MBPEF/p1728285503450299
perfect, thx!
stupid question, when i am in a function inside of m/reduce , is it possible to make it block on a task the function is returning? for core.async i had to reimplement a whole bunch of higher-order functions (`map`, filter, for, ...) and macros to properly handle go-routines returned by such functions
In a coroutine, haskell's do notation is exactly let. The extensions provided by clojure for are not really necessary because you can use the rest of the language (e.g. conditionals, try/catch etc)
(m/ap
(let [a (m/amb 1 2 3)
b (m/? (query-fn a))]
(when-not (> b 5) (m/amb))
(m/? (goroutine a b))))Tasks and flows support failure natively so you get superv.async behavior by default, i.e. if a task process fail then the m/? will rethrow the error and you can catch it in a try block, or let it propagate to parent.
missionary's coroutine implementation doesn't have the bytecode limit problem because the elementary blocks are wrapped in functions (unlike core.async which inlines all the code in a huge case block)
Right. Thanks for providing the example, that is helpful! I think besides error handling I am interested in being able to fork sub processes in the same way that Clojure data structures and function executions on them can be forked without explicitly needing to copy or reconstruct a process. core.async-CSP cannot do that, it is inherently stateful, But missionary/FRP can from what I understand (if there are no conflicting effects).
I'm struggling to grasp your insight, could you be more specific ?
What I would kind of need is a reactive interpreter that can be forked at different stages and run in isolation (or simulation). Ideally that would be effect free, but some effects, e.g. reading from a remote database or querying the web would be fine. I think this would be a very good way to build agent based models, in particular also with LLMs. It allows me to apply the probabilistic inference concepts from Anglican to a more general setting. Anglican does this by having its own program state in a Clojure data structure and running the CPS'ed continuations on the forked program state (after sampling possible futures). I would like to lift this into FRP to operate in a continuous setting instead for just running inference in a Clojure program once.
core.async is not ideal for this, because it is stateful and I cannot easily fork go-routines. In comparison Clojure's lazy sequences can be shared between different execution contexts, because they are values and follow Clojure's persistent data structure semantics.
The same is true for Datahike/Datomic as a distributed database.
I think this memory model is still the main strength of Clojure and what sets it apart.
(the fact that it does not just enforce it through types, but actually does so by construction during the computation)
Does that make more sense? Sorry if this is a bit wide in scope.
I basically want to build an AI engine that can run social simulations with intelligent agents in a distributed environment. I think without "git-like" copy on write memory semantics this will be a mess.
when i am in a function inside of m/reduce , is it possible to make it block on a task the function is returning?No, the reducing function should not block. If you need asynchronous processing before reduction, use ap.most map/filter operations can be implemented directly in ap, if you want to reimplement the usual HOFs that's fine but IMO the direct style is more expressive in practice
(defn map-async [f flow]
(m/ap (m/? (f (m/?> flow)))))
(defn map [f flow]
(m/ap (f (m/?> flow))))
(defn filter [p flow]
(m/ap (let [x (m/?> flow)]
(if (p x) x (m/amb)))))m/eduction also works if your transformation can be expressed as a transducer
Thank you! Interesting that you prefer explicit recursion compared to higher order composition. I can see the benefits of not having to squeeze everything into a monad framework. A friend of mine works on rhine-bayes https://www.tweag.io/blog/2023-10-12-rhine-bayes/ and there it was very helpful for him to make sure he nests the effect handling between probability samplers and FRP correctly. I think this is somewhat sophisticated, but it would be nice if one could compose higher-order abstractions that way.
I watched https://youtu.be/tV-DoiGdUIo. I have build a supervison library for core.async when I created replikativ, because I found its error handling lacking, in particular if you build a distributed database that has a lot of IO. Regarding my original issue of Clojure not having higher-order abstractions that allow parametrizing different ways of "unboxing" subcomputations, I had to for instance reimplement the for comprehension macro to support declarative async computations such as shown here: https://github.com/replikativ/superv.async?tab=readme-ov-file#sequences--collections. One issue that I ran into there was that the nested try-catch blocks for the go-try abstraction blew the method bytecode limits of the JVM if you build larger comprehensions like this. My sense is that missionary is more functional than supervised go-routines of superv.async though, since go routines are processes with an identity while the FRP combinators operate on functions and hence sub expressions can be forked (as they are not identities). I am still trying to fully wrap my ahead around this and see how I can build functional simulations that can be forked like this. I am also the maintainer of https://probprog.github.io/anglican/ (but haven't done much lately), which uses Clojure's referential transparency to run particle filtering/Sequential Monte Carlo through a CPS'ed subset of Clojure. I think missionary might be a nicer way to achieve the same thing even in a distributed setting and reactively.
in a monadic environment like haskell things like this can be addressed transparently by providing a monad transformer stack that handles async. having to rewrite clojure.core functions to deal with different programming paradigms shows a bit of a limitation, i guess. i have seen that missionary has proper semi-group abstractions in some places, providing the proper monoid or group operations to apply could help to generalize this. anyway, still just poking at the basics to figure out how to write idiomatic missionary code.
Please thread follow up comments under the original post. Each post in this (and other channels) notify people who want to be on their toes about things here. While threaded comments only notify those who are following the thread (like anyone who has commented in it).
> limitation friendly challenge to demonstrate a limitation 🙂
Not sure if I was clear here. I think it is a limitation in Clojure. Transducers helped to make sequence processing cheaper, but they need to run synchronously and can not deal with asynchronous code (only with channels as inputs).
I think missionary and FRP are very cool, I am happy that this work is being done.