Clojurians Log v2
missionary
Hey! I really like FRP and am happy that missionary makes so much progress. I am wondering whether there are any well known production apps that use it. The README does not mention any example applications unfortunately. I know electric builds on it, but it is itself still work in progress. I have worked a lot with core.async in the replikativ libraries, and while it also has its quirks, I am fairly confident how it behaves in different application settings (e.g. memory usage, overhead, error handling).
I’m focusing on something related. We’ve just started evaluating graph databases for autonomous knowledge workers (agents). Will try to automate the dataset analysis, query generation and “extract structured data from text” parts first. Using Electric is a no-brainer for us at this point. Long term, I am very curious how the dataflow concepts in missionary can translate into reconfigurable dataflow accelerators (RDAs). Would be interesting to discuss at some point. Our leadership team has just been to Stockholm to visit AI Sweden, might open up an opportunity to collaborate with @uppfinnarjonas
Yes, that makes sense. I have a few extensions to FRP in mind (fully probabilistic/relational inference, distribution with durable indices, full resource awareness for planning etc.). I would be happy to see more work into this direction and collaborate around that.
I think a powerful way to approach an AI engine might be to start from Datalog and extend it accordingly. In particular one can add a better planner and partial evaluator to the current datahike interpreter.
Paula Gear on demoed Asami live here the other day, and the graph extensions to datalog looked very ergonomic. Very interested to hear more about your research and vision in this area. The Norwegian Digitalization department is hiring for an AI lab at moment and it might be possible to nerd snipe them with something in this general direction. I'm attending the gen AI Summit at Oxford tomorrow, rereading the AI Scientist paper to get grounded. The possibilities are really quite exciting.
electric is in production (internal business apps) and electric pushes missionary incredibly hard, far harder than a native missionary app would
electric is "wip" in the same way that the JVM is "wip"
it just gets better every year
here are some metrics on the electric tutorial app launch from last year - https://x.com/dustingetz/status/1642974354479669254
Ok, thanks! I really love your work. I am also aiming at removing the backend-frontend boundary costs with replikativ, even though from a different angle (no asymmetry between frontend-backend, supporting distributed write operations, but no distributed reactive compute graph). So I definitely see the value of what you are doing. I also recently changed datahike to use CompletableFutures, so hopefully it is compatible with missionary now. I am currently prototyping a distributed AI engine https://github.com/whilo/simmie. Ultimately I want to integrate my PhD work on compiling Clojure to diffusion models for probabilistic inference with https://github.com/plai-group/daphne, but for now I am trying to just get the compute model right so I can build a business and collect data for an AI scientist/assistant. It would be helpful to have something like the rama Mastodon demo for missionary on a single JVM machine. I looked into the electric source code, but it takes a bit of time to disentangle it from how pure missionary operates at scale. My friend at OpenAI loves the idea of applying FRP to LLM processing and I think it is perfect for "agents", I just don't have the time atm. to work through a lot of missionary issues if that makes sense. So I will give it a shot and port simmie over from CSP/core.async, I hope I don't get burned.
Btw. the Swedish government's tests on swapping in datahike for Datomic pass now and its performance is also good enough in comparison (I cannot share numbers because Datomic's license forbids publishing benchmarks). If there is anything you would need to use it instead of Datomic as an electric database, lmk!
I have seen that you also are thinking about differential data flow with missionary. We are picking up ClojureScript support for datahike again (just in-memory for now) and I am thinking about building a new query engine that is fully reactive with missionary. Ideally the indices could be shared between all peers (frontend/backend) for parts of the app data and the flows could directly hook into them through datahike's distributed memory semantics (which are fully decoupled readers on snapshots).