Clojurians Log v2

(I added a brief note on reliability under Rationale.)

Looking into muutos, looks great. Quick question if I may. I’m looking to enable server sent events for users, and the events are trigered on update to a particular table. Different SSE sessions for different users will be connected to different K8s pods in my architecture. In this design, do I need to create a replication slot per pod (e.g. if I have 3 pods I need 3 slots, so the particular pod that a user happens to be connected to, can see the updates of interest and ignore/ack the rest) - or is there some other pattern that I haven’t thought of that works for this sort of architecture?

Having a replication slot per pod is one option. However, it's not necessarily very scalable (what if there's a traffic spike and you need more than three pods, for example?). Also, three (permanent) replication slots will obviously consume more disk space and other resources on the database server. Another option would be to put a separate pod running Muutos between the database and the web app pods, and have the Muutos pod propagate changes to the web app pods. The method of communication between the Muutos pod and the web app pods depends on your existing infrastructure. For example, if you're already running some sort of message queue, you could use that. Otherwise, you could look into using SSE there as well, or have the Muutos pod https://clojure.org/reference/repl_and_main#_launching_a_socket_server and ship EDN or Transit to the web app pods, or some other method. In any case, you might want to look into whether https://www.postgresql.org/docs/current/functions-admin.html#PG-CREATE-LOGICAL-REPLICATION-SLOT are suitable for your use case (to moderate resource use on the database server). Also, you'll probably want to https://www.morling.dev/blog/mastering-postgres-replication-slots/#_use_column_and_row_filters at the publication level (exclude irrelevant columns and rows).

Thanks for the comprehensive reply, much appreciated.

If you decide to proceed with Muutos and hit any roadblocks (or successes 🙂), I'd be interested in hearing about them!

so you are speaking pg wire protocol directly? (zero deps would suggest that)

Yes.

nice, this will come in handy with Ripley

Very interested in hearing how it goes! And if you hit any roadblocks.

the main thing of interest is how fast the replication callbacks come... like is it fast enough for "real time" work

It's a real Postgres client written from scratch. Cool!

PostgreSQL sends the logical replication message immediately once a transaction has successfully committed (and only if it does commit successfully -- if it is rolled back, PostgreSQL doesn't send a message). I don't think there's a faster approach for real-time work.

Also, pgoutput emits binary data, so decoding it is quite fast.

> It's a real Postgres client written from scratch. Cool! It's certainly drawn inspiration from pg/pg2, so kudos for making them. 🙂 Before seeing them, I didn't know anything about the PostgreSQL wire protocol.

is it a replacement for Debezium?

Certainly not a replacement for Debezium proper, since Debezium includes/requires Kafka Connect and has way more features (and introduces way more operational overhead). 🙂 Perhaps a (Clojure-oriented) alternative to https://debezium.io/documentation/reference/stable/development/engine.html. Check out the documentation for more details.

I see Muutos more as a building block in your own change data capture solution.

dude amazing

the pg wire protocol is actually pretty nice as binary protocols go, I recently did stuff with it for xtdb 2

Definitely, no complaints. 👍

nice, we have a use case where we need to sync a PostgreSQL table to an ElasticSearch index - so this seems like a good fit. what happens to the events if you client is down ? If I remember my PostgreSQL logical replication, the WALL files (and events) should pile up on disk untill they are processed. If this is true, it means you should not lose any events / data. Can you please confirm? This info would best be put in the docs - I think it's important.

(reading the rationale now - it might be explained there 😄 )

nice, we have a use case where we need to sync a PostgreSQL table to an ElasticSearch index - so this seems like a good fit.Yes, this is precisely the sort of problem Muutos is designed to solve. Making any kind of derived view into the data in your PostgreSQL database, informing downstream consumers (e.g. via a message queue), etc. > what happens to the events if you client is down ? PostgreSQL retains WAL entries until the replication slot consumer (i.e. Muutos) has sent it an acknowledgement message saying that it has successfully processed them. That happens when you call the ack function that is the second argument to the handler function you give Muutos. > If this is true, it means you should not lose any events / data. Correct, you will not lose any data. This is in contrast to LISTEN/NOTIFY. If you NOTIFY but no one's LISTENing, PostgreSQL will discard the message. > This info would best be put in the docs - I think it's important. I don't recall whether I mention that explicitly in the docs. The docs might assume some sort of familiarity with logical replication. I'll double-check when I get the chance -- thanks for the suggestion. 👍

The blog post has a lot more detail: https://blog.nilenso.com/blog/2025/10/29/fight-context-rot-with-context-observability/

Given that nilenso is fairly Clojure-centric as a company (at least, that's my impression from libraries and posts here), I'm curious why you went with TS?

Ohey @seancorfield, good question. Yes, we still write quite a bit of production Clojure at nilenso. • I wanted something browser-first, and without a backend component, for privacy sake. So it had to be something that compiled down to JS. • Secondly, from what I’ve seen, LLMs have been much better at generating TS when compared to clojurescript. • Thirdly, this is a bit of an exploratory project. If I were to do this seriously, and for scale, I would reexamine my tech stack.

I’ve managed to capture almost all my prompts https://github.com/nilenso/context-viewer/blob/main/docs/prompts.md, so I might just indulge in changing the https://github.com/nilenso/context-viewer/blob/main/docs/tech-stack.md, and redoing all this with cljs 😁

Maybe a good opportunity to try Squint? 😄

+1 for implementing in ClojureScript (but I’m biased).

> • I wanted something browser-first, and without a backend component, for privacy sake. So it had to be something that compiled down to JS. Why browser-first? Is it because of familiarity with the tech stack? What about a desktop app? > Observability for contexts. Given a coversation log (messages), this tool will provide a breakdown of its components and their sizes. What do you mean by size? Is that just token count or are there other measures? I believe token counts can be done locally without a provider.

> Why browser-first? Is it because of familiarity with the tech stack? What about a desktop app? Well, I just wanted to prioritise the user interface, because it’s important for observability to be able to visualise the slices and dices. I guess it could be a desktop app too. > What do you mean by size? Is that just token count or are there other measures? I believe token counts can be done locally without a provider. Yeah, token counts are done using tiktoken, no AI there. It’s the “pulling apart” that needs some AI to chunk all the text in the context to meaningful components. That blog post I linked to above has some videos that show what it can do.

oh ok. I'll check out the blog post 👍 . I suspect you could also do that locally if you have a GPU (or are verrry patient).

Ha yes, I’ve mentioned that you could definitely run this against a local model. It doesn’t need very fancy big model skills. gpt-oss models might suffice.

@blueberry have you tried it out on your machine?

reminder

cruel shifts as a barista 🥲

Here you go @blueberry. Sorry that it's a zip file and not some repo, I don't know how ok it is to upload the onnx model and video file used for the example to github. Nix flake included, even though you probably won't be able to use it since cloudflare is down and cachix relies on that. Steps: 1. Jack in with cider 2. Eval example.clj file 3. Eval the expressions in the comment form step by step 4. There's a comment saying when to run inspect and what exactly to click in the inspector window. Hopefully you'll be able to reproduce it on your machine!

When I tried to inspect the seemingly "2 values" output from the function that is returned when calling the network on input-tz, REPL crashed. Is that because it's huge, or because it's actually 2 values? how even

Apparently that's how it prints tensors: a map with some arbitrary amount of values after it

Can you share a minimal project with example code that crashes the repl, so I can test it and see what might be the issue?

yes, when i get home