Fork me on GitHub

@mv We have two Clojure engineers and two JS engineers, plus a UI/UX engineer.


does anyone know some good resources about designing and managing asynchronous system, especially about backpressure and monitoring?


@chrisblom I’m also interested, I looked for Clojure in relation to reactive libraries and saw quite a few, also with some work you could use core.async for the purpose of backpressure.


cool, i've used core.async in the past, but i find it a bit magical and hard to monitor


i look at which libraries to use etc later, i'd like to form a high level overview first.


@chrisblom I would look at this: then read the books recommended by Zach Tellman

simple_smile 4

thanks, that looks like exactly what i'm looking for


Ha, I just posted this same video in response to the question in #aleph and then saw it over here 🙂


It also depends what’s the use case, but for example it would be nice to have a reactive kafka library for clojure, where for example you could determine maximum the amount of messages ‘in flight’. Such that when that limit is reached it stops consuming.


I don’t know what’s possible in #onyx in relation to this


previous versions onyx stopped consuming when a max no. messages are in flight


Nice, maybe I look to #onyx again, it was a bit to complicated to quickly set up.


Hm, I remember that a lot of servers in clojure land dont handle backpressure well. IIRC there was a server designed for exactly this purpose, but I just cannot recall the name.


was it aleph?

❤️ 4

Yea, could be, its definitly worth a look.


@chrisblom I found this fairly recent post from the Wallaroo Labs folks to be a nice overview of handling overload, specifically:


Hi Dave! I hope you are doing well. Overall I liked the article but for me it does reiterate a common failing with Docker orchestration solutions, namely the requirement to have the same number of containers replicated across all your nodes. If you could have an uneven amount, then you could shift your resources towards a service that was having trouble. For example, if you have three services A, B, and C, where A feeds B and B feeds C, and C is currently underwater (its queue is getting backed up faster than it can handle the messages) you could spin up more instances of the container for C hopefully increasing its throughput. However, with many of the Docker orchestration tech that I’ve seen, you’d have to spin up an equal number of A, B, and C containers, which could easily increase the rate of messages into C only exacerbating the problem. In any event, this is great stuff to talk about, thanks for posting the article.


not Clojure-related, but general principles


good links at the bottom too (including one to that Zach Tellman talk)


anyone else watching Zuck's testimony?