Hi! New here. New to core.async.

Can anyone see what’s wrong here:

(go
  (<! (clojure.core.async/timeout 7500))
  ;; Handling in separate thread, making sure to log exception
  (taoensso.timbre/logged-future
    (handle-updates param1 param2)))

Sometimes the work going on inside of handle-updates kill my app for many seconds (rendering it unavailable), until the underlying thread dies. I get an OutOfMemoryError: Java heap space.

I know it may be difficult to answer right away, but since I’m new to core.async I hope I’ve just missed something basic.

my guess is you aren't rate limiting or providing back pressure

My guess is you’re right. Unfortunately I have no clue what this means 🙂

you are doing stuff in a loop as fast as possible, and nothing in the loop is throttling or blocking

like (while true ...your example code...) would generate go blocks and futures as fast as the cpu could handle it with no brakes

also creating a go block, then creating a future is kind of odd

I guess you’re right.

(logged-future (Thread/sleep 7500) ...) or something like it would be better

The reason I create the future is because I want to get it out of core.async, that - as I think I’ve understood - have a limited amount of ressources.

if handle-updates is blocking on io, then sure, but if handle-updates is cpu bound, then running it in a go block likely to be fine

To provide some context: This is actually part of a throttling function. Updates come in. I use an atom to store the updates. If the atom is empty, I start the go block above. Then when the timeout has been reached, I read the value of the atom, reset the atom, and pass the value to the handle-updates function.

handle-updates stores a single update in database

so if e.g. 10 updates come in during the 7500 ms, only the newest is stored.

(in the mean time, while the go block still run, I add new updates to the atom)

As a side note: I haven’t seen the OOM before I started sending the updates to the database. If I just logged the newest update, the OOM didn’t occur.

(I use Datomic BTW, it's the peer library of Datomic that throws the OOM - but only after a while, not immediately - so I think you’re right about the creation of many go loops or similar - I just can’t find the cause, and I don’t use go loops further down the stream)

what I would do is run a single thread (using the thread macro or logged-future) that loops over a channel, and reads from the channel and writes to the database

make that channel use a windowed buffer or whatever, and only do the loop every seven and a half seconds

so now instead of spinning off all these waiting go routines and futures, you have one reusable io thread

I’ll try that! Thanks! I may have difficulties getting it to work, but hey, that’s how to learn stuff 😉

Now I have some pointers to go after. Appreciate it.