Fork me on GitHub
#core-async
<
2017-08-30
>
colinkahn06:08:27

I've got a function that mimics the RxJS startWith method, but wondering if there's a built in way to start a channel with a value:

(defn start-with [ch v]
  (a/put! ch v)
  ch)

mpenet06:08:34

@hiredman sounds like credit based flow control could help

mpenet06:08:09

It s quite common when you can t (or dont want to) block (ex in erlang where process mailboxes size is unbounded, so msgs producers have to be carefull not to flood their consumers mb)

mpenet09:08:22

Sadly a lot of ws lib for clojure ignore these problems

tbaldridge17:08:43

@rplevy this is why I strongly recommend not using push-based subscriptions with web apps.

tbaldridge17:08:06

the protocols in play just don't support good rate limiting and flow control. A much better option I've found is to only send "most recent message" IDs to the app. These messages can then be thrown away via a sliding window. Then the webapp can make a pull-based request to the back end for a range of messages.

tbaldridge17:08:38

This then allows the client to know how far behind "current" it is, and the client also then has the option to do batching, or to throw out entire groups of messages if it gets too far behind.

tbaldridge17:08:56

With push-based protocols you're limited to drinking out of a firehose that the server hands you.

tbaldridge17:08:55

Also, push-based approaches are really hard to write in a way that they properly handle slow clients. I've seen systems where all the clients on a single back-end server are limited to the same rate. I've also seen systems that cause OOM errors on servers due to too much buffering by a slow client.

tbaldridge17:08:10

You can get rid of all those problems by simply using a pull model.

plins21:08:22

does anyone has experienced problems using core.async with the jdbc/with-db-transaction macro? no matter what i do i keep getting Exception in thread "async-thread-macro-18" java.lang.Error: java.sql.SQLException: No operations allowed after statement closed.

tanzoniteblack21:08:11

@plins with-db-transaction closes the connection at the end of the macro, and isn't aware of the fact that code is using it from another thread

tanzoniteblack21:08:00

so if you hand off that connection to another thread that processes async, like core.async, then that will always happen if the macro block is exited from before that core.async block has finished running

tanzoniteblack21:08:47

best solution: don't use core.async with jdbc, because jdbc isn't async (and therefore you're going to block your core.async threads anyways)

noisesmith21:08:23

you can still use jdbc inside async/thread - sometimes that’s useful

tanzoniteblack21:08:25

other solution: initialize the with-db-transaction block inside a core.async go block and then park on waiting for everything that depends on that transaction to have finished

tanzoniteblack21:08:32

and like @noisesmith said, make sure you run your actual DB commands a separate thread (probably via core.async/thread) so you don't block the small core.async thread pool

noisesmith21:08:43

thread lets you coordinate asynchronously with db results while doing other stuff, which isn’t as good as having an async db driver but hey that’s jdbc’s problem fundamentally

plins21:08:48

here is what happening: i have a db-layer/dao with lots of small functions, all of them encapsulates the queries with async/thread on top of it, theres a service/controller, which calls (should) start a transaction and call these functions

noisesmith21:08:23

is there any reason not to scope the transaction to the thread body?

noisesmith21:08:34

or do you mean there’s more than one?

plins21:08:45

i tried to (async/thread (jdbc/with-db-transaction`

plins21:08:56

still having problems

plins21:08:04

let me go for the stack trace just one sec

noisesmith21:08:06

if you aren’t doing async inside the transaction that should be fine…

plins21:08:17

(async/thread
  (jdbc/with-db-transaction
    ....
     (async/thread ;; inside a function by the way
so this is problematic

plins21:08:51

(async/thread
  (jdbc/with-db-transaction
    ....
     (function-which-calls-async/thread
this is clearer

noisesmith21:08:54

if you don’t block on the result of the inner async/thread call, yes it is a potential problem

plins21:08:08

ok i think i got it

noisesmith21:08:09

because the function can return before the thread finishes

plins21:08:42

thank you very much for the detailed explanation everyone 🙂