Fork me on GitHub
#core-async
<
2019-04-02
>
idiomancy00:04:35

Yeah, I'm probably going to be building something that uses a/map to implement something that has the semantics of mix, because you can't exactly add inputs to a map, although you can create another map that is composed of submaps

jjttjj15:04:01

I am trying to a/put! all incoming messages from an external API onto a chan. Ususlly consumers have no issues keeping up with producers but occasionally there will be huge bursts of 10k messages from a single producer. I cannot discard any of these messages. What are the usual strategies for dealing with this? Just make my buffer size large enough to account for the largest burst? Implement some sort of logic to group the messages prior to the put!?

jjttjj15:04:38

the typical amount of messages I'm dealing with is less than 10 per second, and the 10k bursts are rare but predictable, so it doesn't feel quite right to just set the buffer size to 10k, but I could be wrong

bortexz15:04:48

I would just do that and have a big buffer to account for the bursts, but > Implement some sort of logic to group the messages prior to the put!? depending on how you are implementing the producer. You mentioned is an external API, are you requesting from it? Or your program has the API and each request gets somehow feeded into core.async?

jjttjj15:04:42

I am requesting from the API, and it answers asynchronously

bortexz15:04:28

How do you receive the answers?

jjttjj15:04:55

on a socket. So I just wait for things on the socket and put them on a chan

jjttjj15:04:16

I should also note that this is an attempt at a library, so I'm hesitant to make the decision for end users regarding buffer size when only certain types of requests require a large buffer for responses. So I guess the best way around this is to allow the users to just pass the whole chan they require themselves with whatever buffer requirements they need

šŸ‘Œ 4
jjttjj15:04:37

I just wasn't sure if there was some fancy buffer strategy for these type of bursty situations

bortexz15:04:56

Not that I am aware of, I can think of partitions, debounces, and thinks like that, but are more complex than just a big buffer. Iā€™d go with accepting chan as parameter and let the user choose

jjttjj15:04:31

ok great, thanks!

idiomancy18:04:14

wait.... why doesn't this work? when are transducers on channels applied?

idiomancy18:04:17

(go (>! (chan 1 (map (fnil identity ::NOTHING))) nil))

idiomancy18:04:33

that returns a "can't put nil on a channel" error

idiomancy18:04:50

but it seems like it should definitely not do that

Alex Miller (Clojure team)18:04:32

nils are not valid channel values

Alex Miller (Clojure team)18:04:14

you're putting nil on the channel

Alex Miller (Clojure team)18:04:05

transducers are applied after you put it on the channel and before you take it off

Alex Miller (Clojure team)18:04:35

with some intentional hand-waving about whether it's the producer or consumer thread (as both can occur)

idiomancy18:04:38

after I put it on the channel and before I take it off? So, does that mean its applied as at the beginning of the put take operation?

idiomancy18:04:25

how about in cljs?

ghadi18:04:32

Same deal.

ghadi18:04:51

A user of a channel knows not that there is a transducer

idiomancy18:04:10

hahaha, so timing wise, does it occur during the take operation?

ghadi18:04:50

it happens during put

idiomancy18:04:58

but after the put

idiomancy18:04:16

so its part of the put operation, and it happens after its already on the channel

ghadi18:04:19

Sorta yeah. The transducer might be a filter that avoids calling the internal put

ghadi18:04:26

So it can't happen during take

idiomancy18:04:30

as I just verified

idiomancy18:04:50

I was using it to avoid calling the internal put, but that does not work

idiomancy18:04:47

if that was so, this would work (go (>! (chan 1 (map (fnil identity ::NOTHING))) nil))

idiomancy18:04:16

i meant this:

idiomancy18:04:42

(go (>! (chan 1 (filter some?)) nil))

ghadi18:04:15

you can't >! or put! nil, fundamentally

ghadi18:04:35

no matter what the channel arg is

ghadi18:04:41

it's an invariant of the system

idiomancy18:04:52

right, but if the filter occurred before the put operation, then it would never reach that invariant

idiomancy18:04:18

my expectation is that it simply would never reach out to the transducing process (the put operation)

idiomancy18:04:09

hmm. I mean I guess you could just actually use the put operation as the terminal step in a transducer, then...

ghadi18:04:03

you could do a lot of things šŸ™‚

idiomancy18:04:38

hahaha, that you could! thanks for the clarification, its really helpful