This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-08-28
Channels
- # aleph (1)
- # announcements (16)
- # bangalore-clj (1)
- # beginners (78)
- # cider (109)
- # clara (3)
- # cljdoc (6)
- # cljsjs (3)
- # clojure (209)
- # clojure-dev (11)
- # clojure-europe (1)
- # clojure-france (9)
- # clojure-italy (13)
- # clojure-nl (3)
- # clojure-spain (2)
- # clojure-spec (19)
- # clojure-uk (50)
- # clojurescript (41)
- # clojutre (2)
- # core-async (45)
- # cursive (2)
- # datomic (14)
- # emacs (6)
- # figwheel-main (1)
- # fulcro (101)
- # graalvm (1)
- # graphql (3)
- # jobs-discuss (3)
- # kaocha (12)
- # leiningen (8)
- # music (4)
- # off-topic (47)
- # parinfer (8)
- # pathom (17)
- # pedestal (53)
- # re-frame (47)
- # reagent (22)
- # reitit (4)
- # shadow-cljs (49)
- # tools-deps (87)
I need to use a chan that expects n
values put on it then closes. n
will usually be small, less than 10, but in certain scenarios will be a few thousand messages that come in a few milliseconds, which I've found is impossible to for any consumers to keep up with before hitting the pending put!
limit.
What's people's opinion on the best way to address this?
Just use a fixed buffer of a number large enough to definitely account for all cases of n
?
From what I gather from quickly skimming the async source, a fixed buffer doesn't actually allocate anything, so having a very liberal fixed buffer for chan of say 1million even when most of the time it will only buffer one or two items tops wouldn't actually be a big inefficiency right?
Alternatively I could implement an unlimited buffer, but I'm thinking that would probably be worse
but if you know ahead of time a fixed number of messages will be put on a channel, it is often best to create the channel with a buffer that size
a very common case of this is, if you know you will only ever put one item to a channel, create the channel with buffer size 1, you'll see this used in a number of places in the core.async source
But if you are hitting the limit on puts, that is usually an indication of a bad pattern
and sometimes i can know the exact number of expected messages but there will be times when I can guestimate at best
if you are on clojure, switching to >!! will likely superficially solve your problem
I think I can get fancy, and when possible choose an exact buffer size needed, or in the case of the large burst messages, choose the minimal buffer size that allows the messages to be kept up with. But yeah I was just wondering if there's a reason to implement the infinite buffer instead of just choosing a fixed buffer size that's effectively infinite?
there almost certainly is some kind of backpressure mechanism in whatever way the api is pushing data to you
Are you saying the infinite buffer is preferable because the fact that a fixed buffer doesn't currently allocate anything proportional to its buffer size is just an implementation detail and may change in the future?
but using put! without a callback, or something else, can break the chain of backpressure
the buffer you're hitting is not the channel buffer though and is not configurable
if I understand correctly
yeah, I get that there's no way around that, I need to either implement a unlimited buffer or chose an effectively infinite fixed buffer size
it's for a wrapper of a java stock broker api, most of the request types return a single message, but you can request historical data which can return like 10k messages, and unfortunately they all come in one burst and the api provides no mechanism of control for this
core.async isn't a perfect fit for this use case I know but currently I'm using my own hand rolled combo of atoms and promises and I'm starting to thing bending core.async to my will would be better than managing all the concurrency myself
but somewhere you have a single bit of code that is taking those messages and putting them into a channel
and you are losing the backpress right there, you should not move on to put in the next thing until the current thing has been accepted
ah! this makes sense. so maybe doing something like put!
ing the next item in the callback of the first put!
ah ok cool. I was pretty sure that I tried both >!!
and put!
and was getting the pending puts exceeded error message either way, but I don't have the example immediately in front of me
using >!! will block that single thread, so if you have a single loop taking messages and putting them in to a channel that will work, if you have multiple concurrent places that is happening you can still run into the issue with >!!
for simple cases, using >!! will often solve backpressure isssues, for more complicated cases put! with a callback, and of course a lot of example docs show using put! without a callback which is asking for a backpress issue
> which is asking for a backpress issue
Do you mean put!
without a callback is asking for issues caused by not exerting backpressure? i.e. you’re at the mercy of the system incidentally rather than explicitly. i.e. enforcing backpressure through a lack of resources, such as through the 1024 limit, memory leaks due to excessive spawned go
blocks etc?