Fork me on GitHub
#core-async
<
2018-05-21
>
bjr15:05:11

@alexmiller is there a talk or blog post going deeper into the lack of core.async monitoring features? my understanding is if you’re trying to inspect/monitor the contents of the provided buffers in production you’re doing it wrong…but I still feel an itch to monitor buffer contents

bjr15:05:53

(doing it wrong because the backpressure semantics are all the feedback you should need)

Alex Miller (Clojure team)15:05:12

I believe some people have built monitoring buffers but I don’t have a project name at hand right now

Alex Miller (Clojure team)15:05:26

it is something I think core.async should do more with

Alex Miller (Clojure team)15:05:05

that is, there is no reason monitoring is NOT included other than that it hasn’t been included yet

bjr16:05:49

Oh, interesting. This is the example I’ve seen: http://tgk.github.io/2013/10/inspect-core-async-channels.html

tbaldridge16:05:19

@bjr it depends on what you want, but you can get a lot of milage via transducers on channels. Doesn't work if you want to monitor the contents of the buffer, but a map transducer on a channel can easily be used to monitor throughput.

bjr16:05:12

I’m not sure what I actually want here - but I never have convincing answers to questions like @petrus’s likely follow-up “why not?”

ghadi16:05:56

monitoring is tricky because of the channel locks

bjr16:05:01

Channel throughput, last n inputs and last n outputs would be easy enough with transducers.

Petrus Theron17:05:48

Can someone show me how to tap into the last 30 values on a channel using transducers without any obvious pitfalls? I looked at the PersistentQueue impl. and seeing what it does with a seq + vector to get performance, I didn't want to add new slowness. I only periodically care about the last N values for plotting, where N is a small number (30-1000).

bjr12:05:11

I would put a ring buffer [1] in an agent and have a transducer send send inputs to the agent. I’m sure that comes at some cost, but using an agent should minimize the cost felt by the channel. [1] https://github.com/amalloy/ring-buffer

bjr16:05:54

Anyway, this has always been a big source of friction when teaching core.async. The best way I know to show someone “what’s happening” if they cannot (yet) visualize what the code is doing is to add lots of logging.

tbaldridge16:05:05

I agree, it's a source of friction. The problem in giving a good answer is a combination of a few things. * The API for buffers is unpublished, and the locking mechanics around these buffers are not defined in public documentation. * The buffer implementations are not thread safe * The buffer contents can change at any time, so the usefulness of monitoring the contents has limited use outside of debugging * The current implementation of channels would require locking the channel to get read access to the buffer, so a read would require a lock that would block work (monitoring could have a nasty impact on performance) on the channel.

👍 4
tbaldridge16:05:13

So I think the reason this always gets a non-answer, is that there really isn't an answer. There's multiple solutions that could work, but they'd all need some new features in core.async.

4
noisesmith17:05:49

any logic other than simple logging or metrics that relied on knowing what's in a buffer would be broken

👍 4
bjr17:05:55

That’s a really good point. The line between using an introspection feature purely for monitoring vs for other execution may not be obvious to beginners.

noisesmith17:05:44

I've seen people make this mistake who would never describe themselves as "beginners"

bjr17:05:05

Well…there’s no avoiding the spiderman principle

noisesmith17:05:32

speaking of "great power" and "great responsibility", there's also a solid precedent of not introducing features or capabilities into core language or libraries that will be built in footguns, which could explain the buffer api being private and not containing introspection of contents

bjr17:05:52

yeah, that is fair. that’s why my assumption has generally been that introspection features are not there because they’re a bad idea. generally I find clojure core and its libraries quite reliable on that front.

bjr17:05:23

maybe there’s a relatively controlled feature space where limited transparency can be provided

bjr17:05:54

or..i’m wondering what can be done in user space

noisesmith17:05:29

it would violate the constraints of CSP, and if you don't want that there's simpler ways to do racy async

leonoel17:05:51

@noisesmith so attaching a transducer to a channel violates CSP ?

noisesmith17:05:44

Not inherently, but using it to track buffer contents then having logic that relies on that within csp (not just logging/metrics) does.

noisesmith17:05:20

If what you do to one channel depends on your knowledge of what's in another channel's buffer, you're asking for trouble.

leonoel17:05:01

I couldn't agree more

hiredman17:05:08

I don't think I mentioned it here, but I have a patch up (https://dev.clojure.org/jira/browse/ASYNC-184) that should fix most try/catch/finally issues anyone is experiencing using core.async in clojurescript, it would be great to get some eyes on it, I am not very familiar with clojurescript. the patch is a port of the fixes that went in to the clojure side a while back + a new fix for an issue opened since then (there is another ticket with a patch for the clojure side)

👍 8
Alex Miller (Clojure team)17:05:02

which one is the CLJ side?

benzap23:05:11

In regards to the core.async channel monitoring, wouldn't it be trivial to pipe the channel through a diversion for monitoring? I'm sure someone has written a library that does this already. Sort of like a man-in-the-middle attack, but with core.async channels 😄

noisesmith23:05:58

right, that's what's being suggested with transducers

👍 4
tbaldridge23:05:49

it doesn't really tell you how many items are in the buffer however, and doing that for every channel involves a fair amount of overhead as every handoff through a channel involves 3 locks, and two threads synchronizing. So it depends on the use-case.

tbaldridge23:05:28

But I'd love to see some sort of "free" deref of a channel to get some statistics about it.

Petrus Theron17:05:48

Can someone show me how to tap into the last 30 values on a channel using transducers without any obvious pitfalls? I looked at the PersistentQueue impl. and seeing what it does with a seq + vector to get performance, I didn't want to add new slowness. I only periodically care about the last N values for plotting, where N is a small number (30-1000).