Fork me on GitHub
#core-async
<
2017-06-29
>
matan20:06:10

new to async and this channel..

matan20:06:55

is there anything notable built on top of async, for purposes of distributed computing? maybe like Akka, or in other veins?

noisesmith20:06:06

I don't know of anything that directly uses core.async for distributed computing, but aleph's stream abstractions which can be used for host to host communication integrate very nicely with channels

noisesmith20:06:43

pulsar is a system that provides an alternative to core.async, but iirc it can easily be integrated where pulsar is used between hosts and core.async is used within one host

noisesmith20:06:40

@matan my general feeling is that core.async is a glue - it connects things that don't share a fixed clock, but there's little need for the various things it coordinates to use it directly - and a lot of good reason for them not to rely on core.async directly but instead let you eg. throw a put! call into a callback as your coordination point

noisesmith20:06:52

and I have a strong suspicion there's never going to be a network transparent core.async, it relies on certain guarantees that don't exist in distributed setups

matan20:06:27

yeah "distributed" brings in a lot of real-world challenges

matan20:06:42

thanks @noisesmith you're truly omni-channel šŸ™‚

noisesmith20:06:21

yeah - so my inclination is to use something designed for distributed communication - eg. I use kafka which has a bunch of really great properties - and then attach the reader loops to core.async channels

noisesmith20:06:01

I'm a fairly narrow multicast, I suspect you are interested in topics I've had to implement in my clojure project šŸ˜„

matan20:06:55

I think you're just being humble šŸ˜„

joshjones21:06:18

@noisesmith @matan We are using pulsar as the distributed component in our project, which will be very large when it is released late this year. We had to implement some of the core.async protocols on top of it, but the result has been very cool so far. Create channels, send data across the interwebs using >! .. pretty neat stuff.

noisesmith21:06:07

any specific reason for treating pulsar as a channel, instead of briding pulsar's data mechanism to channels?

joshjones21:06:41

what do you mean exactly by ā€œbriding pulsarā€™s data mechanism to channelsā€?

noisesmith21:06:21

that is, taking the thing pulsar provides for receiving data, and calling a channel put! inside it for example

noisesmith21:06:43

or a go block that reads from a channel and then sends whatever it gets via pulsar

joshjones21:06:19

i havenā€™t touched the pulsar part of the code in a while, iā€™m going to look at it now to refresh myself

noisesmith21:06:27

it's an honest quesiton - I don't know enough about pulsar to know how well it matches core.async's abstractions

noisesmith21:06:51

but my first instinct with things that have weird failure modes is not to abstract over them

joshjones21:06:41

i see what you mean now ā€” well, thatā€™s a good general practice I think. But I think at the time we wanted something that would allow us to deal pretty directly with core.async and forget it was pulsar, even subbing out something else if we wanted to. Although, I will say that itā€™s probably better in general to use functions and abstract away even the core.async portions when possible ā€” but since pulsar is heavily async, core.async seemed like a good fit šŸ™‚

joshjones21:06:18

As we get further towards a release we may find that our current approach is too specifically core.async focused, and who knows, maybe weā€™ll swap it out. But it was a very cool way to get things going with the project; but in the end pragmatism will win, whatever that happens to be

noisesmith21:06:19

core.async is a good system for abstracting over, the channel io might be io, but it doesn't have weird failures you need to plaster over

noisesmith21:06:38

I don't know pulsar well enough to know whether it's as nice for abstracting

joshjones21:06:27

iā€™m not the resident pulsar expert either, but itā€™s certainly not as straightforward as core.async ā€” you have to acknowledge messages and the like