new to async and this channel..

is there anything notable built on top of async, for purposes of distributed computing? maybe like Akka, or in other veins?

I don't know of anything that directly uses core.async for distributed computing, but aleph's stream abstractions which can be used for host to host communication integrate very nicely with channels

pulsar is a system that provides an alternative to core.async, but iirc it can easily be integrated where pulsar is used between hosts and core.async is used within one host

@matan my general feeling is that core.async is a glue - it connects things that don't share a fixed clock, but there's little need for the various things it coordinates to use it directly - and a lot of good reason for them not to rely on core.async directly but instead let you eg. throw a put! call into a callback as your coordination point

and I have a strong suspicion there's never going to be a network transparent core.async, it relies on certain guarantees that don't exist in distributed setups

yeah "distributed" brings in a lot of real-world challenges

thanks @noisesmith you're truly omni-channel 🙂

yeah - so my inclination is to use something designed for distributed communication - eg. I use kafka which has a bunch of really great properties - and then attach the reader loops to core.async channels

I'm a fairly narrow multicast, I suspect you are interested in topics I've had to implement in my clojure project 😄

I think you're just being humble 😄

@noisesmith @matan We are using pulsar as the distributed component in our project, which will be very large when it is released late this year. We had to implement some of the core.async protocols on top of it, but the result has been very cool so far. Create channels, send data across the interwebs using >! .. pretty neat stuff.

nice!

any specific reason for treating pulsar as a channel, instead of briding pulsar's data mechanism to channels?

what do you mean exactly by “briding pulsar’s data mechanism to channels”?

that is, taking the thing pulsar provides for receiving data, and calling a channel put! inside it for example

or a go block that reads from a channel and then sends whatever it gets via pulsar

i haven’t touched the pulsar part of the code in a while, i’m going to look at it now to refresh myself

it's an honest quesiton - I don't know enough about pulsar to know how well it matches core.async's abstractions

but my first instinct with things that have weird failure modes is not to abstract over them

i see what you mean now — well, that’s a good general practice I think. But I think at the time we wanted something that would allow us to deal pretty directly with core.async and forget it was pulsar, even subbing out something else if we wanted to. Although, I will say that it’s probably better in general to use functions and abstract away even the core.async portions when possible — but since pulsar is heavily async, core.async seemed like a good fit 🙂

As we get further towards a release we may find that our current approach is too specifically core.async focused, and who knows, maybe we’ll swap it out. But it was a very cool way to get things going with the project; but in the end pragmatism will win, whatever that happens to be

absolutely

core.async is a good system for abstracting over, the channel io might be io, but it doesn't have weird failures you need to plaster over

I don't know pulsar well enough to know whether it's as nice for abstracting

i’m not the resident pulsar expert either, but it’s certainly not as straightforward as core.async — you have to acknowledge messages and the like