anyone: what is the appropriate way to wait for all or a group of go processes to finish?

Looking for something akin to the WaitGroup functionality in golang

ajmagnifico: that sounds similar to the problem jeremyraines just asked about, above. See his gist for an example of composing merge and into.

ah, thanks! I'll check it out

anyone know if there plans to implement a blocking version of async/pipe?

http://dev.clojure.org/jira/browse/ASYNC-65 Rich has vetted that. It would enable you to do what you need

well i’m looking for something that’s not inside of a go block

but in a thread instead like async/pipeline-blocking

not sure i follow you. pipe is an operation from channel to channel. it doesn't do any work besides scuttle data from in to out

you’re right

i was using pipelines and transducers and was looking to convert that to channels w/ transducers and pipe instead

as i don’t need the parallelism

does this make sense or am i thinking about this the wrong way?

i make 3 network calls in my chain and originally did this explicitly in threads

then i changed it to use blocking pipelines which create their own threads

after realizing i don’t need the parallelism, i decided to try putting the transducers in the channels and using pipe instead

but now i don’t get the threads i need

@gabe: maybe a slight aside, but my new favourite way of doing multiple async calls and gathering results is applicative-do syntax (alet) - http://funcool.github.io/cats/latest/#channel - if you use an async network client you don't need to create any threads at all (beyond the client's service pool)

not much use if you can't use an async client tho

that’s where i’m stuck right now (unfortunately)

using https://github.com/mcohen01/amazonica

and he isn’t wrapping the async API

@gabe: another option would be to use https://github.com/ztellman/manifold which will manage threads for you and plays nicely with core.async

@gabe: the relevant concept being the deferred - https://github.com/ztellman/manifold/blob/master/docs/deferred.md

That IS interesting. Thanks!

that’s kinda of how i would do it in the JS world

but less verbose

@gabe: i'm using it in a system with mixed core.async and manifold async components and it's been working great

i might have to consider it. it could really cut down on the explicit use of channels

core.async IS a “low-level” abstraction after all

if you do use it, you can go all high-level with applicative-do syntax too - http://funcool.github.io/cats/latest/#manifold-deferred

thanks for the knowledge man

np... i've been learning myself in the last few months

@mccraigmccraig: you’ve given me some good light reading to do 😉

@erik_price or anybody: Yesterday I asked about waiting for go processes to finish. Here's something I came up with.

Would appreciate comments from anybody

ugh

I think a simple (merge) would probably do the same thing for me

😛

OK, here's a more straightforward question: is there API for determining if a channel is closed? If it still has values available for taking?

Or, given a channel that resulted from a (go) call, is there a way to distinguish between nil (channel closed, no values to take) and nil (`(go (println "hey"))` returned nil) ?

generally inspecting a channel is race-condition prone. If you ask it if it's closed, and it says no, it may be closed by the time you act upon that information

I see.

There is an api for it. I think it's not generally useful.

but to answer your question, re nil, you can't tell the difference.

More generalizations/pontification:

you shouldn't care what's on the other side of a channel.

there isn't any difference is there ? the result of (go (println "hey")) is a closed channel with no values to take

That's right, no difference.

OK

let me get at the real issue that I'm struggling with:

How can I ensure that all go processes have completed before exiting a program?

In golang, you've got WaitGroup, etc.

Is there something equivalent in core.async?

And actually, it's not even just exiting a program.

there might just be times when I want all go processes to complete before proceeding with the remainder of the program.

channels of channels 😃

OK

that brings us to (merge), right?

I wrote this.

It turns out it's mostly equivalent to (but probably less robust than) (merge [channels ...])

merge can get you there.

I take it that what you are waiting on is heterogenous tasks?

not just a pool of identical things

if you have identical things, pipeline is what you want

OK

I noticed that and thought maybe it would be helpful to me.

But what if it were heterogeneous tasks?

otherwise, at a high level, you just gotta remember what you launched off and wait for the channels that represent those tasks

So if you have a channel, from merge or from pipeline or whatever,

nevermind, I think I'm asking the wrong question

("How do you know when there are no more values.")

Golang's waitgroup stuff warns: Typically this means the calls to Add should execute before the statement creating the goroutine or other event to be waited for.

I think I need a paradigm shift!

this stuff is tough.

part of my problem is that when I'm incrementally writing experimental code in emacs/cider and trying to test this stuff out, the full expression typically ends before the go processes have completed.

And I get bogus results.

The await-go-processes function that I wrote keeps track of all the channels returned from (go) calls, and alts!! them until all have returned.

This way I know that all the processes have completed

K, I'll keep thinking about it

thanks @ghadi

again

no prob. Wish I had a silver bullet

I think I just need to change the way I think these things