Fork me on GitHub
#core-async
<
2015-10-08
>
ajmagnifico00:10:29

anyone: what is the appropriate way to wait for all or a group of go processes to finish?

ajmagnifico00:10:39

Looking for something akin to the WaitGroup functionality in golang

erik_price00:10:47

ajmagnifico: that sounds similar to the problem jeremyraines just asked about, above. See his gist for an example of composing merge and into.

ajmagnifico00:10:10

ah, thanks! I'll check it out

gabe16:10:08

anyone know if there plans to implement a blocking version of async/pipe?

ghadi16:10:08

http://dev.clojure.org/jira/browse/ASYNC-65 Rich has vetted that. It would enable you to do what you need

gabe16:10:11

well i’m looking for something that’s not inside of a go block

gabe16:10:31

but in a thread instead like async/pipeline-blocking

ghadi16:10:31

not sure i follow you. pipe is an operation from channel to channel. it doesn't do any work besides scuttle data from in to out

gabe16:10:21

you’re right

gabe16:10:22

i was using pipelines and transducers and was looking to convert that to channels w/ transducers and pipe instead

gabe16:10:33

as i don’t need the parallelism

gabe16:10:38

does this make sense or am i thinking about this the wrong way?

gabe16:10:15

i make 3 network calls in my chain and originally did this explicitly in threads

gabe16:10:41

then i changed it to use blocking pipelines which create their own threads

gabe16:10:09

after realizing i don’t need the parallelism, i decided to try putting the transducers in the channels and using pipe instead

gabe16:10:17

but now i don’t get the threads i need

mccraigmccraig16:10:07

@gabe: maybe a slight aside, but my new favourite way of doing multiple async calls and gathering results is applicative-do syntax (alet) - http://funcool.github.io/cats/latest/#channel - if you use an async network client you don't need to create any threads at all (beyond the client's service pool)

mccraigmccraig16:10:35

not much use if you can't use an async client tho

gabe16:10:04

that’s where i’m stuck right now (unfortunately)

gabe16:10:46

and he isn’t wrapping the async API

mccraigmccraig16:10:55

@gabe: another option would be to use https://github.com/ztellman/manifold which will manage threads for you and plays nicely with core.async

gabe16:10:20

That IS interesting. Thanks!

gabe16:10:44

that’s kinda of how i would do it in the JS world

gabe16:10:04

but less verbose

mccraigmccraig16:10:39

@gabe: i'm using it in a system with mixed core.async and manifold async components and it's been working great

gabe16:10:07

i might have to consider it. it could really cut down on the explicit use of channels

gabe16:10:55

core.async IS a “low-level” abstraction after all

mccraigmccraig16:10:23

if you do use it, you can go all high-level with applicative-do syntax too - http://funcool.github.io/cats/latest/#manifold-deferred

gabe16:10:31

thanks for the knowledge man

mccraigmccraig16:10:59

np... i've been learning myself in the last few months simple_smile

gabe16:10:24

@mccraigmccraig: you’ve given me some good light reading to do 😉

ajmagnifico18:10:49

@erik_price or anybody: Yesterday I asked about waiting for go processes to finish. Here's something I came up with.

ajmagnifico18:10:54

Would appreciate comments from anybody

ajmagnifico19:10:31

I think a simple (merge) would probably do the same thing for me

ajmagnifico20:10:35

OK, here's a more straightforward question: is there API for determining if a channel is closed? If it still has values available for taking?

ajmagnifico20:10:30

Or, given a channel that resulted from a (go) call, is there a way to distinguish between nil (channel closed, no values to take) and nil (`(go (println "hey"))` returned nil) ?

ghadi20:10:32

generally inspecting a channel is race-condition prone. If you ask it if it's closed, and it says no, it may be closed by the time you act upon that information

ghadi20:10:25

There is an api for it. I think it's not generally useful.

ghadi20:10:02

but to answer your question, re nil, you can't tell the difference.

ghadi20:10:20

More generalizations/pontification:

ghadi20:10:28

you shouldn't care what's on the other side of a channel.

mccraigmccraig20:10:50

there isn't any difference is there ? the result of (go (println "hey")) is a closed channel with no values to take

ghadi20:10:08

That's right, no difference.

ajmagnifico20:10:42

let me get at the real issue that I'm struggling with:

ajmagnifico20:10:00

How can I ensure that all go processes have completed before exiting a program?

ajmagnifico20:10:08

In golang, you've got WaitGroup, etc.

ajmagnifico20:10:22

Is there something equivalent in core.async?

ajmagnifico20:10:53

And actually, it's not even just exiting a program.

ajmagnifico20:10:18

there might just be times when I want all go processes to complete before proceeding with the remainder of the program.

ghadi20:10:04

channels of channels 😃

ajmagnifico20:10:23

that brings us to (merge), right?

ajmagnifico20:10:13

I wrote this.

ajmagnifico20:10:45

It turns out it's mostly equivalent to (but probably less robust than) (merge [channels ...])

ghadi20:10:51

merge can get you there.

ghadi20:10:51

I take it that what you are waiting on is heterogenous tasks?

ghadi20:10:07

not just a pool of identical things

ghadi20:10:26

if you have identical things, pipeline is what you want

ajmagnifico20:10:16

I noticed that and thought maybe it would be helpful to me.

ajmagnifico20:10:24

But what if it were heterogeneous tasks?

ghadi20:10:43

otherwise, at a high level, you just gotta remember what you launched off and wait for the channels that represent those tasks

ajmagnifico20:10:48

So if you have a channel, from merge or from pipeline or whatever,

ajmagnifico20:10:00

nevermind, I think I'm asking the wrong question

ajmagnifico20:10:07

("How do you know when there are no more values.")

ghadi20:10:10

Golang's waitgroup stuff warns: Typically this means the calls to Add should execute before the statement creating the goroutine or other event to be waited for.

ajmagnifico20:10:14

I think I need a paradigm shift! simple_smile

ghadi20:10:11

this stuff is tough.

ajmagnifico20:10:41

part of my problem is that when I'm incrementally writing experimental code in emacs/cider and trying to test this stuff out, the full expression typically ends before the go processes have completed.

ajmagnifico20:10:49

And I get bogus results.

ajmagnifico20:10:37

The await-go-processes function that I wrote keeps track of all the channels returned from (go) calls, and alts!! them until all have returned.

ajmagnifico20:10:58

This way I know that all the processes have completed

ajmagnifico20:10:20

K, I'll keep thinking about it

ghadi20:10:51

no prob. Wish I had a silver bullet

ajmagnifico20:10:15

I think I just need to change the way I think these things