Fork me on GitHub
#core-async
<
2017-05-22
>
fabrao10:05:09

Hello all, is that possible to use core.async as a queue? I´d like to start 10 threads and do some action after the 10th thread finishing.

jimmy11:05:44

@fabrao yes it's possible.

fabrao11:05:54

how to do that? I found this in clojure docs

(doseq [n (range 10)
               :let [i (-> n
                           inc
                           range
                           rand-nth)]]                    
         (go
           (<! (timeout (* i 1000)))
           (println n)))
Is that something like this? replace i to thread result and <! is for waiting all threads terminate?

jimmy11:05:20

yes, it looks correct to me

joshjones17:05:27

@fabrao IMO using core.async simply for creating/joining threads is overkill. The following code starts 10 threads, joins them (“join” means “wait for”), and then moves on — there are many ways to do this, but this is probably the easiest:

(let [tp (java.util.concurrent.Executors/newCachedThreadPool)
      your-fn #(println "Hooray!")]
  
  (.invokeAll tp (repeat 10 your-fn))
  
  (println "Everyone's done!"))

joshjones17:05:11

If you’re already using core.async, then awesome — otherwise, you are going to be using it for something that’s already built into the core language, and can be done more concisely

joshjones17:05:26

in fact, I posted this a couple of months back and it may prove useful to see a few options for how to create and join threads in clojure:

rwilson17:05:43

a.k.a Java Concurrency in Practice (in Clojure)

joshjones18:05:03

are those examples also in JCIP? i have never seen it, but intended to do so (i would assume some of them would be, since that’s mostly clojurized java) @U5B9SLY6T

rwilson18:05:39

Not examples, exactly. Reading the gist just reminded me of JCIP because it's basically a guide to Java concurrency utils

noisesmith17:05:36

not only that, but actual work should never be done directly in go blocks, if you need to start a CPU or IO heavy task inside a go block you should use clojure.core.async/thread which returns a channel that you can park on

rwilson17:05:40

Did you mean, "that you can block on?"

noisesmith17:05:00

no, you can block on anything

noisesmith17:05:16

the point of core.async/thread is that you can park your thread, since it returns a channel

rwilson17:05:16

True, but the put/take ops in a thread block are blocking, not parking

rwilson17:05:28

In a go block they're parking

noisesmith17:05:02

the point of thread is that it returns a channel. you can park on that channel. you put CPU or io heavy ops inside thread, that’s all I’m saying

noisesmith17:05:23

if you do the cpu or io heavy jobs inside go itself, you can starve the go thread pool, which breaks everything

noisesmith17:05:56

in fact, I’d assume that code that uses channels inside core.async/thread is probably doing it wrong

rwilson17:05:37

Agreed on starving the go thread pool, I just thought referring to parking in a thread block was a little deceptive, because typically that's not what you'd be doing

noisesmith17:05:54

read it again, I never mentioned parking inside thread

rwilson17:05:34

Ah, parking on the return value, fair point I misread that

noisesmith17:05:58

and the whole reason thread exists is so that you can park on it

rwilson17:05:45

Yes, agreed on that. I just misread your comment as referring to parking in the thread block, not on the returned chan

rwilson17:05:52

Apologies for the confusion

fabrao18:05:02

@joshjones My needs is starting many threads and waiting all of them finishing and do something with results of each one. So, using the "1 - Clojure futures, and deref to join" have I use (doall (map deref threads)) instead of (run! deref threads)?

hlship18:05:53

@joshjones That sounds very much like a use for pmap (parallel map), which maps over items in a collection, but uses a pool of threads to do work in parallel.

hlship18:05:10

One of the factors here is the lifespan of your process; things like pmap and core.async work best for long lived servers, as there's some setup cost for them, and Hotspot needs to get in there and optimize a bit. For something like a startup-and-finish command line tool, manually starting a bunch of one-of threads will probably be better.

joshjones18:05:12

@hlship yes, if the work is in a seqable format already then pmap is quite easy

hlship18:05:58

I ref'ed the wrong person; @fabrao might want to look at pmap.

fabrao18:05:02

I did with pmap like this:

(cp/with-shutdown!
   [pool (cp/threadpool (count (util/localidades)))]
   (let [result (doall (cp/pmap pool #(validar-localidade %) (util/localidades)))]
     (into #{} result)))
with [com.climate.claypoole :as cp] thread pool and pmap from the same library because the regular pmap has limit from CPU/Core, but with long validar-localidade sometimes freeze. So, I´m trying to find alternative to something similar.

fabrao18:05:17

So, I thought using core.async will help me when last function evaluation will close channel to do after work

fabrao18:05:10

and now I don´t know how to solve this 😥

fabrao19:05:48

So do you think the best way in this case is using pmap?

raymcdermott22:05:55

I prefer to use core.async go-loop in some cases over a for-loop because it is simpler to send a signal to stop the loop. Is that heresy?

hiredman22:05:16

for is not a loop

hiredman22:05:55

if you want a loop, you will definitely have trouble using for, because for is not a loop

raymcdermott22:05:08

haha you know what I mean

raymcdermott22:05:55

(while test body) ; if I must

tanzoniteblack23:05:23

@raymcdermott I don't see anything heretical about the snippet you posted

tanzoniteblack23:05:48

well, depending on what reachable looks like, obviously. I would've presumed since you're doing network traffic, that your code to check if a server is up would return a channel, which your snippet doesn't really show anything about. Presumably you're eliding whatever that is for demonstration purposes here?

tanzoniteblack23:05:39

your only other real choice is to use loop/recur directly...but since you're working with channels here, I'm not sure why you'd want to do that? What were you thinking of instead when you said for-loop that makes you think this is heresy?