Hi, I have simple question. I’m loading some stuff from a file via a rest api. It’s all working and now I want to parallelize the process. I don’t really care about return values, etc in the program as errors are being logged to a file so i’ve gone from something like (doseq [foo bar] (call-api foo)) to (doseq [foo bar] (go (call-api foo))) The concern i have is that I’m doing this over multiple collections, and while it works fine for a given doseq/collection, the next doseq shouldn’t start until the last one has completed. Not sure of the best way to make this happen

that's not what go is for

I am not sure that is a good fit for core.async

putting IO in go blocks is a good way to grind everything to a halt

(in the core.async world at least)

well, the io is problematic, but core.async is largely about communication, passing values around

yes

if you don't have any return values or whatever, it doesn't make sense

ah well lol, glad i asked

any suggestions for this use case

?

https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Executors.html#newFixedThreadPool(int)

there are good abstractions for managing parallel execution in the claypoole library

which uses FixedThreadPool as hiredman links iirc

if you find you do want to use values also there's some stuff in the core.reducers library I've inteded to try but I can't vouch for any of it

ah ok yeah Ive used that stuff before, was wondering if there was anything ‘clojury’ lol

claypoole is more clojury

ok will chekc it out

ah looks pretty straightforward

but the nice thing with using a thread pool from clj is clojure functions are callable / runnable

so you don't need to use interop for that part at least

right

I would have expected this snippet to produce "true\nclosed\nfalse", but actually it remains true for the last line.

(let [continue? (async/to-chan (repeat true))]
  (async/go
    (println (async/<! continue?))
    (async/close! continue?)
    (println "closed")
    (println (async/<! continue?))))

a closed channel doesn't return nil until all data buffered is consumed

I don't know how to-chan decides how much it buffers, clearly it's not unbuffered

Yeah I was wondering whether that might be the case

with some experimentation you might figure out the buffer size (32, a common chunk size, would be a decent first guess)

Looks like it's 100: https://github.com/clojure/core.async/blob/2afc2dc5102f60713135ffca6fab993fb35809f0/src/main/clojure/clojure/core/async.clj#L646

cool, there we go