Fork me on GitHub
#core-async
<
2018-01-04
>
eoliphant22:01:38

Hi, I have simple question. I’m loading some stuff from a file via a rest api. It’s all working and now I want to parallelize the process. I don’t really care about return values, etc in the program as errors are being logged to a file so i’ve gone from something like (doseq [foo bar] (call-api foo)) to (doseq [foo bar] (go (call-api foo))) The concern i have is that I’m doing this over multiple collections, and while it works fine for a given doseq/collection, the next doseq shouldn’t start until the last one has completed. Not sure of the best way to make this happen

noisesmith22:01:36

that's not what go is for

hiredman22:01:01

I am not sure that is a good fit for core.async

noisesmith22:01:06

putting IO in go blocks is a good way to grind everything to a halt

noisesmith22:01:13

(in the core.async world at least)

hiredman22:01:37

well, the io is problematic, but core.async is largely about communication, passing values around

hiredman22:01:54

if you don't have any return values or whatever, it doesn't make sense

eoliphant22:01:07

ah well lol, glad i asked

eoliphant22:01:13

any suggestions for this use case

noisesmith22:01:44

there are good abstractions for managing parallel execution in the claypoole library

noisesmith22:01:56

which uses FixedThreadPool as hiredman links iirc

noisesmith22:01:29

if you find you do want to use values also there's some stuff in the core.reducers library I've inteded to try but I can't vouch for any of it

eoliphant22:01:00

ah ok yeah Ive used that stuff before, was wondering if there was anything ‘clojury’ lol

noisesmith22:01:21

claypoole is more clojury

eoliphant22:01:32

ok will chekc it out

eoliphant22:01:05

ah looks pretty straightforward

noisesmith22:01:14

but the nice thing with using a thread pool from clj is clojure functions are callable / runnable

noisesmith22:01:23

so you don't need to use interop for that part at least

cddr22:01:48

I would have expected this snippet to produce "true\nclosed\nfalse", but actually it remains true for the last line.

(let [continue? (async/to-chan (repeat true))]
  (async/go
    (println (async/<! continue?))
    (async/close! continue?)
    (println "closed")
    (println (async/<! continue?))))

noisesmith22:01:46

a closed channel doesn't return nil until all data buffered is consumed

noisesmith22:01:06

I don't know how to-chan decides how much it buffers, clearly it's not unbuffered

cddr22:01:02

Yeah I was wondering whether that might be the case

noisesmith22:01:32

with some experimentation you might figure out the buffer size (32, a common chunk size, would be a decent first guess)

noisesmith22:01:04

cool, there we go