Fork me on GitHub
#core-async
<
2018-04-12
>
ajs17:04:27

@jgdavey don't know your use case for parallelizing work but I can highly recommend the Claypool library for maximum multithreaded flexibility and speed

dimovich20:04:23

I have this code

dimovich20:04:24

even though I have 1000 pages, I'm getting only 8 results. What am I doing wrong?

noisesmith20:04:06

@dimovich you only read from the channel once, shouldn't you be reading it repeatedly until it gets closed by the pipeline?

noisesmith20:04:53

that is, s/reading/taking/g

dimovich20:04:23

@noisesmith I thought <!! will close when there is no more data on the pipeline

noisesmith20:04:57

it reads a single result

dimovich20:04:10

how can I read all of them?

dimovich20:04:22

I tried async/reduce, but same thing

noisesmith20:04:44

by repeatedly consuming in a loop or go-loop, or using a construct that fills a collection from a channel

noisesmith20:04:24

probably the simplest thing is (async/into [] c)

dimovich20:04:01

trying them now, thanks

hiredman20:04:48

the names of the functions sort of lead me to believe they are doing (likely blocking) io, which means you should be using pipeline-blocking or pipeline-async

dimovich20:04:03

yeah, http requests

hiredman20:04:25

it's all there in the docstring for pipeline '''

hiredman20:04:30

If you have multiple
  blocking operations to put in flight, use pipeline-blocking instead,
  If you have multiple asynchronous operations to put in flight, use
  pipeline-async instead.

dimovich21:04:26

(async/<!! (async/into [] (async/pipeline-blocking 4 out-chan xform in-chan))) still gives me 8 results

hiredman21:04:28

what makes you think you should be getting more than 8?

dimovich21:04:13

I have (async/onto-chan in-chan pages). And there are 1000 items in pages.

hiredman21:04:27

(partial map) is just map

dimovich21:04:50

yeah, right 🙂

hiredman21:04:57

are you sure there are 1000 items in pages?

hiredman21:04:40

are your transducers throwing items away?

dimovich21:04:58

no, they either modify, or return original

hiredman21:04:03

what happens if you replace xform with identity?

hiredman21:04:42

you are taking from the channel return from the call to pipeline-blocking, not the output channel you give to pipeline blocking

hiredman21:04:08

the channel that pipeline-blocking returns is nothing

noisesmith21:04:17

@hiredman: I thought I'd checked in the past and it returned the output channel

hiredman21:04:07

I am pretty sure it doesn't

dimovich21:04:13

@hiredman yay, it works 🙂 thanks

hiredman21:04:31

the fact that you go any number of response from the channel returned from pipeline-blocking instead of just getting a closed suggests some kind of bug in your code somewhere

dimovich21:04:55

maybe it didn't return anything. I was getting log entries from the update-stat fns

dimovich21:04:12

@noisesmith @hiredman thank you both for helping!

noisesmith21:04:09

I'm wondering which core.async function I was thinking of, that allowed forwarding data from one chan to another with a transducer, which conveniently returns the destination channel

noisesmith21:04:25

not pipeline* - none of them actually work that way

dimovich21:04:51

you can do (async/chan 1 xform)

noisesmith21:04:23

right, but sometimes you have a channel that comes from someone else, and you want an xform on it