Fork me on GitHub
Daniel Craig19:06:06

If I'm using pipeline-blocking and I don't care about the result of my transformation function, what should I put on the 'to' channel?

Alex Miller (Clojure team)19:06:30

why are you using pipeline-blocking?

Alex Miller (Clojure team)19:06:45

your pipe is leaking on the floor... :)

Daniel Craig19:06:57

Maybe I shouldn't be... I have a feeling I'm in over my head

Alex Miller (Clojure team)19:06:06

I assume your transformation function is doing some potentially blocking i/o?

Alex Miller (Clojure team)19:06:59

do you want it to be parallelized? do you want bounded parallelism? do you want back pressure?

Daniel Craig19:06:15

I want it to be parallelized, it doesn't have to be bounded although I assume that's the practical thing to do, and the back pressure question is one that I'm still thinking about

Alex Miller (Clojure team)19:06:02

I mean, generally you want bounded (unbounded things only stop working at 2 am when you're on call)

😆 3
Daniel Craig19:06:58

I wanted to write a writer library for an existing java service that reads json and then in a very blocking way writes into a Tinkerpop graph database.

Alex Miller (Clojure team)19:06:41

pipeline-blocking will always write a result into an output channel (that will accumulate)

Daniel Craig19:06:13

The service currently takes ~4 hours to run, so I thought that by parallelizing and batching the writes we could achieve a speed up. But I'm not sure how backpressure works in a scenario like this

Alex Miller (Clojure team)19:06:43

if you want parallel, bounded, with backpressure but don't care about the results you could give it channel with a dropping buffer that would just immediately drop the "results" on the floor

Daniel Craig19:06:59

that's a solution

Daniel Craig19:06:55

What's a typical n value to use with pipeline-blocking?

Alex Miller (Clojure team)19:06:30

you could also just have a go-loop feeding a queue/channel and N threads reading from the queue and pushing stuff into the db

Alex Miller (Clojure team)19:06:52

pipeline-blocking is doing a bunch of work to order the results that you presumably don't care about

Daniel Craig19:06:26

Yeah I don't really care about the ordering of the results

Daniel Craig19:06:30

I should just write it in java anyways

Daniel Craig19:06:38

My team doesn't know clojure

Daniel Craig19:06:33

Thanks for your help


how would you do this in java?

Daniel Craig19:06:27

I'd have to do a little investigation on the best way to do it. Streams are new to me but I could probably make a streams-oriented solution