This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-06-16
Channels
- # admin-announcements (1)
- # announcements (1)
- # babashka (130)
- # beginners (120)
- # calva (11)
- # cider (5)
- # clj-kondo (9)
- # cljsrn (17)
- # clojure (63)
- # clojure-australia (1)
- # clojure-canada (21)
- # clojure-europe (37)
- # clojure-israel (4)
- # clojure-uk (6)
- # clojurescript (170)
- # conjure (5)
- # core-async (23)
- # cursive (16)
- # datomic (4)
- # defnpodcast (1)
- # emacs (5)
- # fulcro (1)
- # gis (2)
- # graalvm (31)
- # graphql (4)
- # helix (6)
- # honeysql (16)
- # jobs-discuss (3)
- # juxt (1)
- # lsp (7)
- # malli (20)
- # meander (12)
- # missionary (6)
- # off-topic (50)
- # pathom (4)
- # re-frame (4)
- # react (1)
- # ring (2)
- # shadow-cljs (63)
- # spacemacs (2)
- # sql (15)
- # testing (6)
- # vim (8)
- # xtdb (7)
If I'm using pipeline-blocking and I don't care about the result of my transformation function, what should I put on the 'to' channel?
why are you using pipeline-blocking?
your pipe is leaking on the floor... :)
Maybe I shouldn't be... I have a feeling I'm in over my head
I assume your transformation function is doing some potentially blocking i/o?
Yes it is
do you want it to be parallelized? do you want bounded parallelism? do you want back pressure?
I want it to be parallelized, it doesn't have to be bounded although I assume that's the practical thing to do, and the back pressure question is one that I'm still thinking about
I mean, generally you want bounded (unbounded things only stop working at 2 am when you're on call)
I wanted to write a writer library for an existing java service that reads json and then in a very blocking way writes into a Tinkerpop graph database.
pipeline-blocking will always write a result into an output channel (that will accumulate)
The service currently takes ~4 hours to run, so I thought that by parallelizing and batching the writes we could achieve a speed up. But I'm not sure how backpressure works in a scenario like this
if you want parallel, bounded, with backpressure but don't care about the results you could give it channel with a dropping buffer that would just immediately drop the "results" on the floor
that's a solution
What's a typical n value to use with pipeline-blocking?
you could also just have a go-loop feeding a queue/channel and N threads reading from the queue and pushing stuff into the db
pipeline-blocking is doing a bunch of work to order the results that you presumably don't care about
Yeah I don't really care about the ordering of the results
I should just write it in java anyways
My team doesn't know clojure
Thanks for your help
I'd have to do a little investigation on the best way to do it. Streams are new to me but I could probably make a streams-oriented solution