clojure 2024-07-20 | Slack Archive

Has anyone worked with using libgdx with clojure recently? Does it feel good?

You might get more answers in #C066UV2MV

Given a function pipeline involving for each “item” • request against “quick” API (<1s) • request against “slow” API (~10s) • fairly quick but cpu intensive thing (<0.5s) • async write to db (i.e. a quick API request) • poll API until prior db write is finished (expected to be ~2 - 5s until async job is complete) This thing runs in a lambda. Sometimes just for a single item but sometimes we want to run it with very high concurrency to churn through backfills. What could I do to minimize time spent by a lambda waiting for http responses in this latter scenario?

phronmophobic16:07:20

Is the backfilling run in a lambda? Does it have to be?

cddr16:07:23

probably but still interested in ideas where it’s not if you have ’em 🙂

phronmophobic16:07:18

Depending on where the batch data is coming from and the scale, you could just run it outside of a lambda.

phronmophobic16:07:20

It's kind of hard to say without knowing more specifics: • scale • how often are you backfilling? • Where is the data/instructions for the backfill coming from?

cddr16:07:34

maybe not very high concurrency actually, but >1

phronmophobic16:07:23

Yea, adding extra ops work doesn't seem useful for that case.

phronmophobic16:07:47

I'm assuming each step in the pipeline depends on the previous step?

cddr16:07:56

yep

phronmophobic16:07:35

Most frameworks already handle separate requests concurrently. Is the idea that a request gets a handful of items and you want to parallelize processing the list of items?

cddr16:07:39

yes exactly

phronmophobic16:07:33

There are lots of options. You can go crazy with core.async, you can just use futures, but I think a nice place to start is Executors with a fixed thread pool, https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/Executors.html.

phronmophobic16:07:12

The idea is that you create a global executor service with a fixed threadpool, https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/Executors.html#newFixedThreadPool(int) For each request, you submit each item to the thread pool which gives you a list of futures. Then you deref all of the futures which will wait cause the request thread to wait for all the items to get processed and return their results.

👍 1

cddr16:07:48

What if I want to make sure the write step happens on a single thread? Can I still somehow leverage concurrency in the other steps at the expense of RAM?

phronmophobic16:07:05

You can create a shared singleThreadExecutor for the write operations.

cddr16:07:50

Cool. I think that’s a bit of confirmation for what I was hoping.

cddr16:07:14

(->> (cp/pmap 20 api-quick)
     (cp/pmap 10 api-slow)
     (cp/pmap 10 transform)
     (cp/pmap 1 api-put)
     (cp/pmap 20 wait-until-finished))

👍 1

didibus16:07:20

You can setup lambda to batch process. Your lambda will be given a batch of request, then you can process them all concurrently within the lambda.

👍 1

didibus16:07:31

Feel free to use this little util as well if you want:

(defn map-io
   [n io-fn input-coll]
   (let [sem (Semaphore. n false)
         res (for [e input-coll]
               (do (.acquire sem)
                   (future (try (io-fn e) (finally (.release sem))))))]
     res))

It returns a list of future you can deref whenever you want to await for results.

👀 1

2024-07-20

Channels