Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2024-04-07

Channels

#aws
<
2024-04-07
>
adham07:04:06

An insight from a bug I worked on today Using a lot of pmap can lead to {:cognitect.anomalies/category :cognitect.anomalies/busy, :cognitect.anomalies/message &quot;Ops limit reached: 64} errors! Should I look into core/async to replace what I was using the pmap for?

adham12:04:19

The "easiest" way ended up being something like this 

(let [concurrent 5
        output-chan (a/chan)
        input-coll digicomp/areas-kw]
    (a/pipeline-blocking concurrent
                         output-chan
                         (map (fn))
                         (a/to-chan! input-coll))
    (<!! (a/into [] output-chan)))

jjttjj16:04:51

Probably a good idea to control the parallelism of your requests. These might be useful though for more context/alternative solutions Have you seen https://github.com/cognitect-labs/aws-api?tab=readme-ov-file#ops-limit-reached ? https://github.com/cognitect-labs/aws-api/issues/98#issuecomment-584381822

adham13:04:29

My problem is that I don't know where the client is since I am using Datomic and the errors are coming from functions that are invoking queries. I assume the client is where I run datomic.client.api/client but I am not sure.