aws

adham 2024-04-07T07:04:06.212579Z

An insight from a bug I worked on today Using a lot of pmap can lead to {:cognitect.anomalies/category :cognitect.anomalies/busy, :cognitect.anomalies/message "Ops limit reached: 64} errors! Should I look into core/async to replace what I was using the pmap for?

adham 2024-04-07T12:33:19.732789Z

The "easiest" way ended up being something like this

(let [concurrent 5
        output-chan (a/chan)
        input-coll digicomp/areas-kw]
    (a/pipeline-blocking concurrent
                         output-chan
                         (map (fn))
                         (a/to-chan! input-coll))
    (<!! (a/into [] output-chan)))

2024-04-07T16:43:51.748549Z

Probably a good idea to control the parallelism of your requests. These might be useful though for more context/alternative solutions Have you seen https://github.com/cognitect-labs/aws-api?tab=readme-ov-file#ops-limit-reached ? https://github.com/cognitect-labs/aws-api/issues/98#issuecomment-584381822

adham 2024-04-08T13:23:29.353189Z

My problem is that I don't know where the client is since I am using Datomic and the errors are coming from functions that are invoking queries. I assume the client is where I run datomic.client.api/client but I am not sure.