This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
What archiving formats are supported by Clojure/Java without external libraries? I've found java.util.zip, but are there any others? Compression is not important, but I need to pack multiple files into one, in a format that I can later unpack on a mobile device (not necessarily Java)
I'm consuming a REST API using iteration (https://gist.github.com/leifericf/242fb222966c8cfc8beb04539aaf8fab#file-pulumi-clj-L17-L24). This works great, and making HTTP requests with pmap is super fast. However, it does not allow me fine-grained control to deal with rate limiting. What are some common approaches to gaining more fine-grained control of how many requests are made in parallel, how many requests are made per minute, etc.? I suspect I'll need to use something in core.async and/or maybe Agents?
For parallelism, a semaphore might be enough. For rate limiting, something like https://github.com/sunng87/diehard perhaps
core.async combined with synchronous HTTP, leveraging backpressure to throttle a big ETL job: https://tilton.medium.com/backpressure-99501f23881f
I stopped shy of dynamically tuning the system, by adding/removing workers on the fly, because it was quick work to look the metrics and manually tune for the invariant workload at hand, but self-throttling was a logical next step.
hth!
https://github.com/clj-commons/claypoole I've used fixed-size threadpools with Claypoole for this exact type of problem, and it typically works very well, but it won't let you control the rate of requests per minute directly.