This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
thanks, so it's just the first
0 that causes the problem! So, is there a standard way to read a number as decimal, or should I manually drop the leading zeros?
is there interesting prior art for self-adjusting thread pools? e.g. I have an IO-bound workload, can't know in advance if the ideal amount of threads will be 50 or 200 or ...
it might even depend on day I'm running the task at hand, since 3rd party APIs can be less or more available
so ideal usage is specifying "I want to optimize for throughput", having such a measurer, and a thread pool that adds/decreases threads depending on what it observes. Would be sort of a
Perhaps don't limit the threadpool but do it higher up the stack? See https://gist.github.com/djspiewak/46b543800958cf61af6efa8e072bfd5c > you should always push your blocking operations (IO or otherwise) over to a separate thread pool. This separate thread pool should be caching and unbounded with no pre-allocated size. (emphasis mine) and later in a comment: https://gist.github.com/djspiewak/46b543800958cf61af6efa8e072bfd5c#gistcomment-3158950 > Bounded thread pools contain unbounded task queues that are entirely outside your control. You can't see how many outstanding tasks there are, reschedule them, cancel them, change your semantics, etc. When you start running out of scarce resources, you need to be able to propagate that information back upstream in the form of temporary connection drops, or even better, trigger autoscaling to create more resources. You want to do this at the highest possible level in your stack, because that gives you the greatest semantic control. A thread pool hitting its thread count limit cannot interact with kubernetes to allocate a new pod.
the "easy" answer is to have a theoretically unbounded number of threads, where back pressure is reflected by the external resources instead of the thread pool size
@U06BE1L6T the problem with unbounded is how to use an ideal amount of threads? If I use too few I might be underutilizing resources, if I use too many I might be decreasing throughput, hitting rate limits, etc
@UK0810AQ2 I guess that would work but it merely shifts the problem from one place to the other? 😁
And I don't think that the app itself, without considering the context / world around it, can make reasonable decisions about the optimal number of threads / concurrent operations.
I see, the Gist writing is sufficiently ambiguous to not be particularly useful tbh "saturate your pool until you have to have to scale out" is not universal advice, maybe I just want to run a task from a single node. And again saturating a pool can decrease throughput, hit rate limits, etc.
It's a long shot but perhaps there's something in https://github.com/resilience4j/resilience4j or a similar library? It's a successor of Hystrix which is meant to be more adaptable - but I haven't used it
@U45T93RA6 yes and no, your IO boundedness is the other peer's CPU bound, at some spot. Let them throttle you, and apply backpressure based on failed response codes or long response times
I think a DB pool is a good example of that: https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing
Using a specific (but fictional) example, let's say I'm downloading videos from a provider. Maybe at thread count = 1000, I'm getting 2kbps throughput for each thread because the 3rd-party API splits throughput evenly. I'm still not getting a long response time or a rate limit, but the throughput will be effectively unbearable and equivalent to not getting done with the tasks. Maybe the "curve" is like this: Thread Count of 1 = 1000kbps/thread Thread Count of 5 = 1000kbps/thread (doesn't decrease) Thread Count of 100 = 700kbps/thread (decreases a little) Thread Count of 1000 = 2kbps/thread (decreases a lot) Thread Count of 2000 = 1kbps/thread (further decreases) Thread Count of 2222 = I start getting rate-limit errors So the ideal thread count would could be around 100, give or take, if optimizing for global throughput.
Hope it's not an overly contrived example - personally I've sensed for a long time that this sort of dynamic allocation makes sense for certain use cases, even if not for every. Maybe I should just go and code it next time I encounter this!
Sounds like you need a rate-limiter, but not for incoming requests but outgoing - we've built something like that based on redis, so we can have a pool for workers spread across multiple nodes and we maximize throughput per-tenant, while also staying 50% under API request limits (so that we don't disrupt our customer's API usage - in the early days we happened to screw up one of our customer's custom Zendesk extension, because our sync process would just go all-in and maximize how much data we can fetch). It's tricky to get this right, as you've pointed out: you run into a state where more threads means that everything will slow down. My advice is to figure out max req/s (or whatever rate you have to optimize for) and walk backwards from there
What you're describing falls under the theory of systems control using negative feedback. In your example, you want to maximize throughput, your control signal is the number of threads. Easy to implement using a PID controller https://en.wikipedia.org/wiki/PID_controller Or using some adaptive control, which is more complicated. But it's feasible to reach a "hands off" solution
A simple way you could implement a negative feedback system is by controlling the number of threads and measuring the rate. As long as increasing the number of threads increases the rate, increase the number of threads. When you add a thread and the overall rate decreases you flip a "switch" and stop increasing the number. Maybe even go one back. Since this system is inherently noisy, it's usually beneficial to pass the signal through some hysteresis filter. You can easily implement this system with core.async. Keep in mind that discrete time control is slightly different than continuous time control, too. I'd probably implement something like emitting a constant +1 signal for the number of threads as long as the derivative of the overall rate is positive (i.e. rate[t+1] - rate[t] > 0)
I rarely get opportunities to apply this, but I really enjoyed systems control in school
So to start with, a "bad" implementation will be:
D threads[t] = sign(D rate[t]) with
D being the discrete time derivative operator
That's what a cached thread-pool is for. You do a load-test to see when your host tips over, that tells you what is the MAX size of your pool for your given resource. You set that as the MAX size, but otherwise you make it a cached thread pool. It'll grow and shrink as usage goes up/down.
But you need to use the standard constructor to put an upper-bound:
corePoolSize is like the minimum number of thread, you probably can set it to 0. maximumPoolSize is the max you've identified before you have too many threads and you application starts to slow down. keepAliveTime is how long you want to cache threads, 60 second is a good default. unit is the unit of time for the number you set too keepAliveTime. workQueue is the queue to use for pending tasks. This is where you could handle backpressure if you need too
ThreadPoolExecutor(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit, BlockingQueue<Runnable> workQueue)
(ThreadPoolExecutor. 0 500 ;; Your max threads 60 TimeUnit/SECONDS (SynchronousQueue.))
A good default choice for a work queue is a SynchronousQueue that hands off tasks to threads without otherwise holding them. Here, an attempt to queue a task will fail if no threads are immediately available to run it, so a new thread will be constructed. This policy avoids lockups when handling sets of requests that might have internal dependencies. If you use SynchronousQueue, basically its like not having a queue at all. So each time you submit it will try to reuse one of the cached threads if available, if there isn't any it will create a new thread and run the task immediately in it. But if your max threads is reached, it will reject the task. You can define the strategy for rejection by passing an additional parameter to ThreadPoolExecutor, which is a handler that chooses how to handle the rejection. By default it will just throw an exception when rejected. There's a few availabile choices too if you see here: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ThreadPoolExecutor.html under Rejected tasks but you could also implement your own if you want.
Perhaps only tangentially related, but ztellman has a library somewhat related to this https://github.com/ztellman/dirigiste
If I'm not mistaken, it sounds exactly like the Java's
ThreadPoolExecutor, given its
It doesn't to me, generally java's stock io-oriented executor merely grows/shrinks but it doesn't have a notion of "optimize for x"
Ah, so you want not just automatically increasing/decreasing pool size according to the current load and some delay, but a sort of learning thread pool where it adapts its size while expecting higher load during some specific times of the day/days of the week/etc?
Yes, even when not considering the day of the week I cannot know in advance if the ideal thread count will be 50 or 200 (that is, without some trial/error. Which I'd rather automate)
that happens to be a transitive dep in my prod app however it left me a mixed impression (https://github.com/ztellman/dirigiste/pull/30 plus some other thing that I also had to fix for it to play fine w/ Component) it might have something worth salvaging though!
I have a project, where I want to add a middleware to a reitit router only in the dev profile. Currently I use Integrant to setup my system. Is it possible to define a middleware in the dev folder and pass it to the handler function via the integrant config? Or do you suggest a different solution?