Hey! Context: A first user request an API to run expensive task, say the generation of 1000 images. The API fires a pipeline-blocking job that uses all the cores. Question: If a second user comes in and requests same job, can we split threads equally between to requests so that both users get their processes run at the same time?

setting aside fairness concerns if one user makes multiple requests concurrently, you can have a single pipeline-blocking that serves all incoming requests (plus some wrapping of the data to ensure the right results go to the right place of course)

@noisesmith Thanks for answering. Can we dynamically say, alright, instead of putting user 2 jobs at the end of the queue, let’s “interleave” incoming jobs with other users?

interesting - intuitively it seems like you could do that with a custom buffer implementation

maybe some variant of a prioqueue ?

so, if user 1 got 100/1000 images processed, then user 2 requests 1000 images to be generated, it goes like this user1:101th image, user2:1st image, user1:102th image, etc.

perhaps an atom holding a vector with one chan per user id, and a driver function that iterates over the vector in a loop, taking one item from each chan?

(with logic to skip to the next one if no job is immediately availabe, you could use poll! for that)

just a thought

interesting…

I wonder if there is out of the box solution for this.

not that I know of

I worked on a heavily data intensive application that relied on similar constraints

but our case was easier because each client had a auth token that was exercised by their work, and it had timeouts built in

so we were effectively forwarding upstream limits to have the same behavioral consequences

you can do things with pipeline-async, where the actual jobs are run on some shared limited executor, where the pipeline-async's limit the number of jobs running per kind of task, and the shared executor limits the number of jobs run for all tasks

you can also give jobs coming in on a certain channel higher priority by instead of directly putting jobs directly on the input channel for a pipeline, putting them on the input channel to some copying processing that copies from multiple channels (using alts with the priority argument) to the input of the pipeline