I need to use duckdb (a C FFI call provided by tmlducken https://github.com/techascent/tmducken?tab=readme-ov-file) inside a m/sp. The API requires a connection that is unique per thread. So I guess I would need to provide a custom executer for this because for m/blk I doubt that I can create connections for say a thread pool of 8 threads. Is there an example somewhere for that? Thanks!
From my beginner understanding you do not need any missionary api to create a task as long as it conforms to the task spec. While you can prevent blocking with your own thread pool, the difficulty in your case will be cancelation as I doubt you can interrupt a thread blocked on a FFI call. So it all depends on the use of cancelation by the missionary api the task will be subjected to. Leo will probably have more precise hints to give you.
otherwise just pass your custom executor to m/via like in this example:
https://gist.github.com/leonoel/d13dd1b8045b3167dbf688f57244d221
And use ThreadLocal: https://stackoverflow.com/a/21608858
While pondering on the use of SingleThreadExecutorI realized it can move the control of the connection lifecycle to the main logic thread. The https://duckdb.org/docs/stable/clients/c/connect says:
> While individual connections are thread-safe, they will be locked during querying. It is therefore recommended that each thread uses its own connection to allow for the best parallel performance.
So it's the responsibility of the code creating a connection to assign it to a thread, the API does not enforce single thread usage. With a connection handle in the main thread the latter can cancel a running query using the https://duckdb.org/docs/stable/clients/c/connect#duckdb_interrupt call, which I assume is the best way to handle a cancellation.
Is that why you suggested the use of SingleThreadExecutor?
Although we probably don't need to use SingleThreadExecutor for that, the task calling m/via could return a cancel function closing over the interrupt call.
@fmjrey you're right, m/via propagates cancellation with thread interruption, but if the thread is blocked in a native call then it would not be able to react to it.
if I'm not mistaken tmducken doesn't expose the connection interruption method ?
No it doesn't by the look of it https://github.com/search?q=repo%3Atechascent%2Ftmducken+interrupt&type=code
yes, use m/via with a custom executor
Call https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/Executors.html#newSingleThreadExecutor() for each separate connection that you want to open, so that you give each connection its own single-threaded executor.