missionary

awb99 2026-02-15T01:15:08.946259Z

I need to use duckdb (a C FFI call provided by tmlducken https://github.com/techascent/tmducken?tab=readme-ov-file) inside a m/sp. The API requires a connection that is unique per thread. So I guess I would need to provide a custom executer for this because for m/blk I doubt that I can create connections for say a thread pool of 8 threads. Is there an example somewhere for that? Thanks!

fmjrey 2026-02-15T06:50:15.408129Z

From my beginner understanding you do not need any missionary api to create a task as long as it conforms to the task spec. While you can prevent blocking with your own thread pool, the difficulty in your case will be cancelation as I doubt you can interrupt a thread blocked on a FFI call. So it all depends on the use of cancelation by the missionary api the task will be subjected to. Leo will probably have more precise hints to give you.

fmjrey 2026-02-15T07:05:00.064889Z

otherwise just pass your custom executor to m/via like in this example: https://gist.github.com/leonoel/d13dd1b8045b3167dbf688f57244d221

👍 1
fmjrey 2026-02-15T07:21:19.139849Z

And use ThreadLocal: https://stackoverflow.com/a/21608858

fmjrey 2026-02-16T09:19:33.265459Z

While pondering on the use of SingleThreadExecutorI realized it can move the control of the connection lifecycle to the main logic thread. The https://duckdb.org/docs/stable/clients/c/connect says: > While individual connections are thread-safe, they will be locked during querying. It is therefore recommended that each thread uses its own connection to allow for the best parallel performance. So it's the responsibility of the code creating a connection to assign it to a thread, the API does not enforce single thread usage. With a connection handle in the main thread the latter can cancel a running query using the https://duckdb.org/docs/stable/clients/c/connect#duckdb_interrupt call, which I assume is the best way to handle a cancellation. Is that why you suggested the use of SingleThreadExecutor?

fmjrey 2026-02-16T09:35:01.688109Z

Although we probably don't need to use SingleThreadExecutor for that, the task calling m/via could return a cancel function closing over the interrupt call.

leonoel 2026-02-16T12:20:19.844959Z

@fmjrey you're right, m/via propagates cancellation with thread interruption, but if the thread is blocked in a native call then it would not be able to react to it.

leonoel 2026-02-16T12:22:12.621859Z

if I'm not mistaken tmducken doesn't expose the connection interruption method ?

fmjrey 2026-02-16T12:22:48.337859Z

No it doesn't by the look of it https://github.com/search?q=repo%3Atechascent%2Ftmducken+interrupt&type=code

leonoel 2026-02-15T09:03:15.700139Z

yes, use m/via with a custom executor

Andrew Wilcox 2026-02-16T04:12:22.231979Z

Call https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/Executors.html#newSingleThreadExecutor() for each separate connection that you want to open, so that you give each connection its own single-threaded executor.