yeah, I've got my eye on that. Is the goal that I'd just need the files and the jar or would I need other external things (kafka other) as well?

at the moment it's coupled in the artifact but we could likely release the Arrow query engine separately if there was demand for it

I think I'd be looking for something that I could use in process from a single jar. I've not properly looked at things since the very early days of needing to set up things like kafka (and postgres? been so long I can't even remember)

ah, sorry, my misunderstanding - it'll still be possible to create in-memory/local-disk nodes straight from the core JAR, Kafka is an optional extra in that regard

I thought you'd just be interested in the standalone query engine running queries over existing, provided Arrow files, at which point things like transaction processing etc also aren't necessary

but yes, in-process nodes are certainly still a core feature

the arrow stuff would be interesting, tho the competition there is DuckDB in terms of performance and I'm more performance than expressively constrained in my analysis

https://github.com/techascent/tmducken and general tmd ways of using larger than memory arrow files

sounds like my understanding of what is needed for xtdb was a bit out of date. I had it in the "very capable, but high cost (of effort and admin)" category of tools and it sounds like it might work for some embedded single node purposes

getting used to using it in small personal things and then being able to scale up if needed but being familiar with data modelling and the rest using to tool is a real sweet spot for me