Clojurians Log v2

Found the answer I was looking for here: https://v1-docs.xtdb.com/resources/faq/#comparisons

Hey @taylor.jeremydavid, found this thread using search—am I reading this right that v2 will support in-process embedded in-memory deployment just as v1 did? (This has been such a godsend for testing and I’m loath to give it up 🙂) Relatedly, will v2 offer in-process node implementations appropriate for production, like the LMDB implementation for v1? (Looks like v2 is going long on cloud, but I’m in a domain where in-process is probably the way to go.)

Hey @lgessler in-process access to the query engine is still relevant/important but not a current priority with our Design Partners. Ultimately the API boundary may need to shift, e.g. if we make a big push for ADBC, so we can't guarantee stability for in-process usage right now. There are no active plans to change anything soon though - happy to chat if you're curious!

good to know, thanks!

Hey @oscarlinusericsson - just to add some extra colour here, v2 doesn't have the same all-the-data-lives-locally restriction as v1, but can also support embedding. For ~ideal in-process performance we would look to adding ADBC support in the future, per https://github.com/xtdb/xtdb/issues/3395

Thanks for more information. Is DuckDB part of the default in-memory query engine in XTDB2? I'm sorry that I'm asking such basic questions, but I really don't get how XTDB(2) works in terms of how and where data is availiable/cached/in process/transmitted over network etc.

No problem at all, we've not described the internals of v2 openly enough yet. But it's ~all custom, built on top of the official arrow-java lib (although we maintain a fork). So parser, planner, execution, storage etc. is all written by the XT team and lives in the codebase. DuckDB is an unrelated technology, although shares some similar traits. They've found an interesting niche (ML / data analysts) which XT2 isn't directly targeting (we're focusing on people building apps instead) > how and where data is availiable/cached/in process/transmitted over network etc. the diagram in the readme is mostly accurate still https://github.com/xtdb/xtdb - the summary is that data is only durable via external s3/kafka, but warm local caches are needed to make things fast. Compaction jobs (LSM-style, writing back to S3) also run in the background so that point-query primary key lookups can stay fast in the face of growing history volumes

If you're looking to set XT up in-process, the guide will be the same for the AWS infrastructure (S3, SNS, MSK), just that you'll be in charge of deploying the application itself

Tbh though, unless you have specific requirements for XT2 to be in-process, we'd recommend having your XTDB nodes deployed separately with v2, rather than coupled to the lifecycle of your application.

That’s good to know, I’m only starting the prototype and was wondering whether to go with HTTP client or in-process, I don’t fully understand the tradeoffs

so, deployment wise, only ECS and the image would change, from the default http://ghcr.io/xtdb/xtdb-aws-ea:latest to my custom one?

In-process is likely simpler if you've only got a one-node setup - for a non-critical app, for example, or unit tests. Otherwise, for production usage, long-running XTDB nodes are preferable, give them chance to build up a nice warm cache. If you're deploying your application frequently, depending how you've set it up (particularly if you're using immutable architecture patterns, say) chances are the XT caches will get cleared every time.

understood, that makes sense, thanks! 🙏

A good rule of thumb is that if you've got the kind of app where you're considering MSK and S3, you've probably also got the kind of app where you'll want them decoupled - in-process nodes tend to be single-node, and use local disk or in-memory storage

I didn't realize you could have an in-process node that was persistent to disk -- I've only used them in-memory so far...

@alen would you have the ability to explain how you deployed the in-process node? I'm also developing a prototype and want to keep costs low, but I want it on a persistent server so that I can actually use my prototype from many different places.

I set up a client in the end, but deployed to a cheap digital ocean VM (optionally you can buy some storage volume, but you probably don’t even need it if prototyping)

3 days later... yup, I'm definitely switching to an in-process node for now - AWS is a bit too pricey for a prototype 💰 😅