Fork me on GitHub
#xtdb
<
2023-05-24
>
Namit Shah08:05:41

I was playing around with XTDB 2 and wanted to check if it is currently not possible to use some storage system to store the XTDB documents. Is it only in-memory right now? I know that XTDB 2 is still under development, so I understand if this is not yet possible. However, if it is possible, I would be interested in learning more about how to do it.

jarohen10:05:03

Hey @U04DDKZJERF 👋 We do have initial implementations for a single-node filesystem tx-log/object store, as well as multi-node Kafka and AWS S3 adapters, although these are quite primitive at the moment.

Namit Shah10:05:25

I just found you message in the discuss. Trying out that right now to store the data and work on it. Your message for using local storage: https://discuss.xtdb.com/t/rocksdb-backend-for-v2/208

jarohen10:05:11

I don't think we have much documentation on those as yet, but as a quick guide: • you can specify a local directory tx-log by adding this to your config: {:xtdb.log/local-directory-log {:root-path "/path/to/log/dir"} • a local directory object-store with {:xtdb.object-store/file-system-object-store {:root-path "/path/to/obj-store/dir"}} (excuse the inconsistent naming here, this'll likely change) • a Kafka tx-log with {:xtdb.kafka/log {:bootstrap-servers "localhost:9092", :topic-name "my-topic"}} • an S3 object-store with {:xtdb.s3/object-store {:bucket "my-bucket"}}

❤️ 2
jarohen10:05:19

(you'll need to include com.xtdb.labs/xtdb-kafka "2.0.0-SNAPSHOT" and/or com.xtdb.labs/xtdb-s3 "2.0.0-SNAPSHOT" in your build tool to get support for those two modules)

Namit Shah10:05:13

Thanks. This helps a lot.

🙂 2
🙏 2