Fork me on GitHub
#xtdb
<
2021-03-31
>
tobias09:03:45

Suppose I start with Rocks DB for docs and log storage and later want to switch to Kafka or something else. How hard would the migration be? Apologies if this is covered somewhere and I missed it.

refset11:03:49

Hi @UA9399DFZ there's an item on our roadmap to provide some tooling to help with this, see https://github.com/juxt/crux/issues/1386 You can get most of the way today by transforming and streaming the output from open-tx-log of one Crux instance into submit-tx of another instance. However that approach alone doesn't account for :crux.tx/match or :crux.tx/evict operations, and we'd prefer it to be even simpler. In summary it's definitely possible to migrate between backends, but we'd probably have to assist you as things stand today. We'll try to give this area some attention over the coming weeks.

refset11:03:59

Do you think it would acceptable if transaction times & transaction IDs are not preserved in the migration? Whilst it should also be possible in most (all?) cases, it would be significantly more complex.

tobias11:03:48

Great to hear that tooling for migrating backends is on the roadmap. I don't need it with any urgency, but knowing that it's coming takes away some of the fear of getting my initial setup "perfect". Preserving transaction times and IDs wouldn't be important for my use case, which is just basic CRUD apps.

šŸ‘ 3
refset12:03:13

Thanks, that's good to know, we'll get cracking šŸ™‚

Las13:03:58

Hi there, I have been playing with Crux a little bit recently. I noticed that for single node use cases, SQLite is an option for doc, log (and index?) store. Iā€™m wondering if anyone tried https://dqlite.io in a multi-node setting.

refset14:03:50

Hi šŸ™‚ SQLite isn't supported as a back-end for the KV indexes today. I can't think of a reason why it wouldn't be possible to make it work but the performance won't be nearly as good as with Rocks or LMDB. These ~distributed SQLite stores do look interesting though. I've not had the chance to play with any of them yet - count me interested also!

refset14:03:25

The KV protocol is fairly small if you fancied the challenge though, here's a Redis-based prototype I wrote: https://github.com/crux-labs/crux-redis/blob/master/src/crux/redis.clj

šŸ‘ 3
Las11:04:13

Cool, I will see if I can carve some time out next week.

nivekuil17:03:41

maybe I'm dreaming, but if crux were to integrate arrow into the indexes, would it then be possible to use something like https://github.com/techascent/tech.ml.dataset in situ, i.e. on the local node without any ETL?

šŸ‘ 6
jonpither17:03:19

Stay tuned.

9
jarohen17:03:21

we're certainly keeping an eye on it, looks a great development šŸ™‚

šŸ‘ 9