OK, I found backup in rocksdb: https://github.com/juxt/crux/blob/master/crux-rocksdb/src/crux/kv/rocksdb.clj#L109

searching for how to restore such a backup... (maybe just pass its path to crux/start-node? I'll give that a try...

I tried it for the lmdb backend and it works. Nice!

lmk if you'd rather me delete this monologue 🙂

hey! i went through investigating this recently too, there is this sample: https://github.com/juxt/crux/blob/5991a94545d088690e7fc244b6a8be5587140ffc/docs/example/standalone_webservice/src/example_standalone_webservice/main.clj#L740

thank you!

yes, that seems better and more general than fiddling with the underlying stores directly

for rocksdb at least (not sure about lmbd) you can copy the data directories via the filesystem as long as the node is not running

thanks! I don't want to have to stop the node in order to back it up though

thanks for the detailed explanation!

indeed - if memory serves, that's how crux.kv/backup is implemented for LMDB

15 minutes?! That's quite a big difference since the last time I looked. I think you could probably get something work well with sub-1GB data sets very handsomely, and perhaps a little bigger than that too depending on S3->Lamdba transfer speeds

would have to consider how/whether you can keep it warm somehow, I doubt it'd be financially viable if it needed to boot up every time

I wondered if one could decompose crux in a way that there is a single lambda processing the tx and updating a share kv store which is used by the lambda. But I think one major blocker is that the kv store must support read isolation (snapshots).

@U054UD60U we've not done any experimentation ourselves yet, but it's worth looking at https://github.com/rockset/rocksdb-cloud which has exactly the properties needed. In theory it's a drop-in replacement for normal RocksDB

That’s promising

Yeah, the Rockset team has a lot of the ex-Facebookers that have been working on RocksDB for many years. They've spent time decomposing RocksDB itself so that you have compaction workers etc. there are some good videos / blogs around with more info, e.g. https://rockset.com/blog/remote-compactions-in-rocksdb-cloud/

I wonder if there are java bindings for rocksdb-cloud though. The repo is confusing

🙂

I would start by copying everything (including Java bindings) in crux-rocksdb to create crux-rocksdb-cloud and then figure out how to swap-out the binary. Just guessing though 😅

hehe, if I only had time.

Maybe I’ll try https://github.com/csm/konserve-ddb-s3 it was used by another crux-on-aws spike IIRC. In any case I think the single-tx-processing lambda would be key. No gain in having multiple lambdas do the processing concurrently.

Concurrent tx submission is good for a) high-availability and b) separation of concerns for deploying multiple apps operating against a single instance Also checkout the CruxIngest API which is purely for handling submissions

Which tx-log are you looking at using?

I've created a dynamodb tx-store.

You can submit concurrently to that log but I wonder if you would need multiple lambdas to process the tx into one shared kv store.