Fork me on GitHub
#xtdb
<
2020-05-19
>
ordnungswidrig13:05:28

Could you use backup/restore to kickstart a lambda instance running a crux node?

refset13:05:10

If your node is always going to be small (e.g. < 100 MB) and you don't have strong latency demands then I expect it could be viable, I've not yet heard of anyone trying 🙂

ordnungswidrig13:05:59

I just thinking about these low volume hobby projects you whant to “drop” somewhere at minimal cost.

👍 4
ordnungswidrig13:05:34

One could also use ddb for the kv store, but not sure if that’s viable. You’d need to ensure there’s only one indexer running at a time

ordnungswidrig13:05:11

I’ve got an experimental ddb tx log already which seems to work.

refset13:05:37

ddb as a kv backend will be slow, as the indexes are designed to be local to the query engine, but perhaps that's also good enough for hobbyists

refset13:05:49

Happy to help to any case!

ordnungswidrig13:05:13

regarding ddb kv, a query in crux will hit the kv store many times, right?

refset13:05:55

that's correct, it's "chatty" because the various ranges of index tuples are lazily and individually (i.e. one at a time) streamed out as the query executes

ordnungswidrig13:05:55

does a memkv need to fix into memory?

refset13:05:39

not sure what you mean by "fix" in this context

ordnungswidrig13:05:14

it means “fit” 😛

🙂 4
refset15:05:59

ah, yes it does need to fit into memory indeed. The memkv implementation also makes very little attempt to be compact, performance-tuned or otherwise efficient, as we only built it for testing purposes, though there may be valid memory-only use-cases that justify us revisiting it with more engineering energy in future

ordnungswidrig15:05:49

so in my (toy) use case using rocksdb restore from a backup/snapshot on S3 might work best I guess.

👍 4
ordnungswidrig08:05:07

@U899JBRPF is there a test suite for crux Document Stores and TxLogs? I know about crux-bench but I found it non-trivial to have it run as it seems to interfere with my own AWS credentials when trying to download some bench data from S3.

refset09:05:57

Hey, yep we have some protocol level test namespaces. See the recently added s3 module https://github.com/juxt/crux/blob/cfeb368a5978b3a42d6b2a8f7c698d489e0f988b/crux-s3/test/crux/s3_test.clj

refset09:05:40

Running all the standard tests is also a good exercise

refset09:05:02

the doc_store_test specifically: https://github.com/juxt/crux/blob/cfeb368a5978b3a42d6b2a8f7c698d489e0f988b/crux-test/src/crux/doc_store_test.clj we don't have an equivalently narrow-scoped version for tx-log right now

ordnungswidrig11:05:45

Do these tests exercise concurrent access? I’m not talking about jesper-level verification rather at least simple invariants.

refset12:05:56

I don't think so. Not yet. The doc store test shouldn't really care, as writing & reading docs ought to be idempotent and naively concurrent (since docs either don't exist, exist, or are permanently tombstoned -- there should be no mutations or possibility for race conditions), but a hypothetical tx-log implementation test would ideally prove the essential linearisable behaviour in the face of large amounts of concurrent writes & reads

nivekuil23:05:46

I'd like to do a transaction analogous to a SQL delete where , matching on some attribute. What's the right way to do this in crux? I could query and then delete, but that seems inefficient over the HTTP api.

refset23:05:11

Hi, are you looking to do everything over HTTP in general? We have some new transaction function capabilities in the works currently that will make your requirement very simple to solve, but it's not released quite yet. In the meantime though, you are not going to be able to avoid a two stage process to achieve this over HTTP

nivekuil23:05:34

My specific use case is a little peculiar: I'm actually writing an emacs lisp org-mode based client to crux, of sorts. So yes I think I am stuck with HTTP for the foreseeable future 😛 It also means I can live with slowness, but I'm also happy to try out the new capabilities.

nivekuil23:05:25

Actually, I was leaning towards extending or replacing the built-in REST api with more expressive endpoints, but it sounds like that may be in the works on your end already.

refset23:05:59

Wow, that sounds pretty cool! If you're feeling especially curious you could try building and using the in-flight PR as the main chunk of work was literally just finished today, but otherwise please do check back in a week or so for the next release.

refset23:05:31

I think virtually everyone is just embedding Crux, as it's so much easier to do many things. Forking crux-http-server to build your own application-specific HTTP module could be sensible :)

nivekuil23:05:57

Apparently I might not be stuck with HTTP, having stumbled upon https://github.com/clojure-emacs/clomacs which appears to be FFI done through nrepl. I think I'll play around with this while waiting for the next release. Thanks!

👍 8
nivekuil05:05:33

fyi, I'm not sure how to try master using deps.edn. Going off https://github.com/bhauman/rebel-readline/issues/176 it seems like crux may need to add a deps.edn at the top level, if you feel such an accommodation appropriate.

refset07:05:07

Specifically I meant running lein sub install within the transaction function branch for the PR - I'll discuss the possibility of adding a deps.edn with the team today