xtdb 2021-06-18 | Slack Archive

Good news - Crux spike added to our next sprint 🎉 Lots of question incoming :umbrella_with_rain_drops: 🙂

🙌 9

Yay 🙂 we'll be waiting!

❤️ 3

Hi everyone! Newbie here so sorry if this has been covered. Does anyone know if it's possible to configure Crux with any of the back-ends to encrypt data at rest? Theoretically RocksDB supports this (I read through the C++ PR) but I haven't been able to find a single example of how to configure this in Rocks or any of the other back-ends. (I'm using Crux in embedded mode in a small low-volume server, not standalone.) Thanks in advance!

refset17:06:16

Hey 🙂 I've looked into this briefly before, and seen (presumably) the same RocksDB PR, but I'm not aware of anybody attempting to use it with Crux thus far. Are you explicitly hoping to avoid unnecessary performance costs of fs/block/disk-level encryption?

David Orme17:06:59

> Are you explicitly hoping to avoid unnecessary performance costs of fs/block/disk-level encryption?

David Orme17:06:55

No; from a security point of view I'm more worried that my own server gets cracked than that the hosting provider does.

David Orme17:06:29

Thus: the desire for the database files themselves to be encrypted.

refset19:06:00

Ah okay, that's interesting. I suspect that even with Rocks' baked-in encryption, if your API server is compromised then all bets are off - although I appreciate that having multiple layers never hurts. Is there a compliance / regulatory motivation behind this also?

David Orme21:06:46

> compliance Yes. PII.

refset21:06:52

Got it :thumbsup: Well, if you'd be interested to meet & discuss the options next week I'd be more than happy to spend some time looking at this again in advance

henrik17:06:32

Is pull-many more efficient than several individual pulls?

jarohen18:06:23

yep 🙂 pull-many still only needs one Crux query - it's essentially a select ... where id in (...) rather than select ... where id = ...

👍 3

henrik19:06:58

My SQL is really rusty, so I’m not sure I follow the implication of the example. But I’ll interpret it as there’s some overhead being eliminated :)

henrik19:06:03

I spent another second thinking about what you wrote, and I understand now.

☺️ 6

henrik09:06:58

So, I guess pull-many will return results out of order (as compared to the inputs), and will omit entries where nothing is found. Is this documented somewhere? I looked around and couldn't find anything written about it. If not, it'd be great to have it added to the documentation. This was surprising when working with Pathom, which relies on both qualities (correct order, same number of entries out as came in).

Jacob O'Bryant21:06:13

Sort of a wacky idea I had the other day: if you made a dynamically resolved transaction function, you could use it to create materialized views/whatever the accepted term is from domain events:

(ns foo.handlers)

(defmulti handle-event :handle-event)

(defmethod handle-event :foo
  [ctx event]
  ...)

(crux/submit-tx
  node
  [[:crux.tx/put
    {:crux.db/id :handle-event
     :crux.db/fn '(fn [ctx event]
                    ((requiring-resolve 'foo.handlers/handle-event)
                     ctx event))}]])

(crux/submit-tx
  node
  [[:crux.tx/fn :handle-event {:event/type :foo
                               ...}]])

Then when you want to modify an event handler, you just re-index. (Right?). Not saying it's a good idea, but maybe something fun to try on a week end... though my apps are simple enough that I am for now resisting the urge to go down the event sourcing/ddd rabbit hole too far 😉

😳 2

🙂 2

jarohen21:06:08

interesting 🙂 been wondering about how best to get Crux and event sourcing working together. we've also got a (mostly internal but I don't really see it going away) begin-tx API talking directly to the index store that I've considered using for this kind of use case

jarohen21:06:26

maybe there's room for a little component that sits between an external tx-log and Crux, transforms the events as necessary and calls that API

jarohen21:06:52

there's not a lot in Crux that requires our tx-log - submit-tx, obviously, but if you've got a separate log you likely won't be calling that. good ol' unbundling 🙂

Jacob O'Bryant22:06:41

that sounds like it'd be pretty nifty. for a little while last year I thought we had reached nirvana because of the https://materialize.io launch... but after using it for a while I realized it has memory constraints that make it unsuitable for arbitrary web apps 😞

2021-06-18

Channels