Fork me on GitHub
#crux
<
2020-08-18
>
nivekuil06:08:18

fyi the link in the github readme of https://github.com/juxt/crux to the official documentation is broken atm

jarohen08:08:19

thanks - we'll get right on it

flefik11:08:08

When googling crux this URL often shows up near the top https://opencrux.com/docs , but it 404s. Perhaps it would be desirable to update this deadlink and redirect it to https://opencrux.com/reference/get-started.html ?

jarohen11:08:21

thanks - we're in the process of migrating our docs site at the moment, although I can certainly put a redirect in

flefik11:08:35

I am trying to write a query to retrieve the histories of all documents in a database that share a certain attribute value. Is there a way to efficiently construct this query in crux?

flefik11:08:08

I am trying to construct a view of all changes that affected a customer-id in chronological order for audit purposes

refset12:08:50

If you don't know when the changes are, then you will have to resort to scanning of some kind. It's possible we could expose something to help make it slightly more efficient but really the best answer is to update a dedicated customer entity each time some change that relates to it happens. Is that possible?

flefik12:08:42

yes, my current workaround is to have a customer entity per customer, that has an :event. Then I use (crux/listen ,,, to filter on events I'm interested in and then :crux.tx/put a new version of that customer entity with the latest change embedded.

flefik12:08:01

not yet sure if i can guarantee that events coming on crux/listen are only processed once or how to ensure I don't miss any

flefik12:08:29

it might be better to not listen and instead create the event manually for every change, but that seems a little cumbersome

refset13:08:54

Hmm, so you are writing to Crux from different/independent processes here? i.e. one process that only submits events and another which does other things (including, currently, processing these events)?

flefik13:08:10

yeah that's right

refset13:08:14

Listen events will be processed once per node so are unlikely to be appropriate for generating further puts, though I don't grok the use-case. It seems more likely that you want to be using a transaction function

flefik13:08:38

i've just discovered that. in the process of refactoring now so that for every change i want audited, i also submit a copy of that change in the :event attribute to the customer-entity.

flefik13:08:18

That should allow me to get a chronological list of changes with the history api. It's a little brittle, but I think it's fine for now. It's not huge amounts of data

refset13:08:40

Great, that sounds about right to me. Feel free to share a snippet if you would like reassurance about how brittle (or not) it is

refset13:08:58

A couple more thoughts to make sure my advice is sound: are you assigning each event a UUID id such that each event is its own entity? Are you always going to have monotonic events or will you need to be able to (re-)process events retroactively?

flefik14:08:56

> A couple more thoughts to make sure my advice is sound: are you assigning each event a UUID id such that each event is its own entity? No I was going to just overwrite the same document

flefik14:08:13

> Are you always going to have monotonic events or will you need to be able to (re-)process events retroactively? These will always be monotonic

flefik14:08:27

Thanks for all your help Jeremy

refset14:08:37

thanks, good to know, and no problem! 🙂

flefik23:08:44

@U899JBRPF I settled on something like this:

(ns api.endpoints.events
  (:require [clojure.spec.alpha :as s]
            [crux.api :as crux]))

(s/def ::organization-id uuid?)
(s/def ::entity (s/keys :req [:crux.db/id]
                        :req-un [::organization-id]))

(defn audited-put [node entity]
  {:pre [(s/valid? ::entity entity)]}
  (let [oid (:organization-id entity)]
    (crux/submit-tx
      node
      [[:crux.tx/put entity]
       [:crux.tx/put {:crux.db/id {:events oid}
                      :type       :event
                      :action     :put
                      :event      entity}]])))

flefik23:08:48

one drawback is (I think) that I can only really update one entity per org per submit-tx, but I think that's fine for now. (Assuming you can't update the same document multiple times in the same transaction)

refset23:08:45

nice 🙂 although on that final line you probably want to have (:crux.db/id entity) though, otherwise the entity doc effectively gets stored twice in the document (taking up space)

refset23:08:45

you're right that you can't transact multiple versions of an entity at the same valid time (range) in the same transaction - the indexer only indexes the last version in the transaction. You can validate the behaviour like this:

(c/submit-tx node [[:crux.tx/put {:crux.db/id :foo-1 :a 1}]
                   [:crux.tx/put {:crux.db/id :foo-1 :a 2}]])

(c/entity-history (c/db node) :foo-1 :asc {:with-docs? true :with-corrections? true})

; =>
[{:crux.tx/tx-time #inst "2020-08-20T23:48:30.307-00:00",
  :crux.tx/tx-id 56,
  :crux.db/valid-time #inst "2020-08-20T23:48:30.307-00:00",
  :crux.db/content-hash #crux/id "453a7bd059ac67d80cf95b642c0773413ada1e52",
  :crux.db/doc {:crux.db/id :foo-1, :a 2}}]