datahike 2022-02-05 | Slack Archive

awb9919:02:55

Can someone help me to write a transaction? I have :invoice/id and :lineitem/invoice-id :lineitem/sku :lineitem/qty :lineitem/id. I have a edn invoice with the line items in a seq. So the data that I have needs to add invoice and then link the lineitems to the invoice. Now it gets tricky: i receive a stream of invoice updates. So any time an invoice changes I get all tge data if the invoice. When I feed it to datahike I need to make sure the lineitems in datahike are equal to the lineitems in the invoice. Now most invoice Updates I will receive will not change the lineitems. But it us possible that lineitems have been changed. New lineitem added. Existing lineitem modified, New lineitem Added. I wonder if this can be done all in one transact? So I struggle to understand how deal wirh the facts thst I add a new invoice or lineitem, or I might update existing ones. And I have to avoid a situation where datahike db would still have removed lineitems in the db. Thanks.

Björn Ebbinghaus20:02:02

Transactions are just vectors. So you can build up a vector that makes all the changes at once. This is a great idea in general, as this allows you to have pure functions producing parts of a single transaction as data. The transaction can actually be executed on the edge of your logic. Here is an example:

(defn retract-lineitem-tx [lineitem]
  [[:db/retractEntity [:lineitem/id (:lineitem/id lineitem)]]])

(defn updated-lineitem-tx [current-lineitem new-lineitem]
  (let [ident [:lineitem/id (:lineitem/id current-lineitem)]]
    ;; Your job to implement. Based on what can change. Maybe new-lineitems have more/less attributes..? 
    ;; You should return a valid transaction:
    [[:db/add ident ...]
     [:db/add ident ...]
     [:db/retract ident ...])

(defn group-lineitems [current-invoice incoming-invoice]
  ;; The implementation is up to you
  ;; Maybe return something like:  
  {:new ...
   :updated ... 
   :removed ...})

(defn update-invoice-tx 
  "Builds a transaction that retracts all lineitems that are only in `current-invoice` and adds all lineitems that     are only in `incoming-invoice`."
  [current-invoice incoming-invoice]
  (let [invoice-id (:invoice/id current-invoice)
        {:keys [new updated removed]} (group-lineitems current-invoice incoming-invoice)]
    (concat 
      (mapv retract-lineitem-tx removed)
      (mapv updated-lineitem-tx updated
      (mapv #(assoc % :lineitem/invoice [:invoice/id invoice-id]) new-lineitems)

(defn insert-invoice! [conn invoice]
  (let [tx (if-let [current-invoice (invoice-by-id @conn (:invoice/id invoice))]
             (update-invoice-tx current-invoice invoice)
             (insert-invoice-tx invoice))]
    (d/transact! {:tx-data tx})))

🦸 1

awb9921:02:44

Wow! Thanks a lot @U4VT24ZM3 ! I read everything I could in regards to datasccript / datomic. But I wouldn't have cone up with anything near as elegant! Thanks a lot!

awb9923:02:49

I am creating a app where I store everything that musy be queryable and I care a lot in datahike and all data that I get from other systems in konserve store. So in konserve I store data by type with vec as keys [:invoice 17] [:tracking 13]. What I need from konserve is to get a list of all :invoice items. Or all tracking items. I am not sure if I should use konserve update-in and then use (k/get-in [:invoice]) to get all invoices and (k/get-in [:invoice 17]) to get a specific invoice. I guess that this could slow down transactions? And if i use (k/keys) then I have to use konserve 0.6 but datahike currently uaes konserve 0.5. So this mkght fuck things up. I also could store all invoice ids in konserve in datahike. But then I might have a problem. If the konserve save didn't work and datahike did store it.

timo07:02:50

We are quite confident in konserve 0.6 so you could just use datahike with konserve 0.6. There will be a 0.6 release soon.

timo07:02:05

I am not sure how this k/get-in should slow down transactions considerably. Maybe you can elaborate some more?

timo07:02:13

I also don't understand why you don't want to store the invoice-ids in datahike, are these too many? When you don't go via datahike you won't be able to query them via datahike because they are not indexed. I never tried this and could imagine other issues to come up.

awb9916:02:38

I dont know internals if konserve. But I imagine konserve stores just the key [:invoice] and if I do update-in [:invoice 13] I fear konserve has to write again all elements in [:invoice]. This is my concern.

awb9916:02:35

My reason for wanting to get all existing keys in konserve is for possible inconsistencies. Say the konserve store fails for an invoice because it contsins a non serializable value or something else fails.

2022-02-05

Channels