Fork me on GitHub
#xtdb
<
2020-10-10
>
Luis Santos23:10:21

How do I fetch documents from the transaction log? I'm current using open-tx-log to iterate the transaction log but I cannot find the correct API to fetch the documents. (iterator-seq (:lazy-seq-iterator (api/open-tx-log crux nil nil))) Result: ({:crux.tx/tx-id 0, :crux.tx/tx-time #inst "2020-10-10T23:03:20.284-00:00", :crux.tx.event/tx-events [[:crux.tx/put #crux/id "c5beef2de776a9eb64b6a612abeac7157e1e00a4" #crux/id "0ab888b62775eea2eb2fffe10c9f6bfbf661a792"]]}) Thanks.

refset23:10:02

Hey, you can use true as the third argument to open-tx-log ("with-ops?" in the doc string) to get the documents back at the same time

Luis Santos07:10:35

Thanks :face_palm:

Luis Santos09:10:42

I played a little bit with and I realized it doesn't exactly do what I expected. What I would like to do is to iterate the tx-log and access the entire entity history not necessary the transaction details. I'm trying to use the tx-log as an event store and build a materialized view in Eliasticsearch. I hope it makes sense.

victorb09:10:41

@UK00KCK6D maybe what you're looking for is crux/entity-history?

Luis Santos09:10:37

That works for a single entity given the eid. But how do I get from a transaction id or #crux/id to the eid? Example: {:crux.tx/tx-id 12, :crux.tx/tx-time #inst "2020-10-11T09:52:46.808-00:00", :crux.api/tx-ops ([:crux.tx/fn #crux/id "355705cf62a56669303d2561f29e0620a676c36e" {:crux.db/id #uuid "dd8396cc-49fc-4eaa-9244-d34f32846bd1", :crux.db.fn/tx-events [[:crux.tx/put #crux/id "c65463d9b3458942c7b2b9efe9f79f93f0d6e353" #crux/id "8e8ed569e93ed32379bf44998d3b7d1e50812dee"]]}])} I don't know how to get from this to the entity version.

Luis Santos09:10:44

Thanks for the help.

refset13:10:14

Hmm, I'm not sure I understand the requirement still. Can you describe specifically how open-tx-log doesn't do what you expected? It seems possible that you might be wanting to reify the transactions into the database as additional documents (there are various approaches, with different tradeoffs)

Luis Santos21:10:50

@U899JBRPF I think I found the problem. open-tx-log with-ops? doesn't work with transactions deeper in the tree. I've been playing with transaction functions and it seams that with-ops doesn't work correctly while using transaction functions. It doesn't fetch the documents for the transactions created by the transaction function. Example:

{:crux.tx/tx-id 13,
  :crux.tx/tx-time #inst "2020-10-11T10:06:30.451-00:00",
  :crux.api/tx-ops
  ([:crux.tx/fn
    #crux/id "355705cf62a56669303d2561f29e0620a676c36e"
    {:crux.db/id #uuid "0f3b6190-d8aa-4fdd-8623-7b062e95b447",
     :crux.db.fn/tx-events
     [[:crux.tx/put
       #crux/id "c65463d9b3458942c7b2b9efe9f79f93f0d6e353"
       #crux/id "0eaf7492df47b3867980e992840d5087716cfc4f"]]}])}
I had to use the following to access the transaction
(def lax-sub-tx (:crux.db.fn/tx-events (last (last (:crux.api/tx-ops last-tx)))))

(conform/tx-events->docs (:document-store crux) lax-sub-tx)

refset22:10:29

Ahh, okay that makes sense, thanks. I agree it feels sensible that it should retrieve the nested docs also (I think these should only ever be 1-level deep). I will make a note on the project board for the team to discuss changing behaviour in the next release 🙂 Nice job figuring out the workaround - but as it's not an officially supported API...let's stay in touch!

👍 3
Luis Santos09:10:42

I played a little bit with and I realized it doesn't exactly do what I expected. What I would like to do is to iterate the tx-log and access the entire entity history not necessary the transaction details. I'm trying to use the tx-log as an event store and build a materialized view in Eliasticsearch. I hope it makes sense.