xtdb 2021-11-02 | Slack Archive

tatut08:11:09

the docs on S3 checkpointer (https://docs.xtdb.com/storage/1.19.0/aws-s3/#checkpoint-store) seem out of date, can't find modules with those names

refset09:11:04

Interesting, this looks a mistake in the docs (s/checkpoint/cp), and instead you want :xtdb/module 'xtdb.s3.checkpoint-store/->cp-store , see https://github.com/xtdb/xtdb/blob/e2f51ed99fc2716faa8ad254c0b18166c937b134/modules/s3/src/xtdb/s3/checkpoint.clj#L83

refset09:11:05

Thanks for mentioning it!

Thomas Moerman09:11:47

Hi, I encountered a problem while running with-tx within a transaction function: No implementation of method: :latest-completed-tx of protocol: #'crux.db/IndexStore found for class: crux.kv.index_store.KvIndexStoreTx I'm guessing I hit this known issue: https://github.com/xtdb/xtdb/issues/1434? Some context: I'm implementing a tx-fn that "moves forward" the state of a workflow and its related tasks (reified as an aggregate of crux documents) in response to an event. My idea was to write the tx-fn body as a recursive loop, because some events can "spawn" other consecutive events (e.g. auto-completion of Tasks), all of which should be handled atomically. During the recursion, state changes are accumulated in a vector of transactions, which I intended to use as input for computing speculative db instances at certain points in the recursion, and launch datalog queries on these to inspect for certain conditions (e.g. to check whether the recursion is done). Of course a workaround with more "manual" inspections is possible, the speculative db inspections seemed a very elegant approach though. Perhaps I am stretching the intended purpose of transaction functions here? What do you guys think? Any other approach worth considering? Thanks

Thomas Moerman10:11:59

I'm attempting to work around this by not using a transaction function and accumulating the update transactions including tx/match entries to ensure the rug wasn't pulled underneath us while computing the changes. Caveat: thinking aloud here, cfr rubber ;-)

refset10:11:43

> During the recursion, state changes are accumulated in a vector of transactions I can't quite visualise how the recursion might work...but is there no obvious way to chain the accumulation of the ops but only do the with-tx once (repeatedly)? Can you post an example?

Thomas Moerman17:11:44

The problem I encountered was that computing a speculative db within a transaction function fails, even when with-tx is called only once:

(def workflow-driver
  {:crux.db/id  WORKFLOW-DRIVER-ID
   :crux.db/fn  '(fn [ctx params]
                   (let [db  (crux.api/db ctx)
                         ;; CAUSES EXCEPTION
                         db* (crux.api/with-tx db [[:crux.tx/put {:crux.db/id "123"
                                                                  :bla/msg    "Hello, world"}]])]
                     [[:crux.tx/put {:crux.db/id "456"
                                     :msg        "Hello again"}]]))})

Only tested on this version:

{:mvn/version "21.02-1.15.0-beta"}

Thomas Moerman17:11:10

I worked around the issue by: 1. not using a tx-fn (as it is quite a complicated computation, probably a bit heavy for a tx-fn) 2. doing what you suggest: accumulating the transactions in a collection and computing speculative-db on the "base" db using these. 3. using some match transactions here and there for sanity checks So far it's looking good :thumbsup:

🙂 1

refset23:11:22

Ah, I see now...yes this is definitely the same fundamental issue as you linked initially. I'm glad you figured out an acceptable workaround(!)

tatut13:11:32

is there something special needed for checkpointing lucene, I got the regular LMDB index checpoints to work with S3, but the lucene doesn't seem to do anything

tatut13:11:44

I configured separate checkpointers for indexes and lucene

tatut07:11:38

never mind, it seems the random time was just different, eventually the lucene checkpoint was uploaded as well

tatut07:11:47

but should you share the same checkpointer for both?

refset10:11:49

> should you share the same checkpointer for both? we did discuss this internally at the time of adding checkpointing to Lucene, but we decided to keep the modules isolated and didn't want to create a "centralised" checkpoint system

refset10:11:07

glad you figured it out okay 🙂

2021-11-02

Channels