Now that my attention has been brought to https://docs.xtdb.com/administration/1.22.1/checkpointing/ - that would be another place to clarify! Because its very first sentence says "so that new nodes can start to service queries faster." But that did not seem to be my problem. My node started to service queries right away!! The problem was that the answers were all wrong.

Hey @U0HG4EHMH interesting, thanks for the feedback! Note that if you configure persistent indexes then there's no need to replay from scratch when starting, although arguably it's probably still a good idea to sync anyway (e.g. if you have other nodes submitting transactions). In the not too distant future we plan to add an initialisation phase that could provide better visibility & intuitive clarity to the situation: https://github.com/xtdb/xtdb/issues/1838

I'm sorry this was confusing, but I'm very glad you figured it out 🙂

I don’t know if fetching from EFS is faster than from S3 but atleast I wouldn’t need any special S3 configuration

one obvious downside is that I would need to handle clearing old checkpoints myself, now we just have an S3 bucket lifecycle policy that clears old files automatically

At least https://github.com/xtdb/xtdb/blob/master/project.clj#L47 shows that

https://github.com/xtdb/xtdb/blob/1.22.1/modules/google-cloud-storage/src/xtdb/google/cloud_storage.clj#L11 how does this even work? Is it completely a stub implementation?

Aha, so it just makes sure that the NIO provider is included, and after that all NIO Paths globally understand the gs:// prefix

Ok, so it was a case of google-api-client 2.0.0 flowing in through other deps, and that being incompatible with the version of google-api-services-storages that the google-cloud-nio used in xtdb. Weird error message being google's fault

Here's a code snippet I've been evaluating:

[(xt/q db
       '{:find [(pull profile [*])]
         :in [username]
         :where [[profile :profile/username username]]}
       "jacob")
 (xt/q db
       '{:find [(pull profile [*])]
         :in [user]
         :where [[profile :profile/user user]]}
       (:xt/id user))
 (->> (xt/q db
            '{:find [(pull profile [*])]
              :where [[profile :profile/user]]})
      (map first)
      (filter #(= "jacob" (:profile/username %))))]

[#{}
 #{[{:profile/username "jacob",
     :profile/title "Jacob's profile",
     :profile/user #uuid "e86e5e14-0001-46eb-9d11-134162ce930f",
     :xt/id #uuid "3a98d62c-3258-4bb3-84dd-378ee4a4c7b1"}]}
 ({:profile/username "jacob",
   :profile/title "Jacob's profile",
   :profile/user #uuid "e86e5e14-0001-46eb-9d11-134162ce930f",
   :xt/id #uuid "3a98d62c-3258-4bb3-84dd-378ee4a4c7b1"})]

[#{[{:profile/username "jacob",
     :profile/title "Jacob's top 8",
     :profile/user #uuid "e86e5e14-0001-46eb-9d11-134162ce930f",
     :xt/id #uuid "3a98d62c-3258-4bb3-84dd-378ee4a4c7b1"}]}
 #{[{:profile/username "jacob",
     :profile/title "Jacob's top 8",
     :profile/user #uuid "e86e5e14-0001-46eb-9d11-134162ce930f",
     :xt/id #uuid "3a98d62c-3258-4bb3-84dd-378ee4a4c7b1"}]}
 ({:profile/username "jacob",
   :profile/title "Jacob's top 8",
   :profile/user #uuid "e86e5e14-0001-46eb-9d11-134162ce930f",
   :xt/id #uuid "3a98d62c-3258-4bb3-84dd-378ee4a4c7b1"})]

For now I'll probably lookup documents with the third query--i.e. load them all and then filter outside the query. It's fast enough for my situation, though obviously a bit unsettling 😬

Here's a transaction I just ran which caused the bug to manifest:

([:xtdb.api/match
  #uuid "3a98d62c-3258-4bb3-84dd-378ee4a4c7b1"
  {:profile/username "jacob",
   :profile/title "abc 123",
   :profile/user #uuid "e86e5e14-0001-46eb-9d11-134162ce930f",
   :xt/id #uuid "3a98d62c-3258-4bb3-84dd-378ee4a4c7b1"}]
 [:xtdb.api/put
  {:profile/username "jacob",
   :profile/title "hello there",
   :profile/user #uuid "e86e5e14-0001-46eb-9d11-134162ce930f",
   :xt/id #uuid "3a98d62c-3258-4bb3-84dd-378ee4a4c7b1"}]
 [:xtdb.api/fn :biff/ensure-unique #:profile{:username "jacob"}])

OK, looks like it's probably something to do with the transaction function. I ran similar transactions several times without it and the queries worked. I ran two transactions in between with the tx fn back in, and it broke queries both times. Here's the result of (xt/entity db :biff/ensure-unique):

#:xt{:fn (fn [ctx kvs]
           (when (< 1
                    (count
                     (xtdb.api/q
                      (xtdb.api/db ctx)
                      {:find '[doc],
                       :limit 2,
                       :where (into [] (map (fn [[k v]] ['doc k v])) kvs)})))
             false)),
     :id :biff/ensure-unique}

(For now I'll just disable biff/submit-tx's call to with-tx, which just means if there's contention then biff might try to keep re-rerunning the tx multiple times instead of aborting immediately--not a huge deal. So this isn't urgent. https://github.com/jacobobryant/biff/commit/ba6fb05c0539f18d315592dc5d7cb8b3de2eabe4)

I haven't checked the whole thing at all yet, but first thing I started thinking about is that https://github.com/jacobobryant/biff/blob/ba6fb05c0539f18d315592dc5d7cb8b3de2eabe4/src/com/biffweb/impl/xtdb.clj#L385 await-tx does not necessarily return the transaction you gave it. It makes sure the given transaction is done, and then returns the latest which might already be something else

Of course with-tx shouldn't change the db, but is it possible that it changes the timing?

And it doesn't explain the inconsistent queries you are running against the same db

oh, I wasn't aware of that--thanks for the tip! but yeah, I don't think it's related to the unexpected behavior here.

I guess with-tx must have some unintended mutation in its handling of transaction functions

Hey @U7YNGKDHA thanks for reporting this - I'll be investigating it tomorrow and will respond in due course once I have a repro

Okay it's definitely an issue https://github.com/xtdb/xtdb/issues/1851 - this will get attention soon

Perfect, thanks for looking into this!

Hey @U7YNGKDHA 👋 , just a heads up that the cause has been identified and a fix is being prepared. It is a rocks-specific caching issue. A fix should be available in the next RC (which I'm hoping to provide this week, all being well!) - thanks again for the report! 🙏

thank you again!