Clojurians Log v2

What do you mean “times out”? What is the specific behavior you are seeing? Sync is designed to address peer notification lag. Not every peer will see the same thing simultaneously on a live distributed system Allowing peers to be aware of the work other peers have done sometimes before they would otherwise be notified. It is entirely possible for a peer to not yet know about a transaction that completed via another peer. However Datomics use of immutability ensures that you are able to get a consistent and valid view. Are you sure you are passing the correct t to sync? If you pass a far “future” t, sync will wait for that t. But I'll know more when you describe what timing out means.

Does it actually throw? What error if so?

Timing out means that I deref the returned future with a timeout:

(or (deref (d/sync conn t) 5000 nil)
    (throw ,,,))

(d/basis-t (:db-after tx-report))

@fb I understand the observation and I have a follow up question: are there no subsequent transactions that arrive in this 5 second window?

Also I just want to confirm you're using the peer api? And what version?

assuming peer for all, but want to confirm.

@ghadi i think no. I was curious if there's a race between an atom swap and add-watch?

Peer API, one of the latest versions, need to check (on my phone right now)

we aren't using atoms for this situation, but in general yes add-watch is uncoordinated with swappers. You don't know what the first observed swap will be

In the same process with the same connection with a T you observed from the tx-report-queue, any sync-t on that T should resolve immediately: the db returned from d/db is updated before the report queue gets it

You say: > interestingly: the last time it did the Redis consumer was the same machine that originally submitted the transaction via (d/transact). Is it possible that this wasn't actually the same process?

Note I say "same process" not merely "same machine"

I'm positive it was the same process that did transact and later called sync. There is a Redis stream in between and it's possible another machine read the queue and added the t value to the stream

com.datomic/peer {:mvn/version "1.0.7705"}

Is it possible that this happens: • Instance A: d/transact • Instance B: sees new t in tx-report-queue • Instance B: adds t to Redis stream • Instance A: reads t fro Redis stream • Instance A: calls d/sync, but has not received the new t yet Paired with some race condition that d/sync misses the t value and would only complete for t+1 which does not arrive before the timeout?

Yes, this clears it up: it's totally possible for another process to receive notification of a transaction before the transacting processes does.

We did find a race condition where simultaneously receiving a T and waiting on it could sequence things in such a way that your sync promise would not be resolved until the next transaction arrives. We'll get a fix out for this. We believe this explains the "deref never finishes" behavior if the transaction queue was also relatively quiet at the time (txs arriving more than 5 seconds apart in your case).

Cool! We meanwhile try d/db one more time after the timeout and check its basis-t. We did suspect some race here. Of course it's hard to say if that resolves it

Here’s our workaround for the meantime:

(defn sync-db
  "Returns a `db` value with a basis-t of at least `t`.

   Throws a `sync-timeout-exception` if `conn` hasn't caught up to `t`
   before `timeout-ms` elapses."
  [conn t timeout-ms]
  (let [db (d/db conn)
        seen-t (d/basis-t db)]
    (if (>= seen-t t)
      db ;; already up-to-date
      (let [future-db (d/sync conn t)]
        ;; There's a bug in the Datomic peer library:  `d/sync` can lose a race
        ;; against the connection processing the transaction it waits for.
        ;; When this happens, `future-db` does not resolve until a later transaction
        ;; arrives.
        ;; As a workaround, we suspend our wait briefly after our call to `d/sync`
        ;; and check if the `d/db` value has caught up.  If not, we resume the wait.
        (or (deref future-db 20 nil)
            (let [db (d/db conn)
                  seen-t (d/basis-t db)]
              (if (>= seen-t t)
                db
                (or (deref future-db (max 1 (- timeout-ms 20)) nil)
                    ;; Check one last time to report the latest `t` available and
                    ;; to handle edge cases like GC pauses or thread starvation.
                    (let [db (d/db conn)
                          seen-t (d/basis-t db)]
                      (if (>= seen-t t)
                        db
                        (throw (sync-timeout-exception timeout-ms t seen-t))))))))))))

The first/outer guard isn't necessary—d/sync itself is doing the same thing

Maybe I went overboard there. I suspected that there is some overhead in creating and then waiting on an already-completed future, compared to reading the current db value. Premature optimisation?

It's a promise not a future

(class (d/sync conn 1)) ;; => datomic.promise$settable_future$reify__9119

It implements ConpleteableFuture, but it's a promise

There's no extra threads or tasks for syncing

“Db already synced” immediately resolves the promise in the calling thread before returning

OK, putting the name discussion aside, you want to emphasise that just deref-ing the already-delivered promise does not involve blocking on threads so the overhead should be neglectable?

Yes