Clojurians Log v2

I've tracked down my bug and it is indeed due to d/sync biasing towards an old database. Specifically, when using the Cloud client or the db-local client, I see something like this:

[(-> conn datomic.client.api/db :t) (:t (datomic.client.api/sync conn 1))]
=> [12113236 12113236]

[(-> conn datomic.client.api/db :t) (:t (datomic.client.api/sync conn 1))]
=> [12113236 1]

Technically d/sync is meeting the contract of its documentation. But the behavior is unexpected and requires mitigation to get the "bias" towards the "current" db.

is "1" a real basis-t?

apparently so... because that comes from a real REPL session. Or at least it qualifies as a value for "> or =".

can we see the full db values in the ion case?

Sure.

I don't believe that the db's basis-t is actually 1

A screen shot of my remote REPL session just completed:

I'm wanting to see the thing returned from db and sync, not just the :t of it

OK.

so what can that db see?

Certainly not what the "current" db sees. And that is how I got burned.

Of course I used basis-t of 1 to illustrate the problem. Doesn't matter what the value is, the bias is towards that basis-t, not current.

that's not a "bias", it's just returning whatever you put in, right? What if you put in a larger number?

I call it a bias since the docs leave open the possibility of returning any value from the basis t argument through max t. Anywhere on that spectrum is technically meeting the specs of the doc. Where on that spectrum the actual db's t value lands is what I call the bias. Please give me another word to describe the "leaning" towards one end of the spectrum or the other!

If I put in a larger number, it returns the db at that larger number.

(but not current)

(all this only with the ion client)

Here's an example with a larger number.

The cloud client biases towards current. So does db-local. The ion client biases towards the t value argument. I'm not sure about peer.

so it doesn't know the actual current t of the db, it's giving you a handle with a required minimum t. What happens if you try to query with the db with a T that is in the future?

Yikes!

exactly, so now query with that 99999999 db

(which I assume is in the far future/not reached yet)

I'm going to do that... but my point is that the bias is unexpected and inconsistent even if technically correct. It means using sync with the cloud client is equivalent to sync followed by db with the ion client (that's how I'm working around this behavior).

Using that db yields an exception:

Execution error (ExceptionInfo) at datomic.core.anomalies/throw-if-anom (anomalies.clj:94).
Database does not yet have t={:database-id "e684b9c0-af66-450f-b29b-da7e6f080575", :t 9999999999999}

sync returns a database that includes T, without communication. If client does not know T and cannot communicate about it, what would it do besides this?

I don't think we are seeing this the same way. The cloud client (not ion client) does the expected thing: returns the latest database available without communication that also includes t. The ion client does not.

It returns the database at t. Despite having a newer one "on-hand".

I understand that you want sync to return value whose :t is "max(oldest-t-client-might-already-know-about-somehow, sync-t-param)", but that is not what it guarantees.

I totally agree. This is not a bug per the docs. But it is surprising and inconsistent (across clients).

maybe ion-client could match this (I don't know--I don't know what state it has available), but the behavior has to devolve in at least some cases to "I don't know this T, so I'll echo back what you asked for and let that serve as a precondition for future queries"

and a truly stateless client could never do anything other than echo back the T

Agreed. It would be more useful if d/sync never returned a database older than what d/db returns (on the same thread). No extra cost, but provides a more useful feature than the current behavior.

d/conn requires communication

The current ion behavior is akin to as-of.

I was not aware that d/conn required communication... that does change my perception.

A lot of your expectations sound very peer-api-like

The irony then is that the cloud client does the "expected" thing.

this should not be the same as "as-of"

:t is a basisT limit; as-of-t is a filter. they are different keys on the db

Whoops... I realized I mean d/db... you said conn.... not sure if we were talking about the same thing.

(I'm going to update my comment above)

To restate: It would be more useful if d/sync never returned a database older than what d/db returns (on the same thread). No extra cost, but provides a more useful feature than the current behavior.

IDK what thread means with a fundamentally async, wire-compatible api

(sorry for the confusion... I wrote d/conn earlier... which doesn't even exist.

perhaps modified to "older than what (d/db conn) returned the last time it was called"?

for same object conn

? Consider this code:

[(d/db conn) (d/sync conn t)]

the expressions do, but where does the work happen?

Don't care.

in process? in another thread? on another machine?

this requirement also makes stateless conns impossible to implement

However it works with db-local and the cloud client would be really really useful for the ion client. If not, I have to hack around the differences with d/db right after d/sync.

If that means a stateful connection... that is frustrating.

It's not a huge deal, I can just write my own wrapper around d/sync to get consistent behavior.

A note in the docs would have saved me hours of debugging. Just saying....

It would really be better to use a :db-after

That's clearly the ultimate and right solution. And I do that in many situations. But in this particular situation it is impractical (still possible, just really impractical).

Much cheaper to wrap d/sync.

I have confirmed that d/sync returns the latest database in this simple case:

(let [{conn :datomic/connection :as system} @user/sysref]
  [(-> conn d/db :t) (:t (d/sync conn 11992651))])
[12090046 12090046]

Per https://docs.datomic.com/transactions/client-synchronization.html#sync > sync takes a basis point t, and it returns a database value that includes point t.

Right, which leaves open the question of whether the bias is towards the latest t or the t parameter. IOW, A. ...that includes t and is as recent as possible or B. ...that includes t and is as close to t as possible I think the B interpretation is weird, but my apps behavior suggests that might be happening. Could also be a bug in my app...

the bias is toward no communication, so it's going to be whatever the node has on-hand unless that's not good enough

IOW sync-with-t is designed to be as cheap as possible while still satisfying the docstring. Any biases are impl details

Is it true that if a node has db at tA then it "has" all the dbs at tB where tB < tA?

Depends on what you mean by "db"

semantically, the basis t of a db contains all information before it, because that's datomic's time model

Right. So, with no communication, the sync call can return db at tA or tB with no communication.

(assuming it has on hand the later of those two)

> sync takes a basis point t, and it returns a database value that includes point t.

to "include it" means >= t

I got that. My lack of understanding is around the bias.

I have some evidence that the bias is towards t, not towards the latest on-hand.

It's not a "bias", you are just seeing relativistic effects of information propagation

nodes aren't holding on to older db values just to satisfy sync calls; they always have only the latest they know about

I use the term bias loosely... the docs do not provide any information one way or the other and, strictly speaking, the behavior I am seeing meets the contract of the docs. BUT, I would LIKE to know if it is possible that asking for a sync at time t even when the node holds a db well after t might sometimes return something other than the latest.

today it does not; no guarantees about tomorrow, because it still satisfies the contract either way

That is great to know and implies my problem is elsewhere.

The purpose of sync is to make sure the "light cone" of some other process's causality has reached you.

That was my understanding as well and that is how I'm using it.

Some context: ten threads (from ten concurrent Lambdas handling an SQS queue) all do a sync on the same t value and then CAS on a database value to refresh a single OAuth token. One thread succeeds, the other nine fail and are requeued. Some short time later, the remaining nine do the exact same thing. If the sync returns the latest db, then they see the work done by the thread that refreshed the token and just use the fresh token. But what I'm seeing is that the nine threads are trying to refresh the token again -which would be the behavior if the "bias" was towards t instead of the latest (with the already-refreshed token).

It's probable that I have bug or some other unexpected competition between threads, ion nodes and the db that is causing the issue so the hunt goes on.

this seems like a perfectly reasonable thing that could happen?

Bias towards t instead of latest?

no, you sync on a t, but something happened since then

that invalidates what you want to do

I don't think I'm being clear.

the 9 failing threads are retrying with the same original t parameter? (which doesn't include the refreshed token)

In the first round, I expect all ten threads to read (after sync) that the token needs to be refreshed. I then expect one thread to succeed and nine to fail on the CAS (gating the refresh to be serial). The nine that failed the CAS are requeued ... Some time later the nine messages are tried again -probably in a matter of ~10 seconds and probably concurrently. I expect all nine to sync, get the latest db and find a fresh token with no need to refresh. Instead, I'm seeing the nine compete for the CAS to refresh again.

What is the parameter to sync in each case?

Yes, the nine failing threads are retrying with the same t. But using sync, not as-of!

so, whatever node answered the sync has seen T, but hasn't seen the T with the refreshed token yet ("latest")

possibly... but my query group (where this is running) only has one instance.

So concurrency is thread-wise in one node -not across nodes.

so that is suspicious, but regardless in a robust concurrent system you use the transaction failing to refresh as the way to learn latest t and either sync on it or just use the token it returns.

transaction "aborts successfully"--the work you wanted done was already done, here's what it was

the transactor is the only place where there are no relativistic effects possible

Yes, I could do that. And maybe I need to. But it is a PITA because I need to enqueue a new message instead of simply failing to process the existing message.

the existing message can't use a different token that it learned via refresh attempt?

The message is to use the token to retrieve some data from an API. First, ensure there is a fresh token and then use it to get the data.

Ensuring the token is fresh may require a refresh, and that must be gated to prevent token loss.

So returning the exact same message to the queue (with a t value in the message) is semantically reasonable and very easy. Updating the t value in the message would change the logic quite a bit. FWIW, the t value is in the message to avoid some relativistic problems I have encountered where node A writes to the db and enqueues an SQS message that is handled by node B before the db segments from A have propogated to B.;

IDK what to tell you. If token refresh needs to be atomic, you gotta do that work in a transactor. You can't just "sync harder"

CAS protects against concurrent token refresh nicely. No need for a tx-fn!

(each token has a fingerprint and I CAS on that)

[:db/cas attr fingerprint fingerprint]

the pain you are describing is that if CAS fails, you know you are out of date, but you aren't resetting your sync moment

RIght... because the sync moment is a "not before" concept and the token is a "at least" concept. They seem to be at odds ... if the "bias" is not towards the latest db on-hand.

If the bias is towards latest, this should not be a problem. On retry, the nine threads see the fresh token and just do the work.

again, "latest db on hand" --- of whom? You want the latest db the transactor saw (because your tx failed), not what some node saw

even then, you could still fail, because something happened in between issuing a transaction and the transactor attempting to commit it

I want to see the latest db that the node saw -the one from the one thread that succeeded in updating the token!

and how will you know what that db is, if not told by the transactor?

If ion node A succeeds in a transaction (from a thread updating the token), I would hope that other threads doing a sync would get nothing later than that db.

I was under the impression that if node A succeeds in a transaction, it's basis t (from d/db) will not be earlier than the db-after that transaction.

... for all threads.

even in pro (where that assumption is safer, because I know there's a single in-process atom holding the db and I know if I saw the tx-data that atom has already been updated), I would use the :db-after, not d/db again on the connection. In cloud I have no idea: maybe you hit a different node, maybe each ion has its own db state 🤷

One node only in this case.

I'm saying it's not correct to write code that makes those assumptions in a distributed system, regardless of circumstances where one reasons it might/should work anyway.

you must always be prepared for a transactor to tell you the world has moved on

normally the window for this is quite small (milliseconds), but always exists. the anomaly here is that you say the window is multiple seconds

that's a performance problem though, not a correctness one

If that is absolutely true, then you could never use (d/db conn). You would need to store the t value and serialize your work externally. Without some assurance that (d/db conn) returns a db after the last transaction on the same node and thread, (d/db conn) could always return the exact same db... forever.

> (d/db conn) could always return the exact same db... forever.

transactions must always be prepared for that

I am not talking so much about transacting but rather read-after-transact.

read-after-transact is what :db-after is for 🙂

I get that -absolutely. But the statistics in my case seem to be particularly unlikely.

I'm prepared for some "failures" where the SQS messages are processed due to db propogation delays. But in my case, the frequency is disturbing.