datascript.db/entid

Maybe, but it’s also possible you’d have to fix a few things. But why? Clojure’s sorted set is in every way inferior to persistent-sorted-set

Because it might be able to handle datomic generated db ids. Our application doesn't encapsulate the syncing process, so the db ids are hard to translate to shorter ones that the persisted sorted set can handle.

Trading perf for correctness would be a worthwhile trade off for us until i can setup something better or even just a way to fix problems faster we might discover in prod.

This is datascript limitation, not pss

You can easily lift it from datascript, it's completely artificial check

it's basically just used to distinguish entity ids from transaction ids

Thanks, this is immensely valuable to me. @U050UBKAA I have some simple questions that i should probably ask to make sure i understand completely. Are "transaction ids" temporary ids? I thought those were distinguished from entity ids by being negative.

When you say "lift" it, where "it" is the entid check to see if the number is less then some number, what do you mean? I could imagine forking datascript and changing the entid function to not have that max check, but maybe there is a better way? I can think of other ways: try catch block, you can probably just redefine a symobl, though i have never done that.

I thought, from reading this https://github.com/tonsky/datascript/issues/292, the entity ids in cljs had to be less then some max number, because of a bit wise operation that existed in the persisted sorted set library. But i suppose that's what the user claj suggested, but you never directly confirmed.

To be clear, this is an example of a db/id from datomic which throws if i try to transact it into datascript:

(def conn (datascript.core/create-conn {}))
(datascript.core/transact! conn [{:entity/name "foo" :db/id 285873023227265}])

I removed the (> eid emax) check and in cljs it doesn't throw any more (obviously!) but it also doesn't transact, so i'm confused because i thought you were implying it might be ok to remove that comparison check. 😕

> entity ids in cljs had to be less then some max number, because of a bit wise operation that existed in the persisted sorted set library Oh, JavaScript! Yes that might be the case

> JavaScript stores numbers as 64 bits floating point numbers, but all bitwise operations are performed on 32 bits binary numbers. Before a bitwise operation is performed, JavaScript converts numbers to 32 bits signed integers. After the bitwise operation is performed, the result is converted back to 64 bits JavaScript numbers.

I don’t think we use bit operations on keys though

But Datomic ids certainly exceed 32 bits that’s for sure

Seems to work in JS

Try increasing these numbers

(def ^:const tx0   0x20000000)
(def ^:const emax  0x7FFFFFFF)
(def ^:const txmax 0x7FFFFFFF)

To the values that make sense for Datomic

emax/txmax probably something like Number.MAX_SAFE_INTEGER

tx0 is the boundary between entity ids and transaction ids. Check with Datomic where it draws the line and use the same

If I remember correctly, I limit those ids to 32-bit ints because they were (rumored) to be represented more optimally in JS engines. From correctness point, you can go as far as Number.MAX_SAFE_INTEGER (which I believe is something like 2^52-1), which should cover Datomic ids I hope

Why is there a different max for a transaction id (which i believe is either an entity id or a temp id) and entity ids? To get a sense of the difference, what would happen if tx0 and emax were set to the same value?

technically emax should be tx0 - 1. Not sure why it’s set here to the same as txmax

maybe entity ids were overflowing tx0 for somebody and index lookups stopped working

My previous question should have been why do we need tx0? To help me clarify, the id space seems to look like:

0 ---------------------------------------- emax
0        tx0----------------------------- txmax
 ^^^^^^^^^ ~ why have a range that tx cannot be but the eid can be? I assume this is because sometimes clients want to explicitly set there eids in a transaction, and that range (0-tx0) is what the library uses when they don't?

> technically emax should be tx0 - 1. Not sure why it’s set here to the same as txmax By chance did you mean to say txmax - 1?

0....eids....emax....txs...txmax

that was the idea

In case anyone ever reads this thread, i want to point out that "transaction id" is the tx in a datoms [e a v tx op] and is necessary above and beyond an e/eid because we keep the history of an eid. see https://docs.datomic.com/cloud/whatis/data-model.html#example-entity for details. Some of my questions above are confusing because i misunderstood what the term meant.

i'm not sure in our case we need to increase the transaction id max, because were not getting more transactions and.

ty ty niki. This right now it seems like i have everything i need.

This topic of using datomic ids in datascript instead of making an artificial mapping between them, as I do now, is very interesting to me too. So @U0DJ4T5U1 at the end of the day you just increased the txmax, emax, tx0 and now you can use the same ids? If possible can you also tell me to which values?

Our fe browser app can use the emax to the javascript max safe int.

@U0GE6JTKK

@U0DJ4T5U1 thank you. I'll try that too 🙂