Fork me on GitHub
#datascript
<
2022-08-15
>
Drew Verlee02:08:16

in updating to version 1.0.0 i'm running into these console messages Highest supported entity id is 2147483647, got 316659348807653 I assume that corresponds to this Throw if transacted entity id is out of range https://github.com/tonsky/datascript/issues/292 so as i understand it. The db/id were trying to use in datascript e.g :db/id 316659348807653 , is too big and so we need to keep some mapping between the datomic ids and the datascript ones. I'll have to think about that... i'm sure people have done this before. Has anyone written there ideas down? i'm sure re-posh has some logic to handle this for instance.

Drew Verlee02:08:06

or like https://github.com/metasoarous/datsync > • Every entity gets a :datsync.remote.db/id mapping to the Datomic id

Drew Verlee02:08:44

this datasync function might be useful (dat.sync.client/datomic-tx conn tx) if it wasn't depreciated...

seepel02:08:53

Hi there, I'm doing something similar, so far I've had success saving the server’s id in the diatomic client. I basically have a pair of functions that transmit datoms (or should the plural be data? :thinking_face:) between server and client that manage sorting out the translation for the ids.

👀 1
seepel03:08:43

It's probably similar to datasync. Client side I have a unique attribute called :server/id. When the client sends data to the server it replaces the datascript id with the value of that attribute, otherwise it sends the diatomic id as a string. Then the server sends back the tempids map from the transaction result, and the client uses that to set the :server/id attribute. When the server sends data to the client it renames the :db/id field to :server/id

Drew Verlee14:08:37

@U03NXD9TGBD what do you mean "otherwise it sends the diatomic id as a string" Shouldn't it be find to replace the shorter client db/id's with the :server/id as an integer as long as you don't do an operation on it which only works with the shorter version (see datascript issue for details that i don't fully understand myself).

Drew Verlee14:08:34

Though, wouldn't a better solution be to just have a uuid attached to each datom and sync on that? I think datomic's build in db/id's aren't meant to be universal unique, more for indexing purposes.

Drew Verlee16:08:28

Here is my plan. In order to sync datoms they will use the db.unique/value db/uuid. The only additional cleanup is that ill have to have each peer ignore the db/id attribute, possible just by stripping it or renaming it before being sent.

Drew Verlee16:08:00

probably removing it is better, i feel like renaming it would just lead to confusion....

Drew Verlee17:08:22

What does this comment mean from tonsky mean.. > It seems that long ids are no longer supported on JVM since 0.18.0. I also added a validation that throws on both JVM and JS if value is out of range. why does the jvm care about ids? ids of what?

seepel20:08:35

Sorry, I apparently left out some context. I let datascript manage the ids as usual, and I attach the datomic id under :server/id . So when I want to send data to the server there are two cases. Where the client entity already has a :server/id from datomic, and the other when a client entity has been created on the client, in which case there won't be a :server/id. In both cases the datascript db/id is the normal datascript id. Here is my client->server transformation, it's purpose is to transform a datascript :db/id into the matching datomic :db/id.

(defn ref->datomic [db ref]
  (or (-> (and ref (d/pull db '[:server/id] ref))
          :server/id)
      (str ref)))
If the :server/id exists then that should be the id that gets sent to the server. If the :server/id doesn't exist, then I send the datomic id as a string. Datomic interprets string ids as tempids and in the transaction result returns a mapping of tempid to permanent id. My sync endpoint on the server returns this map to the client. And then the client updates the :server/id for each newly created entity. Here is the code for that.
(defn save-sever-ids [[client-id server-id]]
  [:db/add (edn/read-string client-id) :server/id server-id])

(defn sync-pending-txs! []
  (go (let [edn-params (pending-txs->datomic)
            response (<! (http/post "/sync"
                                    {:with-credentials? true
                                     :edn-params edn-params}))]
        (->> (get-in response [:body :tempids]) ; Get the newly created entity ids
             (map save-sever-ids) ; Convert them into transaction commands
             (into [])
             (d/transact! @sync-conn))
        (reset! pending-txs []))))
Hopefully that's more clear?

seepel20:08:34

> In order to sync datoms they will use the db.unique/value db/uuid. The only additional cleanup is that ill have to have each peer ignore the db/id attribute, possible just by stripping it or renaming it before being sent. This is a good plan, unfortunately because of other reasons specific to my app, There are some entities that I can't attach a UUID to, so I need to go through this translation. For details, I'm basically making a markdown editor so I'm tracking dom nodes. I attach a unique UUID to each html tag via attributes (ie <p dom-id="<UUID>">). However text nodes don't have attributes, so I can't treat them the same and I have to go fishing into the db to figure out how to sync them.

Drew Verlee20:08:44

thanks a lot @U03NXD9TGBD. I think we might end up doing the server/id translation for a bit to until we have time to think through using the uuid. It's nice because it just adds some computational overhead and doesn't effect any existing entities in the db.

👍 1
seepel20:08:33

So far it's worked out for me, but there could be some gotchyas that I haven't uncovered… If you hit any edge cases I'd be interested to hear about them!

Drew Verlee20:08:59

It would seem like the client or the server could hold the mapping, why prefer the client?

seepel20:08:18

I expect there to be multiple clients all with different ids figured it's easier to track one server id rather than multiple client ids.