Clojurians Log v2

I'd be interested in seeing this - have implemented datascript with an indexed db transaction log synced to postgres recently, but have been thinking datomic as the sync store might be nicer

Thank you kindly @danbunea. Scrambling to finish up a few things at work before taking my family on holiday Saturday, but have bookmarked this to poke at when I get back

If it's convenient I'd certainly be interested. I'll chuck what I've done up in a repo next week (busy fighting with some other tooling at the minute) if it's of any interest. Reasonably happy with the sync piece -- implemented using replicant and clj-statecharts, but a little bit concerned the way I'm syncing transactions to postgres might be naive and won't scale. Also, I want an excuse to play with datomic :)

could you please share your decision on making datascript not temporal?

No list of features, no

What do you mean not temporal?

keeping the history of entities and attributes (immutable)

Oh. Yeah, it wasn’t really needed for UI problems

I wonder how I could actually give you the code, but let me try. An endpoint on the server returns the server schema and the datoms, and they're like:

(def server-schema [{:db/id 89 :db/ident :user/id :db/valueType :db.type/string :db/cardinality :db.cardinality/one :db/unique {:db/id 38, :db/ident :db.unique/identity}} {:db/id 82 :db/ident :user/name :db/valueType :db.type/string :db/cardinality :db.cardinality/one} {:db/id 95 :db/ident :inbox/id :db/valueType :db.type/long :db/cardinality :db.cardinality/one :db/unique {:db/id 38, :db/ident :db.unique/identity}} {:db/id 120 :db/ident :inbox/owner-id :db/valueType :db.type/ref :db/cardinality :db.cardinality/one} {:db/id 105 :db/ident :app/id :db/valueType :db.type/keyword :db/cardinality :db.cardinality/one :db/unique {:db/id 38, :db/ident :db.unique/identity}} {:db/id 99 :db/ident :app/name :db/valueType :db.type/string :db/cardinality :db.cardinality/one} {:db/id 107 :db/ident :bunea.projects.todo/chat-id :db/valueType :db.type/ref :db/cardinality :db.cardinality/one :field/position 2 :field/is-parent-ref? true :field/ref-entity :chat} {:db/id 119 :db/ident :bunea.projects.todo/text :db/valueType :db.type/string :db/cardinality :db.cardinality/one :field/position 4 :field/label "What"} ]) (def server-datoms [[101155069755522 :user/id :user/user] [101155069755522 :user/name "User"] [101155069755541 :inbox/id 1] [101155069755541 :inbox/owner-id 101155069755522] [101155069755588 :app/id :taia] [101155069755588 :app/name "Taia"]])

then on the client: Datascript part (defn update-my-db [datomic-response] (-> datomic-response datomic->datascript transact-and-return-schema-and-chats! )) where:

(defn- update-if [o prop func] (if (get o prop) (update o prop func) o)) //from datomic schema to ds (defn datascript-schema [datomic-schema] (->> datomic-schema (filter #(nil? (some #{(:db/ident %)} [:fressian/tag :field/type :field/position :field/label :field/retracted? :field/ref-entity :field/is-parent-ref? :relations]))) (map (fn [field] [(:db/ident field) (-> (select-keys field [:db/valueType :db/cardinality :db/unique ]) (update-if :db/unique (fn [v] (if (map? v) (:db/ident v) v))) ((fn [attrs] (if (= :db.type/ref (:db/valueType attrs)) attrs ;;else (dissoc attrs :db/valueType)))) ((fn [attrs] (if (= :db.cardinality/many (:db/cardinality attrs)) attrs ;;else (dissoc attrs :db/cardinality)))))])) (into {}))) Transform the datomic datoms to ds datoms using ds schema (defn datascript-datoms [client-schema server-datoms] (->> server-datoms (map (fn [[datomic-e datomic-a datomic-v]] (if (= :db.type/ref (get-in client-schema [datomic-a :db/valueType])) [(str datomic-e) datomic-a (str datomic-v)] [(str datomic-e) datomic-a datomic-v]))))) (defn datomic->datascript [response] (let [schema (datascript-schema (:schema response)) datoms (->> (:datoms response) (datascript-datoms schema) (mapv (fn [[e a v]] [:db/add e a v])))] (-> response (assoc :schema schema) (assoc :datoms datoms))))

(defn transact-and-return-schema-and-chats! [ion-data] (d/transact! (get-connection) (:datoms (:datoms ion-data)))

a schema transformation test: (def custom-field-idents [{:db/id 39 :db/ident :fressian/tag :db/valueType :db.type/keyword :db/cardinality :db.cardinality/one :db/doc "Keyword-valued attribute of a value type that specifies the underlying fressian type used for serialization."} {:db/id 73 :db/ident :field/type :db/valueType :db.type/keyword :db/cardinality :db.cardinality/one} {:db/id 74 :db/ident :field/position :db/valueType :db.type/long :db/cardinality :db.cardinality/one} {:db/id 75 :db/ident :field/label :db/valueType :db.type/string :db/cardinality :db.cardinality/one} {:db/id 76 :db/ident :field/is-parent-ref? :db/valueType :db.type/boolean :db/cardinality :db.cardinality/one} {:db/id 77 :db/ident :field/ref-entity :db/valueType :db.type/keyword :db/cardinality :db.cardinality/one}]) (def fields-schema [{:db/id 89 :db/ident :user/id :db/valueType :db.type/string :db/cardinality :db.cardinality/one :db/unique {:db/id 38, :db/ident :db.unique/identity}} {:db/id 82 :db/ident :user/name :db/valueType :db.type/string :db/cardinality :db.cardinality/one} {:db/id 95 :db/ident :inbox/id :db/valueType :db.type/long :db/cardinality :db.cardinality/one :db/unique {:db/id 38, :db/ident :db.unique/identity}} {:db/id 120 :db/ident :inbox/owner-id :db/valueType :db.type/ref :db/cardinality :db.cardinality/one} {:db/id 101 :db/ident :bunea.projects.todo/id :db/valueType :db.type/long :db/cardinality :db.cardinality/one :db/unique {:db/id 38, :db/ident :db.unique/identity} :field/position 1} {:db/id 119 :db/ident :bunea.projects.todo/text :db/valueType :db.type/string :db/cardinality :db.cardinality/one :field/position 4 :field/label "What"} {:db/id 107 :db/ident :bunea.projects.todo/chat-id :db/valueType :db.type/ref :db/cardinality :db.cardinality/one :field/position 2 :field/is-parent-ref? true :field/ref-entity :chat} {:db/id 112 :db/ident :bunea.projects.todo/done? :db/valueType :db.type/boolean :db/cardinality :db.cardinality/one :field/type :boolean :field/position 3 :field/label "Done?"} {:db/id 81 :db/ident :bunea.projects.todo/date :db/valueType :db.type/instant :db/cardinality :db.cardinality/one :field/type :date :field/position 5 :field/label "When"} {:db/id 81 :db/ident :bunea.projects.todo/signatures :db/valueType :db.type/ref :db/cardinality :db.cardinality/many :field/type :signatures :field/position 6 :field/label "Signatures"}]) (def server-schema (concat custom-field-idents fields-schema)) (def client-schema {:user/id {;; :db/valueType :db.type/string ;; :db/cardinality :db.cardinality/one :db/unique :db.unique/identity} :user/name {;; :db/valueType :db.type/string ;; :db/cardinality :db.cardinality/one } :inbox/id {;; :db/valueType :db.type/long ;; :db/cardinality :db.cardinality/one :db/unique :db.unique/identity} :inbox/owner-id {:db/valueType :db.type/ref ;; :db/cardinality :db.cardinality/one } :bunea.projects.todo/id {;; :db/valueType :db.type/long ;; :db/cardinality :db.cardinality/one :db/unique :db.unique/identity ;; :relations {} :field/position 1} :bunea.projects.todo/chat-id {:db/valueType :db.type/ref ;; :db/cardinality :db.cardinality/one :field/ref-entity :chat :field/is-parent-ref? true :field/position 2} :bunea.projects.todo/done? {;; :db/valueType :db.type/boolean ;; :db/cardinality :db.cardinality/one :field/type :boolean :field/position 3 :field/label "Done?"} :bunea.projects.todo/text {;; :db/valueType :db.type/string ;; :db/cardinality :db.cardinality/one :field/position 4 :field/label "What"} :bunea.projects.todo/date {;; :db/valueType :db.type/instant ;; :db/cardinality :db.cardinality/one :field/type :date :field/position 5 :field/label "When"} :bunea.projects.todo/signatures {:db/valueType :db.type/ref :db/cardinality :db.cardinality/many :field/type :signatures :field/position 6 :field/label "Signatures"}}) (deftest mapping-datomic-to-datascript-schema-test (testing "schema mapper should -transform datomic datoms for schema to datascript schema" ;; (cljs.pprint/pprint (diff ;; (m/datascript-schema server-schema) ;; client-schema)) (is (= (m/datascript-schema server-schema) client-schema))))

and the datoms: (def client-schema {:user/id {:db/unique :db.unique/identity} :user/name {} :inbox/id {:db/unique :db.unique/identity} :inbox/owner-id {:db/valueType :db.type/ref} :app/id {:db/unique :db.unique/identity} :app/name {} :bunea.projects.todo/chat-id {:db/valueType :db.type/ref :field/position 2 :field/is-parent-ref? true :field/ref-entity :chat} :bunea.projects.todo/text {:field/position 4 :field/label "What"} }) (def server-schema [{:db/id 89 :db/ident :user/id :db/valueType :db.type/string :db/cardinality :db.cardinality/one :db/unique {:db/id 38, :db/ident :db.unique/identity}} {:db/id 82 :db/ident :user/name :db/valueType :db.type/string :db/cardinality :db.cardinality/one} {:db/id 95 :db/ident :inbox/id :db/valueType :db.type/long :db/cardinality :db.cardinality/one :db/unique {:db/id 38, :db/ident :db.unique/identity}} {:db/id 120 :db/ident :inbox/owner-id :db/valueType :db.type/ref :db/cardinality :db.cardinality/one} {:db/id 105 :db/ident :app/id :db/valueType :db.type/keyword :db/cardinality :db.cardinality/one :db/unique {:db/id 38, :db/ident :db.unique/identity}} {:db/id 99 :db/ident :app/name :db/valueType :db.type/string :db/cardinality :db.cardinality/one} {:db/id 107 :db/ident :bunea.projects.todo/chat-id :db/valueType :db.type/ref :db/cardinality :db.cardinality/one :field/position 2 :field/is-parent-ref? true :field/ref-entity :chat} {:db/id 119 :db/ident :bunea.projects.todo/text :db/valueType :db.type/string :db/cardinality :db.cardinality/one :field/position 4 :field/label "What"} ]) (def server-datoms [[101155069755522 :user/id :user/user] [101155069755522 :user/name "User"] [101155069755541 :inbox/id 1] [101155069755541 :inbox/owner-id 101155069755522] [101155069755588 :app/id :taia] [101155069755588 :app/name "Taia"]]) (def client-datoms [["101155069755522" :user/id :user/user] ["101155069755522" :user/name "User"] ["101155069755541" :inbox/id 1] ["101155069755541" :inbox/owner-id "101155069755522"] ["101155069755588" :app/id :taia] ["101155069755588" :app/name "Taia"] ]) (testing "mapper should -transform datomic datoms to datascript datoms, mapping all ids as strings, also all refs as strings" (is (= (m/datascript-datoms data/client-schema data/server-datoms) data/client-datoms)))

and it seems ChatGPT explained it better: https://chatgpt.com/share/685bc9bb-8a94-8009-aa67-aa61551c664b

I can show you how I sent transactions back to the server and how I keep a map with the correspondence between the ids of the databases

@itai Datahike was aiming for this datalog dream as an open source endeavour and is the closest to that realisation implementation wise. It’s a big task to achieve something like this and no one in any area of the community, across any of the datalog tools is working on this.

i have heard about datahike recently in a datomic thread didn't have the time to dive into it yet

I did it a few years ago. I used a websocket to send transactions and I was storing a map with the corresponding data script ids to datomic ids. It worked really well, but then I moved the project to electric v2 and I only use Datomic. I used re-posh which is like re-frame but the db is a datascript database

I can share the code if you’re interested

But now I am travelling so only next week

@cormacc Did you ever get a chance to share your code? I'd be interested in seeing it -- I've done a handful of complex UIs with datascript and statecharts in Electron desktop apps where I just persisted the state as a save file. I'm looking at doing hosted web apps now, and having a list of the transactions seems extremely useful both for implementing user-facing version-control-ish features, as well as providing visibility into how people are actually using the app, being able to reproduce bugs, etc.

Hey, sorry, I've lost track of this. Let me find the code and share it

added the code here: https://chatgpt.com/share/685bc9bb-8a94-8009-aa67-aa61551c664b

(defn execute ([transaction logged-user] (execute transaction logged-user true)) ([transaction logged-user send-message?] ;; (prn execute transaction logged-user) (go (try (let [tempids (tempids) schema (schema) tx (-> {:datoms transaction} (update :datoms #(datomic-datoms schema % tempids)))] (-> (<? (transact tx logged-user)) (save-changes-and-send-message schema tempids send-message?) )) (catch :default e (println e) ))))) (defn tempids [] (-> (d/pull @(get-connection) [:datomic-ids] [:model/id :model]) :datomic-ids)) (defn schema [] (:schema @(get-connection))) (defn datomic-datoms [client-schema server-datoms tempids] (->> server-datoms (map (fn [[op e a v]] (let [datomic-e (get tempids e (str e))] ;; (println a (get-in client-schema [a :db/valueType]) " " (get tempids v (str v))) (if (= :db.type/ref (get-in client-schema [a :db/valueType])) [op datomic-e a (get tempids v (str v))] ;else [op datomic-e a v])))))) (defn transact ;; request ;; tx {:datoms [[:db/retract 96757023244447 :bunea.projects.todo/text "Limpiar coche"] ;; [:db/add 96757023244447 :bunea.projects.todo/text "Limpiar coche 🚘"] ;; [:db/add "57" :bunea.projects.todo/chat-id 96757023244433] ;; [:db/add "57" :bunea.projects.todo/text "Barbacoa"]]} ;; logged-user {:id ...} ;; ;; response: ;; {:datoms [[:db/retract 96757023244447 :bunea.projects.todo/text "Limpiar coche"] ;; [:db/add 96757023244447 :bunea.projects.todo/text "Limpiar coche 🚘"] ;; [:db/retract 96757023244447 :bunea.projects.todo/text "Limpiar coche"] ;; [:db/add 87960930222246 :bunea.projects.todo/chat-id 96757023244433] ;; [:db/add 87960930222246 :bunea.projects.todo/text "Barbacoa"]] ;; :tx {:tx-id 13194139533321 ;; :tx-time #inst "2021-10-31T08:49:35.053-00:00"}} [tx logged-user] (post "/api/v1/authed/transact" tx (get-in logged-user [:user/tokens :access_token])))

I’d love to do it, but realistically it probably won’t happen any time soon

do you have a list of features / milestones that you can share that required to get there? where would you start?

There's an ugly/early/out-of-date version extracted here about 9 months back: https://github.com/cormacc/cljserial. I'll try and make some time this week to pull across some of the relevant refinements/improvements from the app I'm actively working on. Will report back here when I do.