datomic 2024-03-29 | Slack Archive

jjttjj15:03:56

I believe I've read that it's a best practice to batch transactions (though can't immediately find refence to it and maybe it's "unofficial" advice), eg calling (partition-all 1000 ...) and calling d/transact on those batches. Is the same true of using d/with, and if so, how do you accumulate a result? is it possible to get a new with-db from the result of with? Or since it's just an in memory thing it's fine to just transact huge amounts of datoms? let's say STATS here is a sequence of maps that each have 6 double attributes.

;; this wont work, the db here will just be the result of transacting the last batch only
  (let [conn (d/with-db conn1)
        db (:db-after
            (reduce
              (fn [_ batch]
                (d/with conn {:tx-data batch}))
              conn
              (partition-all 1000 STATS)))]

    (d/q
      '[:find (count ?e)
        :where [?e ::stat]]
      db))

favila16:03:32

> is it possible to get a new with-db from the result of with? it returns the same as d/transact, so it includes :db-after , which will be a with-db (unless this is broken in cloud--it definitely works on on-prem)

favila16:03:00

Anything you can (d/transact conn …) you can (d/with db)

favila16:03:38

just use the :db-after returned by the previous d/with

jjttjj16:03:06

Hmmm thought I tried that but I'll check again (yeah this is for cloud by the way).

jjttjj16:03:03

isn't :db-after a db and not a conn? so not something you can transact against?

favila16:03:38

right, but you’re not transacting, correct? You are d/with-ing

favila16:03:10

fwiw your code above should be

jjttjj16:03:26

(let [conn (d/with-db conn1)
        db   (reduce
               (fn [conn batch]
                 (:db-after (d/with conn {:tx-data batch})))
               conn
               (partition-all 100 STATS))]

    (d/q
      '[:find (count ?e)
        :where [?e ::stat]]
      db))

Ah you're right this does work, thanks

favila16:03:31

(let [with-db (d/with-db conn1)
        db (:db-after
            (reduce
              (fn [with-db batch]
                (:db-after (d/with with-db {:tx-data batch})))
              with-db
              (partition-all 1000 STATS)))]

    (d/q
      '[:find (count ?e)
        :where [?e ::stat]]
      db))

favila16:03:35

jinx

😄 1

favila16:03:37

you have to manually accumulate the subsequent dbs instead of relying on d/transact side-effects

favila16:03:11

you can simulate what d/transact and a connection is doing by keeping an atom with the latest with-db in it and calling it “conn”.

favila16:03:36

On on-prem there’s even a library for this: https://github.com/vvvvalvalval/datomock

👀 1

2024-03-29

Channels