Clojurians Log v2

Some nice links https://learn-some.com/ https://tonsky.me/blog/unofficial-guide-to-datomic-internals/

> Take care that they aren't too long how long is “long”? they won't be more than 2 MB or so because at that point they'll hit ARG_MAX anyway

I think that in datomic cloud the limit is 4kb? Or 4mb? In datomic on-prem there is no limit - but you will have performance issues//it will require to tweak configurations to make it work.

the docs say 4k. ok, this is manageable, i can store larger command lines as response files and that’s probably a better design anyway.

the max overall size of the DB is around 20 GB (https://groups.google.com/g/android-building/c/CGZRqcRFtkk) and if that happens i'll have succeeded beyond my wildest dreams lol

> When in-memory index gets too big (memory-index-threshold, e.g. 32 Mb), transactor starts current index re-built. It is done by merging latest current index with in-memory index how does persistence work for the in-memory index? is it crash resistant?

> • Strings are limited to 4096 characters in Cloud and Datomic Local. Datomic does not enforce this limit in Pro, but users are strongly encouraged to enforce it. https://docs.datomic.com/schema/schema-reference.html#notes-on-value-types

The important thing to keep in mind is that Datomic is designed to store facts, which it stores in covering indexes chunked into large segments. Large values increase the size of those segments in storage and take up memory whenever you read something "nearby" in the index.

re: in-memory index, see https://docs.datomic.com/indexes/background-indexing.html and https://docs.datomic.com/operation/caching.html Rich also discusses the architecture at https://youtu.be/Cym4TZwTCNU?si=6TtVbIWrQT3zBtsu&t=2109.

oh interesting, the value types page mention that datomic does have tuples but they can be at most 8 values long

> how does persistence work for the in-memory index? is it crash resistant? If process dies, "in-memory" is blown away and picks up when you re-launch process (transactor) because it goes off what is durably written and can still identify novelty.

https://stackoverflow.com/questions/44645938/how-to-implement-sorted-to-many-relationships-in-datomic soultion 3's link is broken, but the accepted answer has a newer link to the same thing, i guess: https://github.com/dwhjames/datomic-linklist then other answers also mention datofu (https://github.com/vvvvalvalval/datofu), which has a few more useful facilities.

https://docs.datomic.com/reference/best.html is not exactly what i'm looking for but it's close

oh this is helpful https://docs.datomic.com/reference/entities.html

There are these tutorials that helped me. https://docs.datomic.com/tutorial/tutorials.html

it doesn't have very much about schemas 😓 https://docs.datomic.com/client-tutorial/client.html#transact-schema

i'm trying to figure out how to model ordered lists and it seems ... quite complicated

Maybe or maybe not helpful to you: I have some objects in my schema that the user can arbitrarily sort, so I just have this generic attribute that I can attach to any of these objects as needed:

{:db/ident           :item/sort-key
  :db/valueType       :db.type/long
  :db/cardinality     :db.cardinality/one}

Try https://max-datom.com/

> trying to model ordered list congratulations on picking far and away the hardest db modeling problem

maybe i should just serialize EDN into the database or something idk

i don't actually need to query it, just store it

> Try https://max-datom.com/ this is adorable omg

maybe another thing to ask: datomic schemas seem very verbose to me, even compared to SQL. is that just normal, i'm not missing anything?

Maybe less verbose:

(defn- attr
  ([ident type cardinality] (attr ident type cardinality nil))
  ([ident type cardinality doc] (attr ident type cardinality doc nil))
  ([ident type cardinality doc more]
   (let [type (keyword "db.type" (name type))
         cardinality (keyword "db.cardinality" (name cardinality))
         more (if doc (assoc more :db/doc doc) more)]
     (merge {:db/ident ident
             :db/valueType type
             :db/cardinality cardinality}
            more))))

(defn- component
  "A component attribute"
  ([ident cardinality]
   (dissoc (attr ident :ref cardinality "" {:db/isComponent true}) :db/doc))
  ([ident cardinality doc]
   (attr ident :ref cardinality doc {:db/isComponent true})))

(def test-schema ;; define & add more related attr
  [(attr :test/name :string :one "The name of test")
   (attr :test/joined-at :instant :one "When test was created")
   (component :test/children :many "The children to carry with test")])

(def schema
  (vec (concat test-schema
               [(attr :transaction/namespace :string :one "Which namespace initiated the transaction")])))

(defn install-schema! [conn]
  (d/transact conn {:tx-data schema}))

> maybe i should just serialize EDN into the database or something idk Take care that they aren't too long. Large strings (or bytes for that matter) are not suitable for storage in Datomic. Better to store large values somewhere else (e.g. s3) and store the pointer as a datom.

something to throw in the mix is that if you want to leverage composite key tuples for uniqueness constraint purposes, all the member attributes need to be together on the same entity. a contrived example: won't work because the values are on different entities

:partner/user REF to :user/* entity
:user/partner-external-id STRING

:user/partner REF to :partner/* entity
:user/partner-external-id
:user/partner+external-id-tuple :db/tupleAttrs [:user/partner :user/partner-external-id]