Clojurians Log v2

My first way to confirm how many attributes we used was a simple count:

(datomic.api/q
  '[:find (count ?a) .
    :where 
    [?a :db/valueType]]
  (datomic.api/history db))

=> 52850

(datomic.api/q
  '[:find (max ?e) .
    :where 
    [?e :db/valueType]]
  (datomic.api/history db))

=> 1047018

And in the error message does "elements" refer to "schema entities" or to something else like datoms?

It refers to entities in partition 0 (:db.part/db)

do you have lots of those? e.g. maybe partition entities or idents?

if you are on peer, you can do this:

(->> db :elements (map class) frequencies)

(->> db :elements (map class) frequencies)
=> {datomic.db.Partition 3, nil 994148, datomic.db.Attribute 52850, datomic.db.ValueType 16, datomic.db.Function 2}

did you manually assign the db-id of an attribute and skip a whole bunch of numbers?

or accidentally transact data into partition 0 using :db.part/db tempids?

or maybe you have a schema metalayer that uses lots of entities in partition 0?

nil 994148 is the problem; those are either real entities consuming the count, or skipped numbers

i don't think so but could it be related to how we transact the schema elements? we don't use the map syntax and use [:db/add ...] with tempids. we do have some meta schema stuff that might be ending up partition 0 although I thought they were all just ref datoms hanging off schema elements and pointing to entities in the user partition

> I thought they were all just ref datoms hanging off schema elements and pointing to entities in the user partition Not sure what you mean here. To clarify: [:db/add attr :my-schema-meta x] doesn't consume an entity id in part0; [:db/add (d/tempid :db.part/db -1) ...] or {:db/id attr :component-attr {...}} does

k i'll dig into what the others things are in partition 0 a bit

this may help

(transduce
 (take-while #(zero? (d/part (:e %))))
 (completing
  (fn [m [e a]] (update m (d/ident db a) (fnil inc 0))))
 {}
 (d/seek-datoms db :eavt 0))

{:db/index 42178,
 :db/unique 9,
 :db/valueType 52850,
 :meta/schema 947,
 :db/tupleType 2,
 :db/isComponent 4547,
 :db/lang 2,
 :db/fulltext 2,
 :attribute/content-type 4,
 :db/cardinality 177986,
 :db/doc 166,
 :db/ident 58971,
 :db/code 2,
 :attribute/malli-schema 4,
 :db.install/valueType 16,
 :db.alter/attribute 73445,
 :attribute/resolves 26,
 :db.install/function 2,
 :db.install/partition 3,
 :db/tupleAttrs 2,
 :db.install/attribute 52850,
 :fressian/tag 16,
 :attribute/groups 31}

...schema alteration

oof

I did not see that coming

yeah i know we had an issue a while ago with some of our attributes toggling back and forth between one/many cardinality (we infer schema and two competing tools weren't agreeing on cardinality). Maybe that's still in play. The above is from running your snippet on the history db

if I sum those i get 464061 though which is less than the 994148 nil entries, should I not expect those to match?

it's not the full story, but it's more than doubling your attr count

what does :db.alter/attribute represent?

a schema alteration

what classifies as an alteration? changing the ident on an attribute?

any of these: https://docs.datomic.com/schema/schema-change.html

is implicit, and was explicit on older syntax. see e.g. https://docs.datomic.com/schema/schema-change.html#removing-avet-index

(->> (datomic.api/seek-datoms (datomic.api/history db) :eavt 0)
     (take-while (fn [x] (zero? (datomic.api/part (:e x)))))
     (map :e)
     (dedupe)
     (count))
=> 52880
(->> (datomic.api/seek-datoms (datomic.api/history db) :eavt 0)
     (take-while (fn [x] (zero? (datomic.api/part (:e x)))))
     (map :e)
     (count))
=> 464061

what might the rest of the story be if both of those are still well below 2^20? something causing it to skip IDs?

that's all I can think of

maybe informative to seek over EAVT, noting delta of E vs previous E, looking for big delta

surprised at the number of retractions

there are 2831 gaps in numbering ranging in size from 2 to 10781