@robert-stuttaford Hi! The ever-decreasing index for latest entities is working great, but I stumbled over this from DataScript: https://github.com/tonsky/datascript/wiki/Tips-&-tricks#getting-top-10-entities-by-some-attribute - Quote Nikita: > "Reverse return a special view on an index that allows walking it in the reverse direction. This operation is allocation free and about as fast as direct index walking." Would this be possible in Datomic as well? Or is this possible for DataScript because it keeps everything in memory? Any thoughts? 🙂

i'll try it and let you know, @magnars 🙂

very quick testing seems to suggest that the same thing works in Datomic, @magnars

which is -ing awesome

whoa, that is impressive

with a warm cache, counting 50k datoms forwards and backwards takes the same amount of time

trying a bigger index

is that using a special trick, or does it work out of the box with reverse?

(time (->> (d/datoms (db/db) :aevt :chat.event/client-uuid)
             seq
             reverse
             count))

  ;; with no reverse
  "Elapsed time: 925.572616 msecs"
  3809777

  ;; with reverse
  "Elapsed time: 2667.018015 msecs"
  3809777

nearly 3x slower for a larger index

but still quite quick

yeah, not bad

(time (->> (d/datoms (db/db :events) :aevt :event/uuid)
             seq
             reverse
             count))

  ;; with no reverse
  "Elapsed time: 1492.233743 msecs"
  6576672

  ;; with reverse
  "Elapsed time: 4446.358167 msecs"
  6576672

But the implementation of reverse in clojure.core is not lazy, and looks like (reduce1 conj () coll)

that-shrug-emoji 🙂

Nikita says "This operation is allocation free", so he can't be talking about that implementation of reverse.

@tonsky, any thoughts?

@magnars https://github.com/tonsky/datascript/blob/0f942b9666c7c7bfbecb5a45368c61569718d1ff/src/datascript/btset.cljc#L838

i guess we'll have to ask the fine Cognitects if a similar thing happens for Datomic indexes.. @marshall, @jaret? 🙂

no idea what datomic does

I think you can get a pretty good idea just by trying to iterate reasonably big database in reverse

@robert-stuttaford you clearly don’t want to do seq before reverse

if there were any kind of optimizations that might kill it

user=> (time (first (d/datoms (db/db) :aevt :event/uuid)))
"Elapsed time: 0.322985 msecs"

user=> (time (first (reverse (d/datoms (db/db) :aevt :event/uuid))))
"Elapsed time: 124.845047 msecs"

user=> (time (count (seq (d/datoms (db/db) :aevt :event/uuid))))
"Elapsed time: 188.143681 msecs"

pretty sure there’s no optimization for it

which is a real bummer: all they had to do is implementing reverse iterator

you can write to mailing list and they might consider that for the next version

That's very interesting. Thanks for the answer, @tonsky. I'll keep my ever-decreasing index for now then. 🙂

I remember reading (that was ~2 years ago, but still) that they recommend to store negative values if you want quick access to latest, not first, datoms

we even use to store negative timestamps (as longs) back in these days, for that reason

yeah, that’s the solution

I believe B-Tree index can handle insertion to the head just as well as it does instertions to the tail

thanks @tonsky!

The structure of Datomic's indexes in storage makes iterating in reverse non-trivial. When you call reverse you're just using ordinary Clojure sequence reverse, which realizes the whole sequence in memory.

Thanks for the clarification, Stuart. 👍

is 100k datums still consider the the rough size limit for datomic transactions?

10 billion is the figure I hear most often Ignore me.

per transaction?

^whoops. misread

I wouldnt suggest over 100k. That actually is a bit high, but of course like everything, it depends

ideally you’re in the thousands to tens of thousands max

per transaction

I'm trying to excise an entity, though in my tests the entity is available immediately to (d/touch ) on a new db instance... Is there some kind of indexing period when the entity will still be there?

I'm successsfully excising in other parts of my app... My excised entity does have some ref'd entities that are components, I don't know if that makes a difference.

How many datoms are you excising? At some point after the excision the indexing job runs. The resulting index will no longer contain the datoms excised. The indexing job however is proportional to the size of the entire database.

you can utilize sync-index to force the execution of the indexing job

Cool. I'll look into that... It's in my tests, just a single entity with 5 datoms, and a component with 2. I don't totally need to excise it, I'm just a little paranoid after reading about 10B datoms and I'm not particularly worried about preserving history on this individual entity type.

@jdkealy everything in Datomic is optimized around immutability so in general it’s a terrible idea to excise. I’d leave it to the “legally compelled to remove this fact” case.

@jdkealy To echo what Ben said I highly recommend that you not go that route. @marshall and I have been discussing this “problem” with other clients and if you think you’d benefit maybe we can chat about solutions

Yes, I'd like that... I'm fine with adding some attribute like archived:true/false or something like that... I'm really just concerned with the 10BN datoms I've been reading about. I think this kind of data I might want to use a different data store for.

@jdkealy We’re happy to set up a call. Can you email me at <mailto:[email protected]|[email protected]> ? Also, you could consider noHistory on some attributes if you don’t want/need any history tracking on them

Cool, will do marhall

thanks @marshall

I think I'm missing something obvious here: why is it that

'[:find ?id :where [?id :foo/bar 33]]

'[:find ?id :where [?id _ 33]]