This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # aws-lambda (3)
- # beginners (37)
- # boot (294)
- # carry (1)
- # cider (38)
- # cljs-dev (37)
- # cljsjs (88)
- # clojure (187)
- # clojure-android (2)
- # clojure-austin (1)
- # clojure-dusseldorf (9)
- # clojure-hk (3)
- # clojure-italy (12)
- # clojure-russia (36)
- # clojure-spec (55)
- # clojure-uk (27)
- # clojurescript (75)
- # community-development (5)
- # conf-proposals (2)
- # copenhagen-clojurians (3)
- # cursive (9)
- # datomic (54)
- # devcards (5)
- # devops (3)
- # dirac (69)
- # emacs (6)
- # ethereum (1)
- # euroclojure (1)
- # events (3)
- # funcool (1)
- # hoplon (20)
- # immutant (4)
- # luminus (14)
- # midje (4)
- # om (178)
- # om-next (2)
- # onyx (47)
- # pedestal (19)
- # protorepl (20)
- # re-frame (14)
- # reagent (54)
- # ring (2)
- # ring-swagger (7)
- # test-check (10)
- # uncomplicate (11)
- # untangled (9)
- # yada (9)
@robert-stuttaford Hi! The ever-decreasing index for latest entities is working great, but I stumbled over this from DataScript: https://github.com/tonsky/datascript/wiki/Tips-&-tricks#getting-top-10-entities-by-some-attribute - Quote Nikita: > "Reverse return a special view on an index that allows walking it in the reverse direction. This operation is allocation free and about as fast as direct index walking." Would this be possible in Datomic as well? Or is this possible for DataScript because it keeps everything in memory? Any thoughts? 🙂
very quick testing seems to suggest that the same thing works in Datomic, @magnars
with a warm cache, counting 50k datoms forwards and backwards takes the same amount of time
(time (->> (d/datoms (db/db) :aevt :chat.event/client-uuid) seq reverse count)) ;; with no reverse "Elapsed time: 925.572616 msecs" 3809777 ;; with reverse "Elapsed time: 2667.018015 msecs" 3809777
(time (->> (d/datoms (db/db :events) :aevt :event/uuid) seq reverse count)) ;; with no reverse "Elapsed time: 1492.233743 msecs" 6576672 ;; with reverse "Elapsed time: 4446.358167 msecs" 6576672
But the implementation of
reverse in clojure.core is not lazy, and looks like
(reduce1 conj () coll)
Nikita says "This operation is allocation free", so he can't be talking about that implementation of reverse.
I think you can get a pretty good idea just by trying to iterate reasonably big database in reverse
user=> (time (first (d/datoms (db/db) :aevt :event/uuid))) "Elapsed time: 0.322985 msecs" user=> (time (first (reverse (d/datoms (db/db) :aevt :event/uuid)))) "Elapsed time: 124.845047 msecs" user=> (time (count (seq (d/datoms (db/db) :aevt :event/uuid)))) "Elapsed time: 188.143681 msecs"
That's very interesting. Thanks for the answer, @tonsky. I'll keep my ever-decreasing index for now then. 🙂
I remember reading (that was ~2 years ago, but still) that they recommend to store negative values if you want quick access to latest, not first, datoms
I believe B-Tree index can handle insertion to the head just as well as it does instertions to the tail
The structure of Datomic's indexes in storage makes iterating in reverse non-trivial. When you call
reverse you're just using ordinary Clojure sequence
reverse, which realizes the whole sequence in memory.
I wouldnt suggest over 100k. That actually is a bit high, but of course like everything, it depends
I'm trying to excise an entity, though in my tests the entity is available immediately to (d/touch ) on a new db instance... Is there some kind of indexing period when the entity will still be there?
I'm successsfully excising in other parts of my app... My excised entity does have some ref'd entities that are components, I don't know if that makes a difference.
How many datoms are you excising? At some point after the excision the indexing job runs. The resulting index will no longer contain the datoms excised. The indexing job however is proportional to the size of the entire database.
Cool. I'll look into that... It's in my tests, just a single entity with 5 datoms, and a component with 2. I don't totally need to excise it, I'm just a little paranoid after reading about 10B datoms and I'm not particularly worried about preserving history on this individual entity type.
@jdkealy everything in Datomic is optimized around immutability so in general it’s a terrible idea to excise. I’d leave it to the “legally compelled to remove this fact” case.
@jdkealy To echo what Ben said I highly recommend that you not go that route. @marshall and I have been discussing this “problem” with other clients and if you think you’d benefit maybe we can chat about solutions
Yes, I'd like that... I'm fine with adding some attribute like archived:true/false or something like that... I'm really just concerned with the 10BN datoms I've been reading about. I think this kind of data I might want to use a different data store for.
@jdkealy We’re happy to set up a call. Can you email me at <mailto:[email protected]|[email protected]> ? Also, you could consider noHistory on some attributes if you don’t want/need any history tracking on them
I think I'm missing something obvious here: why is it that
works (finds several ids), but
'[:find ?id :where [?id :foo/bar 33]]
returns an empty set? I could imagine that being disallowed by datomic, but if so, I'd expect an error rather than just getting no results.
'[:find ?id :where [?id _ 33]]