How do you know all are in the object cache? index-range doesn’t provide io-stats. How does the index-range compare to just d/datoms over the same attr? e.g. (time (count (d/datoms db :avet attr)))

and when you say “all of the time”--is this cpu time or wallclock time?

do you use index-range with an “end” parameter? Does it make a difference if you don’t?

Great questions. • Object Cache: my actual query is something like this [:find ?foo :in $ ?bar :where [(function-that-uses-index-range $ ?bar) ?foo] , and, pleasantly surprisingly - this seems to accurately report io stats. The underlying function only uses index-range. I also am running the profiling after the first execution of the query. • I should've said that i'm sampling, not profiling, so it can be imprecise, but is reported as cpu time • For my use case, not using end would be fetch way too much data. The composite tuple's schema is [:db.type/ref :db.type/instant], and so my range would typically look something like this: start: [some-entid #inst "2020-01-01"] end:`[some-ent-id #inst "2023-01-01"]`

I’m asking about “end” and vs d/datoms because maybe comparing values until the end of the range is what is slow. Both of these would remove that.

it could also just be allocations

query is going to realize every ?foo

and going to do set arithmetic on those values

d/index-range is lazy--is it also 700ms when you just run that function by itself with e.g. count to realize it without allocating all of it?

This is also where a comparison with d/datoms is instructive

Ah, interesting, (time (count (d/datoms db :avet attr))) takes 400ms

another comparison you can make, if function-that-uses-index-range is simple to port to a query, is to see how the query does when expressed directly with [?e :attr ?v][(<= ?start ?v)][(< ?v ?end)]

Yeah, my introduction of function-that-uses-index-range was introduced because the query performance was much worse. Raw usage of index-range without realizing every ?foo is much faster (200ms). I think that's the clue I needed

If your output set is significantly smaller than the input, consider reducing over chunks of your index-range input and running the query for each chunk

I don’t really understand why this is better; I theorize that it’s all about memory pressure from large sets

Got it, okay this is helpful, thank you.

e.g. (into #{} (comp (map :e) (partition-all 10000) (mapcat #(d/q query db %)) (d/index-range db attr start end))

map/reduce style