This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
For a large d/q
on datomic cloud query which returns a large amount of data and takes almost one minute to finish, does it make sense to split the large query into many small queries and run these queries in parallel and combine the results? I think this could help reducing the query time but I am not sure whether this can reduce the memory requirement?
Thanks @U09R86PA4, very useful information. Another technique I'd like to try is to use custom query/aggregate functions to filter and reduce the result set returned by Datomic, I think for large data set, this might perform better than pulling all results and doing the computation and filtering in Clojure, since this would required less data transferred from Datomic to Clojure
You can test whether this is worth it by doing some simple aggregate with zero cost and max result set size compression (eg count) and measuring the difference with and without it
Aggregates still realize the entire result set so you really only save IO and de/serialization time