I've had the same problem, my approach has been to offload the work to ElasticSearch. I think Datomic is just not well suited for low-latency analytical queries that span a lot of data; fortunately, thanks to the txReportQueue and the Log API, it's very well suited to be a source for derived data systems.

Also note that the current Datalog engine is not completely immune to the N+1 problem; I've observed that running a Datalog query which only needs one index access is still 100x slower than using the raw index API - as if there was some startup time associated with the Datalog engine

Of course I encourage you to profile and draw your own conclusions

thanks, these very useful insights! I’m already working off derived data (source data is also in datomic but in different partitions), but there are just some conditions that datalog seems to fall flat under

I’m also trying to keep the stack very flat and simple, it is not a huge app, just a lot of varied data (but not “big data” either)

you should use Datalog when you need to model joins. if you know you’ll only be using one index, you can almost certainly do it faster with d/datoms, because Datalog will always produce two result sets one for the clause, and one for the find expressions. reductions over d/datoms produce only one.

Thanks Rob, I’ll have a look. I’m really sure I can do it with one index, if not, it won’t be a big jump to restructure the derivatives to make this possible

ZOMG Rob! You just opened my eyes to something amazing