Fork me on GitHub
#datomic
<
2021-10-11
>
Hannes Simonsson12:10:30

Hi, I'm just starting out learning about datomic and have run into an issue with querying entity id's. When I run the following query it returns all my activation id:s with all corresponding entity id:s.

  (db/q '[:find ?a ?b
          :where [?a :activation/id ?b]]
         db)
However, if I instead run:
  (db/q '[:find ?a 
          :where [?a :activation/id "some-id-I-know-is-stored-in-the-db"]]
         db)
Then it returns an empty list instead of the entity id I expect it to return, there should be only one since activation id is a uuid. Anyone know what the problem is, have I misunderstood something?

favila12:10:48

Uuid literal syntax in Clojure is #uuid “the uuid”

favila12:10:21

Is that what you are doing? A string and an actual uuid object are not going to compare equal

Hannes Simonsson12:10:34

That was it, thank you so much. Been banging my head on this for some time.

kirill.salykin13:10:51

hi Why there is no query planner in datomic and is there a plan to add it eventually?

Linus Ericsson13:10:47

There is no query planner in Datomic, I think this is because building a good query planner is hard, and also it is not always perfect, which could lead to suprising (very slow) query performance. That said, it is quite easy to build a query planner on your own, but probably best suited for data that you know much about.

lispers-anonymous15:10:53

How might you go about building a query planner on your own? First you say it's hard, then you say it's easy. Seems like it would be very hard to do on your own without concrete insights into the performance of your queries. In something like postgres you can use analyze and explain directives to help with that (granted, those are for explaining what their query planner is doing). Datomic does not provide anything.

kirill.salykin16:10:42

Good point, no much query stats available

raspasov08:10:49

I think the point was that it’s relatively “easy” to build a query planner for data that you know. General query plan that works well for unknown data is always “hard”.

kirill.salykin16:10:25

Was hoping that having nubank behind datomic may help to bring in some additional features...

dvingo17:10:14

I don't buy the argument that it is too hard of a problem - xtdb features a query https://github.com/xtdb/xtdb/blob/80b79534589b98c899495d86b70c05797e991c37/core/src/xtdb/query.clj#L1664 and has a small team of core contributors.

dvingo17:10:33

I asked in this channel in the past about why there is no query planner and never got a response, so maybe someone will answer this time.

vncz18:10:46

I’m curious about this. Why are you looking for a query planner in Datomic? Are you seeing poor performances or anything that makes you want one?

lispers-anonymous18:10:47

One use case for a query planner would be a single query that can be run against many databases. It's one thing to look at the query and re-order clauses to perform well for one particular database. But when the query runs against multiple databases, the shape and the dimension of data could mean that optimizing the query order for one database causes the query to perform poorly against another. It's very hard to optimize something like this at development time. When you are running a query with a lot of rules that call each other (sometimes recursively) that makes it even more difficult to think about how clauses should be ordered, or to even determine what actual order the clauses are run in the first place. It just feels like something that should be done programatically at run time. The right information is not always available when writing the queries.

kirill.salykin18:10:36

Also missinh query planner means that it is responsibility of a developer

dvingo19:10:04

some reasons I can think of: • the optimization of clause ordering can change over time as the # of entities that match a clause can change as the application lives for more time (something that was performant at one point in time may not be in the future) • developing on a team with a library of rules, where it is not feasible to inspect the clause ordering of all the rules (especially when rules invoke other rules, or invoke subqueries) • developer ergonomics - the same reasons you want to program in a higher level programming language - these are details a compiler should deal with

dvingo19:10:22

And to the original prompt - yes this was spurred by performance problems. There were more than a handful of code changes on a large project related to clause ordering causing poor performance. So stated again, the benefit of a query planner is cost savings - all the debugging and dev time needed to address this (and again addressing it once doesn't mean the problem is "fixed").

catjam 1
Ivar Refsdal19:10:04

We've also been hit by poor query performance, and I've reproduced an OOM (OutOfMemoryError) with a large list binding here: https://gist.github.com/ivarref/0d3d34eeeffbc4625d6120727368e405 In production this happens (OOM) once two input parameters become a total of ~1000 items.

dvingo20:10:29

Also, my original question (from June) was not requesting a query planner, I was asking if one was ever considered in datomic, and if so why there isn't one (what are the tradeoffs etc) because I don't see that documented anywhere.

drewverlee20:10:20

How would a query planner work in this case? Inspect your access patterns and DB and try to guess and future cases and re-index and re-order clauses? My naive guess is to why it's not there is because they would be worried about trying to be to smart and creating unpredictable behavior that would backfire. I would be interested to know if there is anything that prevents us from writing a query planner.

dvingo16:10:26

there's an open source one available here if you want to see one way to go about it: https://github.com/xtdb/xtdb/blob/master/core/src/xtdb/query.clj