datalevin 2023-05-26 | Slack Archive

andersmurphy11:05:46

So I’m finding when I use collection bindings. I.e pass a collection and :in $ [?x …] I’m much more likely to run out of memory, even if the collection contains a single value. Inlining the values and performing an or doesn’t result in running out of memory. Is there something that makes collection bindings inherently expensive? This implementation does run out of memory (with large datasets):

(d/q '[:find (pull ?a [:artist/name])
       :in $ [?c ...]
       :where [?a :artist/country ?country]
              [?country :country/name ?c]]
     db ["Canada" "Japan"])

The implementations bellow don’t run out of memory (with large datasets):

(d/q '[:find (pull ?a [:artist/name])
       :where [?a :artist/country ?country]
       (or [?country :country/name "Canada"]
         [?country :country/name "Japan"])]
  db)

(d/q '[:find (pull ?a [:artist/name])
         :in $ ?c1 ?c2
         :where [?a :artist/country ?country]
         (or [?country :country/name ?c1]
           [?country :country/name ?c2])]
    db
    "Canada"
    "Japan")

Huahai15:05:30

the current query engine is from datascript. It should be obvious that product relations (cartesian product) are calculated to handle collection binding, hence OOM. I will focus on finishing the new query engine after 0.9.0 is released.

andersmurphy16:05:42

Thanks! That makes sense. I was running aground because I was reading datomic docs and they suggested use the collection bindings. I guess they have a query planner/engine. Have to say my experience with datalevin so far has been really good, the documentation is great and the query performance has been solid and didn’t realise that was without an optimised query engine. Looking forward to that when it comes! Thanks for open sourcing this and all the hard work!

🙏 2

Huahai22:05:37

Datomic doesn’t have a a query optimizer either, most of the queries in Datomic would be slower than us, but there are a few cases where they are faster. This is probably one of those.

👍 2

2023-05-26

Channels