Clojurians Log v2
Clojure programming
Channels
Apps
datalevin 2026-09-30
Samuel Ludwig 2026-09-30T18:33:42.941349Z
Running into a possibly rule-related performance oddity, and I'm not sure if its a bug or just a skill issue.
My data-model is similar to examples I've used in the past: I have a collection of entities (companies, for this example) that I'm tracking relationships between (the :rel entities), which are queried using as rules (defined in-thread)
(def schema
{:company/name {:db/valueType :db.type/string, :db/unique :db.unique/identity}
:rel/from {:db/valueType :db.type/ref}
:rel/to {:db/valueType :db.type/ref}
:rel/type {:db/valueType :db.type/keyword}})(d/q
'[:find ?c2
:in $ % ?c1
:where
(alias ?c1 ?c2)]
(d/db core/conn) query/base-rules 17108)(d/q ;; slow
'[:find ?c2
:in $ % ?c1
:where
(alias ?c1 ?c2)
[?c2 :company/name]]
(d/db core/conn) query/base-rules 17108)explain, it looks like the [?c2 :company/name] may have been resolved first
(both queries return the same values)
There might be some interaction with magic-rewrites happening? Running the above query with *magic-rewrite?* set to false returns instantly again:
(binding [datalevin.rules/*magic-rewrite?* false]
(d/q
'[:find ?c2
:in $ % ?c1
:where
(alias ?c1 ?c2)
[?c2 :company/name]]
(d/db core/conn) query/base-rules 17108))alias-net, which is the same as alias but also includes an [(identity ?a) ?b] clause, and it has the same fast performance regardless of the *magic-rewrite?* variable.
LASTLY (I'm sorry that this is a lot): it looks like that of the queries slowed by the magic-rewrite, that they are slowed down more when more unrelated :rel's are added to the DB.
The reproduction for this one is a little more involved (and, forgive me, partially slop-generated), because it only becomes apparent at scale, I will include it in-thread
If this is a known possibility, is there a general guideline of when magic-rewrites should be enabled or not? Or should they always remain enabled?Samuel Ludwig 2026-09-30T18:33:53.458869Z
Reproduction:
(require '[datalevin.core :as d]
'[datalevin.rules :as rules]
'[datalevin.query.cache :as qcache]
'[datalevin.util])
(def schema
{:company/name {:db/valueType :db.type/string, :db/unique :db.unique/identity}
:rel/from {:db/valueType :db.type/ref}
:rel/to {:db/valueType :db.type/ref}
:rel/type {:db/valueType :db.type/keyword}})
(def rules
'[[(rel ?type ?from ?to)
[?r :rel/from ?from]
[?r :rel/type ?type]
[?r :rel/to ?to]]
[(bidirectional-rel ?type ?a ?b) (rel ?type ?a ?b)]
[(bidirectional-rel ?type ?a ?b) (rel ?type ?b ?a)]
[(transitive-rel ?type ?a ?b)
(bidirectional-rel ?type ?a ?b)]
[(transitive-rel ?type ?a ?b)
(bidirectional-rel ?type ?a ?c)
(transitive-rel ?type ?c ?b)]
[(alias ?a ?b)
(transitive-rel :alias ?a ?b)
[(!= ?a ?b)]]
[(alias-net ?a ?b) [(identity ?a) ?b]]
[(alias-net ?a ?b) (alias ?a ?b)]])
;;; Seed data
(defn seed-db
"Creates 2000 companies, with 20 groups of aliases of 2-6 companies, and creates `n-other` random :rel entities. Returns an eid from each group of aliases."
[n-other]
(let [n-companies 2000
rnd (java.util.Random. 42)
dir (str (System/getProperty "java.io.tmpdir") "/dl-magic-repro-" (random-uuid))
conn (d/create-conn dir schema)
cid #(- (inc %))
groups (->> (range 20) (map #(+ 2 (mod % 5))) (reductions + 0) (partition 2 1))]
(d/transact! conn (mapv #(hash-map :db/id (cid %), :company/name (str "company-" %)) (range n-companies)))
(d/transact! conn (vec (for [[from to] groups, i (range from (dec to))]
{:rel/type :alias
:rel/from [:company/name (str "company-" i)]
:rel/to [:company/name (str "company-" (inc i))]})))
(when (pos? n-other)
(d/transact! conn (vec (repeatedly n-other
#(hash-map :rel/type :redirect
:rel/from [:company/name (str "company-" (.nextInt rnd n-companies))]
:rel/to [:company/name (str "company-" (.nextInt rnd n-companies))])))))
[conn dir (vec (range (ffirst groups) (second (last groups))))]))
;;; Timing
(defn median-ms [f xs]
(let [ts (vec (sort (for [x xs]
(let [t (System/nanoTime)] (f x) (/ (- (System/nanoTime) t) 1e6)))))]
(nth ts (quot (count ts) 2))))
(defn q [magic? query db & args]
(binding [rules/*magic-rewrite?* magic?
qcache/*cache?* false]
(set (apply d/q query db rules args))))
(def transitive-q '[:find [?x ...] :in $ % ?c :where (transitive-rel :alias ?c ?x)])
(def alias-q '[:find [?x ...] :in $ % ?c :where (alias ?c ?x)])
(def alias-net-q '[:find [?x ...] :in $ % ?c :where (alias-net ?c ?x) [(!= ?c ?x)]])
(def alias+attr-q '[:find [?x ...] :in $ % ?c :where (alias ?c ?x) [?x :company/name]])
(def cases
[["(transitive-rel :alias ?c ?x)" true transitive-q]
["(transitive-rel :alias ?c ?x)" false transitive-q]
["(alias ?c ?x)" true alias-q]
["(alias ?c ?x)" false alias-q]
["(alias-net ?c ?x) [(!= ?c ?x)]" true alias-net-q]
["(alias ?c ?x) [?x :company/name]" true alias+attr-q]
["(alias ?c ?x) [?x :company/name]" false alias+attr-q]])
;; Testing against a db with only the alias rels, and against a db with 2000 additional random rels:
(let [dbs (mapv seed-db [0 2000])
eids (fn [conn aliased] (mapv #(d/entid (d/db conn) [:company/name (str "company-" %)]) aliased))]
(doseq [[conn _ aliased] dbs, [_ magic? query] cases, c (take 3 (eids conn aliased))]
(q magic? query (d/db conn) c))
(println "Median ms per query, over 10 companies that have aliases (query cache off):\n")
(println (format "%-36s %-10s %14s %16s" "where" "magic" "0 other rels" "2000 other rels"))
(doseq [[label magic? query] cases]
(println (format "%-36s %-10s %14.1f %16.1f" label (if magic? "on" "off")
(double (median-ms #(q magic? query (d/db (first (dbs 0))) %) (take 10 (eids (first (dbs 0)) (last (dbs 0))))))
(double (median-ms #(q magic? query (d/db (first (dbs 1))) %) (take 10 (eids (first (dbs 1)) (last (dbs 1)))))))))
(println "\nAll queries equivalent?:"
(every? true?
(for [[conn _ aliased] dbs, c (take 10 (eids conn aliased))]
(let [db (d/db conn)
expected (q false transitive-q db c)
aliases (disj expected c)]
(and (= expected (q true transitive-q db c))
(= aliases (q true alias-q db c) (q false alias-q db c) (q true alias-net-q db c)
(q true alias+attr-q db c) (q false alias+attr-q db c)))))))
(doseq [[conn dir] dbs] (d/close conn) (datalevin.util/delete-files dir)))Samuel Ludwig 2026-09-30T18:34:58.097149Z
Running the above produces the following measurements for me:
Median ms per query, over 10 companies that have aliases (query cache off):
where magic 0 other rels 2000 other rels
(transitive-rel :alias ?c ?x) on 60.6 172.8
(transitive-rel :alias ?c ?x) off 4.5 4.1
(alias ?c ?x) on 56.3 169.1
(alias ?c ?x) off 4.5 4.2
(alias-net ?c ?x) [(!= ?c ?x)] on 4.3 4.4
(alias ?c ?x) [?x :company/name] on 805.0 642.3
(alias ?c ?x) [?x :company/name] off 10.1 13.1
All queries equivalent?: true