This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-02-01
Channels
- # announcements (14)
- # architecture (30)
- # aws (34)
- # babashka (18)
- # beginners (114)
- # biff (5)
- # calva (128)
- # clerk (155)
- # clj-kondo (60)
- # clojure (82)
- # clojure-dev (25)
- # clojure-europe (20)
- # clojure-nl (1)
- # clojure-norway (17)
- # clojure-spec (13)
- # clojure-uk (3)
- # community-development (4)
- # core-logic (4)
- # cursive (5)
- # datomic (21)
- # deps-new (13)
- # emacs (5)
- # funcool (5)
- # graphql (3)
- # hyperfiddle (1)
- # introduce-yourself (1)
- # jobs (2)
- # kaocha (1)
- # london-clojurians (1)
- # lsp (13)
- # malli (16)
- # off-topic (6)
- # other-languages (1)
- # pathom (18)
- # re-frame (23)
- # releases (1)
- # remote-jobs (2)
- # tools-build (1)
- # tools-deps (12)
- # vscode (1)
- # xtdb (27)
if one tests a potential schema and profiles some queries with test data using dev-local do the results predict real-world performance once deployed to AWS ?
the query engine will work the same way but almost certainly the cardinalities of your entities are smaller in your tests, as well as differing in storage and caching characteristics
given enough realistic sample data, you can use the newly released https://blog.datomic.com/2023/01/Query-Stats.html functionality to gain confidence
ok thank you
but does local query performance vs cardinality not even scale similarly to that on AWS ?
and surely a change in :where
order which resulted in better performance of a query locally would also result in better performance on AWS ?
yes I get that
this seems unlikely, because of what I know about the separation between the query engine and the datasource. The query planner doesn’t even make index choices or resolve keywords to attributes
it gives the datasource a pattern, and gets (ultimately) an iterable of things matching that pattern
Datomic executes your :where
clauses in the order you specify: https://docs.datomic.com/on-prem/best-practices.html#most-selective-clauses-first
That said, with the addition of https://docs.datomic.com/on-prem/api/query-stats.html, you can now observe per-clause cardinalities and adjust your clause ordering accordingly.
is grounding a unique value enough to bust the cache when running a query?
'{:where [[?item :item/id 123]
[(ground "some-unique-value") ?grounded]]}
Are you referring to the caching mechanism mentioned here https://docs.datomic.com/on-prem/best-practices.html#parameterize-queries?
With the new query-stats feature (which is great!) I’ve been able to observe that it doesn’t always execute the :where
clauses in the specified order. Here is a real example from a debugging session yesterday. Note that the [(!= ?group ?group2)]
clause has been moved up in the :sched
plan compared to its location in the original :where
clause. I’m just wondering if that reordering is predictable/stable/consistent or if it depends on the cardinality of the data in some way.
:query-stats
{:query
{:find [[?group2 ...]],
:in [$ ?group],
:where
[(not [?group :group/archived?])
[?group :group/workspace2 ?wksp2]
[?group :group/name ?name]
[?group2 :group/workspace2 ?wksp2]
(not [?group2 :group/archived?])
[?group2 :group/name ?name]
[(!= ?group ?group2)]]},
:phases
[{:sched
(([(ground $__in__2) ?group]
(not-join [?group] [?group :group/archived?])
[?group :group/workspace2 ?wksp2]
[?group :group/name ?name]
[?group2 :group/workspace2 ?wksp2]
[(!= ?group ?group2)]
(not-join [?group2] [?group2 :group/archived?])
[?group2 :group/name ?name])),
Correct! Predicate clauses, such as the !=
clause you mentioned, do get reordered when the query schedule is created so that all required logic variables are bound before that predicate clause is used and get moved close to the clause where the predicate will be used.
You should also see that [(!= ?group ?group2)]
isn't listed as a clause under :clauses
, but instead is attached to the [?group2 :group/workspace2 ?wksp2]
clause underneath the :preds
key.
Correct! Predicate clauses, such as the !=
clause you mentioned, do get reordered when the query schedule is created so that all required logic variables are bound before that predicate clause is used and get moved close to the clause where the predicate will be used.
You should also see that [(!= ?group ?group2)]
isn't listed as a clause under :clauses
, but instead is attached to the [?group2 :group/workspace2 ?wksp2]
clause underneath the :preds
key.