This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # announcements (28)
- # asami (12)
- # aws (3)
- # babashka (69)
- # babashka-sci-dev (34)
- # beginners (52)
- # biff (3)
- # calva (20)
- # clj-kondo (4)
- # cljsrn (4)
- # clojars (1)
- # clojure (90)
- # clojure-czech (2)
- # clojure-europe (33)
- # clojure-nl (11)
- # clojure-norway (35)
- # clojure-seattle (1)
- # clojure-uk (5)
- # clojurescript (87)
- # cursive (10)
- # datascript (5)
- # datomic (35)
- # defnpodcast (1)
- # emacs (8)
- # events (4)
- # fulcro (1)
- # google-cloud (2)
- # graphql (2)
- # hispano (2)
- # honeysql (5)
- # hoplon (2)
- # hugsql (1)
- # jobs (7)
- # kaocha (9)
- # lsp (102)
- # meander (13)
- # observability (7)
- # off-topic (56)
- # overtone (2)
- # pathom (47)
- # podcasts (1)
- # rdf (30)
- # reagent (16)
- # reitit (1)
- # releases (2)
- # remote-jobs (26)
- # rewrite-clj (10)
- # tools-deps (4)
- # vim (5)
- # vscode (4)
- # xtdb (41)
I have a deep structure shredded as documents that link A->B->C (document A has an attribute that is the id of B and so on)... now I want to find As by an attribute in C, this requires the query to consider all As and is quite slow... in datomic this kind of query is much faster as it has the VAE index for all entity references. Any tricks to make that faster other than pulling the search fields up to A and maintaining those manually.
Are the attributes always known in advance? If so, XT has an AVE index to avoid scanning, but the index may not being factored into the join order in an optimal way. Please can you share the query and the :vars-in-join-order for xtdb.query/query-plan-for?
I had an or clause filter that checked that either an attribute in A matches a text-search
or an attribute in C matches a text-seach... just searching from C the query performs well enough
tried looking at flamegraphs, but I don't understand internals enough... either of the clauses in the or individually are fast, (the whole query with one clause 200ms) then when combining, it goes to 16 seconds...
layered-idx->seq step seems to feature heavily, but can't say that is a culprit
Sadly regular flamegraphs can't follow the execution of individual clauses meaningfully, but even just a very crudely anonymised version of the query (+ :vars-in-join-order) with the logic vars replaced with foo/bar/etc should definitely be enlightening. Equally feel free to DM me
that sounds like a reasonable guess... it is much faster if I replace the text-search or clauses with regular triple clauses
is there an easy way to verify? I could probably monkey patch the text search code in the REPL...
I added a call counter to
xtdb.lucene/resolve-search-results-a-v and that at least is called 18282 times in my query, is that called for each potential result
Is there a way to know how exactly nested fields get broken up into datoms? Like, exactly what are the facts that can be matched against in a query
Can you provide an example? In general, nested fields won't match xtdb datalog queries, so I'm not sure what you mean. (Only top-level attrs in a doc are matched.)
And then - this might be a silly thing - but is there any particular reason the Java API exposes/works with Dates instead of Instants?
My guess is that this is because of historical reasons/backward compatibility,
Largely this, yes.
The dev team is taking a good hard look at data types right now, and this is definitely getting a serious revisit. The tentative plan is to come up with a stricter set of types which are deeply and precisely supported.
it's at least in part due to a preference for handling every temporal coordinate internally as a long (64-bits), which has a direct and widely-recognised correspondence with
While I'm asking dumb questions, are there plans out there for managed installs? Like, heroku addons, that kind of stuff
Not a dumb question at all. 🙂 Almost clairvoyant. We spoke to a large number of folks recently about what they'd like from xt and "management" (for various values of "management") came up repeatedly. On one end, that might just be a simpler Docker install or an AWS Marketplace component. On the other end, there's full-blown DBaaS.
@U3JH98J4R Out of curiosity, is a Heroku Addon specifically the thing you'd like to see us implement?
I'd like to hear what the answer is for each hat, if you have a sec to list them out.
for the company I work for, we are doing all our infra via pulumi on AWS and we have a strong preference for a managed service over anything we roll ourselves
for the people I teach, and would want to throw them the java library or the http api, I would want something single click on heroku and a straightforward local install similar or more convenient than postgres
for my personal doodlings I would want heroku and/or a docker image just so i can use it without thinking much
> managed service Just to completely disambiguate, this means a DBaaS that offers you an API but no direct access to the hardware, correct? (As opposed to something in AWS Marketplace, say, where you'd still manage the instance yourself.)
@U3JH98J4R We'd love to know. There's a wide spectrum between producing prebuilt marketplace binaries and actually managing other peoples' data. We're not averse to either, but actually running a full-blown DBaaS means investing heavily in admin/ops work. Managing other peoples' data isn't something to take lightly. 🙂
I think that as xtdb supports variety of backends for storage it should somehow be choosable with the back-end in mind. So for instance I imagine that on Heroku you could take a Postgres image and use XTDB with that Postgres as a storage engine. But some apps may run XTDB with in-memory only configuration. And then there's kafka and others. So I think that every one-click deployment can be very different about what it does.