This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2023-05-03
Channels
- # aleph (7)
- # announcements (6)
- # babashka (34)
- # beginners (5)
- # calva (1)
- # cider (3)
- # clerk (11)
- # clj-http (1)
- # clj-kondo (35)
- # clojars (6)
- # clojure (114)
- # clojure-australia (1)
- # clojure-brasil (2)
- # clojure-europe (73)
- # clojure-hamburg (3)
- # clojure-nl (1)
- # clojure-norway (27)
- # clojure-uk (4)
- # events (3)
- # graalvm (13)
- # gratitude (4)
- # helix (17)
- # hoplon (1)
- # hyperfiddle (65)
- # instaparse (4)
- # jobs-discuss (6)
- # lsp (2)
- # meander (2)
- # rdf (4)
- # re-frame (51)
- # reitit (28)
- # releases (1)
- # sci (20)
- # shadow-cljs (9)
- # tools-deps (4)
- # xtdb (44)
https://docs.xtdb.com/clients/clojure/ has multiple broken links in the html and in the docstrings pointing to pages under https://xtdb.com/reference/
I'm having a weird problem where we cannot find any document by trying to find it by an attribute (which is a map, no idea if that's incidental) in :where but we can find it by simply filtering over all result documents for that attribute. But cannot reproduce as pushing the same document to a clean database (no matter if it's inmem, rocks or postgres) will make the same query work. Any idea how to debug this kind of problem?
are the docs introduced by a tx function expansion? just out of curiosity, as there were some problems with that (#1825)
I haven't, however, tried putting the same documents again in the production db, just on a fresh one
these kinds of problems in production are super scary, hope you get to the bottom of it
Hey @U8ZQ1J1RR is there a chance that the map has metadata on it (either at the time of submit, or in your query)? Clojure metadata can throw off the hashing which is used for comparisons
It doesn't, I'm afraid. Trying to figure out what is happening with that map otherwise. Checking all the types recursively etc
Nope, not that either. Contents are clojure.lang.Keywords and java.lang.Integers without meta
So I can find the document if I keep a handle to the original document, and get the variable from there. But not if I print it, and then query with a map I just created by hand. Even though they both are equal according to =
> trying to find it by an attribute (which is a map, no idea if that's incidental) To confirm my understanding, the value under the attribute is a map? You're not using a map as a key at all here?
{
...
:expense/customer-ref {:external-system :acme, :ref 123456789}
:expense/organization #uuid "abbaabba-1234-1234-1234-1111abcd2222"
:expense/type :invoice
...
}
And we are trying to fetch the document like
(xt/q (xt/db db)
'{:find [(pull e [*])]
:where [[e :expense/customer-ref customer-ref]
[e :expense/organization organization-id]
[e :expense/type type]]
:in [customer-ref organization-id type]}
customer-ref organization-id type)
Finding with a '"simpler" attribute value works nicely, it is this single attribute with a map as value which causes issues.hmm - have you been able to repro this with a simple in-memory XTDB node outside the context of your app?
Quite frustrating indeed. I have code where
[get-by-ref api/db customer-ref1 org-id :type)
get-by-ref api/db customer-ref2 org-id :type)
(= customer-ref1 customer-ref2)]
returns the actual document, nil, and true (where the first ref is taken with (:customer-ref (first data)) and second is copy pasted value)I'm trying to find the relevant parts of xtdb code where it does the comparisons. Any pointers?
I have a minimal sample now, though I need to remove all the cruft (tomorrow). The key finding is this:
(comment
(xt/submit-tx node [[::xt/put {:xt/id 1 :ref {:key :val :int 1}}]]))
(comment
(xt/q (xt/db node)
'{:find [e]
:where [[e :ref ref]]
:in [ref]}
{:key :val :int 1}))
(comment
(xt/q (xt/db node)
'{:find [e]
:where [[e :ref ref]]
:in [ref]}
{:key :val :int
(jsonista/read-value (jsonista/write-value-as-string 1))}))
First query works, second doesn't. Also if the values are plain ints, not inside a map, even the jsonista version works
so is this the cause?
user> (type (jsonista.core/read-value "1"))
java.lang.Integer
user> (type 1)
java.lang.Long
jsonista is returning a number as int, whereas clojure reader as long… they both look exactly the same in the REPL and are equalbut they don’t serialize the same
xtdb.codec> (with-open [out (java.io.ByteArrayOutputStream.)] (freeze-map (java.io.DataOutputStream. out) {:n (int 1)}) (.toByteArray out))
[112, 1, 106, 1, 110, 42, 0, 0, 0, 1]
xtdb.codec> (with-open [out (java.io.ByteArrayOutputStream.)] (freeze-map (java.io.DataOutputStream. out) {:n 1}) (.toByteArray out))
[112, 1, 106, 1, 110, 100, 1]
if you coerce the number returned by jsonista by calling long
on it, does that query then work?
But that's only needed when the value is inside a map. Plain int and longs seem to be interchangeable in XTDB queries
Pretty tricky situation. https://github.com/xtdb/xtdb/blob/master/core/src/xtdb/codec.clj#L231 seems to coerce ints to longs, but changing the serialization of maps is bound to cause headaches too.
And to be a bit safer, I suppose that deep map normalization needs to do all the same coercions xtdb does for plain values, not just ints. So floats to doubles, bigints to bigintegers and so on
I appreciate you both digging in to figure that out, thank you! Agreed this is rather subtle footgun - the team will review and reflect. In the meantime I have filed an issue and will keep you posted https://github.com/xtdb/xtdb/issues/2528
