I'll take a look, thanks 🙂

oops, yes thanks for flagging this!

I checked that attribute-stats shows the key as being indexed

and other attributes work ok?

At least two others that we use for the query, yeah

are the docs introduced by a tx function expansion? just out of curiosity, as there were some problems with that (#1825)

if you can’t reproduce by just putting a doc

No, just plain puts

I haven't, however, tried putting the same documents again in the production db, just on a fresh one

these kinds of problems in production are super scary, hope you get to the bottom of it

Hey @U8ZQ1J1RR is there a chance that the map has metadata on it (either at the time of submit, or in your query)? Clojure metadata can throw off the hashing which is used for comparisons

It doesn't, I'm afraid. Trying to figure out what is happening with that map otherwise. Checking all the types recursively etc

Hm, need to check the meta recursively

Nope, not that either. Contents are clojure.lang.Keywords and java.lang.Integers without meta

So I can find the document if I keep a handle to the original document, and get the variable from there. But not if I print it, and then query with a map I just created by hand. Even though they both are equal according to =

(variable is a wrong word here, the value I mean)

> trying to find it by an attribute (which is a map, no idea if that's incidental) To confirm my understanding, the value under the attribute is a map? You're not using a map as a key at all here?

can you share the map verbatim? (feel free to DM)

{
...
 :expense/customer-ref {:external-system :acme, :ref 123456789}
 :expense/organization #uuid "abbaabba-1234-1234-1234-1111abcd2222"
:expense/type :invoice
...
}

(xt/q (xt/db db)
            '{:find [(pull e [*])]
              :where [[e :expense/customer-ref customer-ref]
                      [e :expense/organization organization-id]
                      [e :expense/type type]]
              :in [customer-ref organization-id type]}
            customer-ref organization-id type)

hmm - have you been able to repro this with a simple in-memory XTDB node outside the context of your app?

Not yet

Quite frustrating indeed. I have code where

[get-by-ref api/db customer-ref1 org-id :type)
 get-by-ref api/db customer-ref2 org-id :type)
 (= customer-ref1 customer-ref2)]

I'm trying to find the relevant parts of xtdb code where it does the comparisons. Any pointers?

https://github.com/xtdb/xtdb/blob/master/core/src/xtdb/codec.clj#L160 is a bit hairy 😕

I have a minimal sample now, though I need to remove all the cruft (tomorrow). The key finding is this:

(comment
  (xt/submit-tx node [[::xt/put {:xt/id 1 :ref {:key :val :int 1}}]]))

(comment
  (xt/q (xt/db node)
        '{:find [e]
          :where [[e :ref ref]]
          :in [ref]}
        {:key :val :int 1}))

(comment
  (xt/q (xt/db node)
        '{:find [e]
          :where [[e :ref ref]]
          :in [ref]}
        {:key :val :int 
         (jsonista/read-value (jsonista/write-value-as-string 1))}))

First query works, second doesn't. Also if the values are plain ints, not inside a map, even the jsonista version works

And this hits us, because we get data from external sources as json

ah 😕 there must be something lossy happening with Jackson

and/or Nippy isn't detecting the types correctly

is this with the latest jsonista version and xt versions?

so is this the cause?

user> (type (jsonista.core/read-value "1"))
java.lang.Integer
user> (type 1)
java.lang.Long

but they don’t serialize the same

xtdb.codec> (with-open [out (java.io.ByteArrayOutputStream.)] (freeze-map (java.io.DataOutputStream. out) {:n (int 1)}) (.toByteArray out))
[112, 1, 106, 1, 110, 42, 0, 0, 0, 1]
xtdb.codec> (with-open [out (java.io.ByteArrayOutputStream.)] (freeze-map (java.io.DataOutputStream. out) {:n  1}) (.toByteArray out))
[112, 1, 106, 1, 110, 100, 1]

Sounds very plausible

if you coerce the number returned by jsonista by calling long on it, does that query then work?

Have to do the opposite, but yes. Coercing to same integer types will help

But that's only needed when the value is inside a map. Plain int and longs seem to be interchangeable in XTDB queries

Pretty tricky situation. https://github.com/xtdb/xtdb/blob/master/core/src/xtdb/codec.clj#L231 seems to coerce ints to longs, but changing the serialization of maps is bound to cause headaches too.

I guess there’s no choice but to normalize the maps to either form in application code

Yeah. What a footgun

And to be a bit safer, I suppose that deep map normalization needs to do all the same coercions xtdb does for plain values, not just ints. So floats to doubles, bigints to bigintegers and so on

I appreciate you both digging in to figure that out, thank you! Agreed this is rather subtle footgun - the team will review and reflect. In the meantime I have filed an issue and will keep you posted https://github.com/xtdb/xtdb/issues/2528

Got it, thanks!