correction, I can retrieve it with xt/entity on the same snapshot, but not with xt/q

(it's the in memory database)

It looks like it is working fine with LMDB, at this point I suspect an "in memory" indexing issue here

Hey @U0C8489U6 that sounds strange, and definitely something I'd like to get to the bottom of if you have the time/patience. What am I missing from this attempt at a repro gist https://gist.github.com/refset/c8116f0862abaa94db7752af540549d5 ?

Looks good to me...but weird here it was not working...I have tried the empty map in order to by pass the config..thanks for the repro..I'll have to see what is going on here

@U899JBRPFcould you please try to do the xt/q query in a separate function call with a new xt/db passed in? I know it sounds silly but that's the only difference I see

I was also trying to get all the enties with just

:where [[?e :cohesic/type :measurement]]

I tried myself, thanks for the repro file, and...I cannot repro with such setup

Another funny thing that happens here is that

(clojure.pprint/pprint
     (xt/entity (xt/db xtdb-node) (get-in entities [0 :xt/id])))

(clojure.pprint/pprint
     (xt/entity (xt/db xtdb-node) {:measurement-id "left-ventricle.end-diastolic-volume",
                                   :diagnostic-report-version-id "8aae221f-7c0d-439a-9d86-5305078960c8"}))

(prn (= (get-in entities [0 :xt/id])
            {:measurement-id "left-ventricle.end-diastolic-volume",
             :diagnostic-report-version-id "8aae221f-7c0d-439a-9d86-5305078960c8"}))

Note that if I use a "normal" :xt/id (even a str of that map) - I see the entity in all the queries

Ok, new development, if I hard-code the tx-ops vector it seems to work

(like in your repro)

ok I put them side by side

[[:xtdb.api/put
  {:cohesic/type :measurement,
   :measurement/measurement-id "left-ventricle.end-diastolic-volume",
   :measurement/diagnostic-report-id
   "15718872-f75a-4a3e-911b-570b84ad97ed",
   :measurement/value [1.0 2.0 3.0],
   :xt/id
   {:measurement-id "left-ventricle.end-diastolic-volume",
    :diagnostic-report-version-id
    "53cfbb74-beea-4630-b729-c0172591ee99"}}]
 [:xtdb.api/put
  {:cohesic/type :measurement,
   :measurement/measurement-id "left-ventricle.end-diastolic-volume",
   :xt/id
   {:measurement-id "left-ventricle.end-diastolic-volume",
    :diagnostic-report-version-id
    "8aae221f-7c0d-439a-9d86-5305078960c8"},
   :measurement/diagnostic-report-id
   "21d45db0-1213-479d-92b1-81bf954ec148",
   :measurement/value [1.0 2.0 3.0]}]]

for some reason I re-eval-ed the namespace that does the transformation before sending tx-ops to XT and it started to work (!?!)

do you have Clojure metadata on the ID map?

uhm let me check

I think Nippy will serialize Clojure metadata by default, and I'm not sure XT does anything to strip it out (either on the way in, or out)

https://github.com/xtdb/xtdb/blob/a6e1e32e8814270f0d47acf3f6c705c9ae8fd195/core/src/xtdb/codec.clj#L611-L614

(Spamming related links for reference, sorry!) https://github.com/xtdb/xtdb/issues/1510

it looks like I have some meta

{:file file:/home/cokap/git/cohesic/acuity/appserver/src/com/cohesic/acuity/infra2/interpretation/data_models/measurement.clj, :line 39, :column 17, :end-line 42, :end-column 85}

would that be the issue, should I manually strip it out before-hand?

(I never had to do anything for the other entities...I wonder why this is special...)

Yep, that is almost certainly what you need to do, unless you can think of a way in which this is useful for your use case

It was arguably a mistake to not strip out metadata aggressively during submission, but I think that ship may have sailed. I don't have that #1510 issue booted up... feel free to chime in on that thread though (if this does seem to be the problem, anyway)

Sorry you hit this 😕 I'll add a warning to the docs

Ah, I already did 😅 https://github.com/xtdb/xtdb/commit/faebb74cc7ab5882d19b9afc8033e319a6e9efd8

😄

I think this meta is only happening because of REPL ?

cause the app seems to work fine when running

hmm, quite possibly! which editor/repl plugin are you using?

well it's cider 😄

oh no 🙈 cc @U050V1N74 (for escalation)

which version?

latest cider on master, but let me see if deleting the meta (I guess nil suffices) solves

Cider 1.4.1 (Kyiv)

oh wow, that solved

with specter we do now

(sp/multi-transform
         (sp/multi-path [(sp/must :measurement/value) sp/ALL (sp/terminal numbers/some-double)]
                        [(sp/must :xt/id) sp/META (sp/terminal sp/NONE)]))

oh neat, that looks powerful

FWIW, this happens in kaocha tests as well, which instrument the vars

(depending on the implementation of course)

it might be worth removing metadata from throughout the entire document body, if the machinery is already to hand

@U03219RPYTS @U01PAJR4N69 we finally have a winner here :D

Thanks for your help on this Is there a way to hook an automatic transformation that does this on ingestion? As a workaround before it will be addressed on the XT side ..

ok sorry, I read that issue and commented there the repro repo is there and it mimics your gist above https://github.com/xtdb/xtdb/issues/1510#issuecomment-1142604153

what an informative thread, good to know... I never use metadata myself but if tools do it for you, this could be a problem. Was it really the newer CIDER version causing it? I'm on CIDER 1.2.0 (Nice) and haven't noticed any problems with map ids

indeed it's a terrible issue :( I run into it often since I think :xt/id should always be a map. haven't had an issue with cider yet though

I finally got rid of it by generating docs through a malli-powered validation system

I'm starting to feel that metadata as an invisible "side channel" that can hide info is a bad thing

you start to question your own sanity when 2 pieces of data that look similar, behave differently... but it is interesting that it is serialized, as normal edn/transit doesn't

@U11SJ6Q0K @U797MAJ8M we have a colleague on CIDER 1.3 and no issue was detected so definitely something new in CIDER 1.4