Fork me on GitHub
#xtdb
<
2022-05-30
>
richiardiandrea22:05:44

Hi there, I have an issue with put-ing the following payload

{:cohesic/type :measurement,
 :measurement/measurement-id "left-ventricle.end-diastolic-volume",
 :measurement/diagnostic-report-id
 "15718872-f75a-4a3e-911b-570b84ad97ed",
 :measurement/value [1.0 2.0 3.0],
 :xt/id
 {:measurement-id "left-ventricle.end-diastolic-volume",
  :diagnostic-report-version-id "53cfbb74-beea-4630-b729-c0172591ee99"}}
I have been trying to save it and retrieve it, with an await-tx in between the two so that I give time to XT to sync but for some reason the get never sees it...I am able to retrieve it from the same (xt/db node) snapshot but not from another snapshot (at the same tx-time..). How would I go to try and debug that?

richiardiandrea22:05:28

correction, I can retrieve it with xt/entity on the same snapshot, but not with xt/q

richiardiandrea22:05:39

(it's the in memory database)

richiardiandrea22:05:37

I found a test with xt/entity and that's what I see working https://github.com/xtdb/xtdb/commit/dd03a3613f9a076116d87ca0500f2350302ecec8 But what if I wanted all entities given a top level key for them?

richiardiandrea23:05:33

It looks like it is working fine with LMDB, at this point I suspect an "in memory" indexing issue here

refset07:05:19

Hey @U0C8489U6 that sounds strange, and definitely something I'd like to get to the bottom of if you have the time/patience. What am I missing from this attempt at a repro gist https://gist.github.com/refset/c8116f0862abaa94db7752af540549d5 ?

richiardiandrea15:05:24

Looks good to me...but weird here it was not working...I have tried the empty map in order to by pass the config..thanks for the repro..I'll have to see what is going on here

richiardiandrea15:05:14

@U899JBRPFcould you please try to do the xt/q query in a separate function call with a new xt/db passed in? I know it sounds silly but that's the only difference I see

richiardiandrea15:05:58

I was also trying to get all the enties with just

:where [[?e :cohesic/type :measurement]]
only

richiardiandrea15:05:48

I tried myself, thanks for the repro file, and...I cannot repro with such setup

richiardiandrea16:05:13

Another funny thing that happens here is that

(clojure.pprint/pprint
     (xt/entity (xt/db xtdb-node) (get-in entities [0 :xt/id])))
works but the hard-coded
(clojure.pprint/pprint
     (xt/entity (xt/db xtdb-node) {:measurement-id "left-ventricle.end-diastolic-volume",
                                   :diagnostic-report-version-id "8aae221f-7c0d-439a-9d86-5305078960c8"}))
Does not and
(prn (= (get-in entities [0 :xt/id])
            {:measurement-id "left-ventricle.end-diastolic-volume",
             :diagnostic-report-version-id "8aae221f-7c0d-439a-9d86-5305078960c8"}))
is true

richiardiandrea16:05:42

Note that if I use a "normal" :xt/id (even a str of that map) - I see the entity in all the queries

richiardiandrea17:05:42

Ok, new development, if I hard-code the tx-ops vector it seems to work

richiardiandrea17:05:53

(like in your repro)

richiardiandrea17:05:35

ok I put them side by side

[[:xtdb.api/put
  {:cohesic/type :measurement,
   :measurement/measurement-id "left-ventricle.end-diastolic-volume",
   :measurement/diagnostic-report-id
   "15718872-f75a-4a3e-911b-570b84ad97ed",
   :measurement/value [1.0 2.0 3.0],
   :xt/id
   {:measurement-id "left-ventricle.end-diastolic-volume",
    :diagnostic-report-version-id
    "53cfbb74-beea-4630-b729-c0172591ee99"}}]
 [:xtdb.api/put
  {:cohesic/type :measurement,
   :measurement/measurement-id "left-ventricle.end-diastolic-volume",
   :xt/id
   {:measurement-id "left-ventricle.end-diastolic-volume",
    :diagnostic-report-version-id
    "8aae221f-7c0d-439a-9d86-5305078960c8"},
   :measurement/diagnostic-report-id
   "21d45db0-1213-479d-92b1-81bf954ec148",
   :measurement/value [1.0 2.0 3.0]}]]
The first is the one in your repro, which appears in the DB, the second one does not (!?!)

richiardiandrea19:05:05

for some reason I re-eval-ed the namespace that does the transformation before sending tx-ops to XT and it started to work (!?!)

thinking-face 1
refset19:05:08

do you have Clojure metadata on the ID map?

richiardiandrea19:05:49

uhm let me check

refset19:05:02

I think Nippy will serialize Clojure metadata by default, and I'm not sure XT does anything to strip it out (either on the way in, or out)

refset19:05:40

(Spamming related links for reference, sorry!) https://github.com/xtdb/xtdb/issues/1510

❤️ 1
richiardiandrea19:05:24

it looks like I have some meta

{:file file:/home/cokap/git/cohesic/acuity/appserver/src/com/cohesic/acuity/infra2/interpretation/data_models/measurement.clj, :line 39, :column 17, :end-line 42, :end-column 85}

richiardiandrea19:05:52

would that be the issue, should I manually strip it out before-hand?

richiardiandrea19:05:12

(I never had to do anything for the other entities...I wonder why this is special...)

refset19:05:18

Yep, that is almost certainly what you need to do, unless you can think of a way in which this is useful for your use case

👍 1
refset19:05:03

It was arguably a mistake to not strip out metadata aggressively during submission, but I think that ship may have sailed. I don't have that #1510 issue booted up... feel free to chime in on that thread though (if this does seem to be the problem, anyway)

❤️ 1
refset19:05:36

Sorry you hit this 😕 I'll add a warning to the docs

richiardiandrea19:05:25

I think this meta is only happening because of REPL ?

richiardiandrea19:05:38

cause the app seems to work fine when running

refset19:05:23

hmm, quite possibly! which editor/repl plugin are you using?

richiardiandrea19:05:43

well it's cider 😄

refset19:05:24

oh no 🙈 cc @U050V1N74 (for escalation)

👀 1
refset19:05:29

which version?

richiardiandrea19:05:59

latest cider on master, but let me see if deleting the meta (I guess nil suffices) solves

richiardiandrea19:05:17

Cider 1.4.1 (Kyiv)

👍 1
richiardiandrea20:05:20

oh wow, that solved

🙌 1
richiardiandrea20:05:54

with specter we do now

(sp/multi-transform
         (sp/multi-path [(sp/must :measurement/value) sp/ALL (sp/terminal numbers/some-double)]
                        [(sp/must :xt/id) sp/META (sp/terminal sp/NONE)]))

refset20:05:22

oh neat, that looks powerful

richiardiandrea20:05:28

FWIW, this happens in kaocha tests as well, which instrument the vars

👍 1
richiardiandrea20:05:39

(depending on the implementation of course)

refset20:05:01

it might be worth removing metadata from throughout the entire document body, if the machinery is already to hand

richiardiandrea20:05:01

@U03219RPYTS @U01PAJR4N69 we finally have a winner here :D

🙌 2
richiardiandrea20:05:30

Thanks for your help on this Is there a way to hook an automatic transformation that does this on ingestion? As a workaround before it will be addressed on the XT side ..

richiardiandrea20:05:43

ok sorry, I read that issue and commented there the repro repo is there and it mimics your gist above https://github.com/xtdb/xtdb/issues/1510#issuecomment-1142604153

🙏 1
tatut07:06:21

what an informative thread, good to know... I never use metadata myself but if tools do it for you, this could be a problem. Was it really the newer CIDER version causing it? I'm on CIDER 1.2.0 (Nice) and haven't noticed any problems with map ids

nivekuil08:06:28

indeed it's a terrible issue :( I run into it often since I think :xt/id should always be a map. haven't had an issue with cider yet though

😞 2
nivekuil08:06:58

I finally got rid of it by generating docs through a malli-powered validation system

tatut09:06:43

I'm starting to feel that metadata as an invisible "side channel" that can hide info is a bad thing

tatut09:06:42

you start to question your own sanity when 2 pieces of data that look similar, behave differently... but it is interesting that it is serialized, as normal edn/transit doesn't

👍 2
richiardiandrea17:06:21

@U11SJ6Q0K @U797MAJ8M we have a colleague on CIDER 1.3 and no issue was detected so definitely something new in CIDER 1.4