Fork me on GitHub
#xtdb
<
2021-03-17
>
nivekuil06:03:45

in submit-tx, api/conform-tx-ops is being called: https://github.com/juxt/crux/blob/master/crux-core/src/crux/node.clj#L196 but this is not called in submit-tx-async 30 lines below. I have some custom submit code that followed the submit-tx-async logic and it seems like adding the api/conform-tx-ops call as in submit-tx fixed a weird bug.. not sure of the intention there. is the tx-ops param of submit-tx-async different from that of submit-tx, namely being conformed beforehand?

jarohen10:03:33

looks like that call should have been added to both, yep 😕 will add it in. the api/conform-tx-ops is to convert Java types into Clojure types - the impact of not doing it should (as far as I can see) be limited to differing behaviour if Java Maps/Lists are submitted. does that match with what you're seeing?

nivekuil11:03:56

seems like it must be the case, but the error was baffling: crux.IllegalArgumentException: invalid tx-op: Unfreezable type: class jdk.internal.net.http.HttpClientFacade and the :op it showed was a totally innocuous looking crux.tx/put with a legit doc (clojure map, no weird java objects) and a valid time. no idea why calling it seems to help or how a HttpClientFacade could have gotten in there.btw, could conform-tx-ops be written without the (map vec)? as in

(defn conform-tx-ops [tx-ops]   (mapv (fn [tx-op]           (mapv #(if (instance? Map %) (into {} %) %)                 tx-op))         tx-ops))

nivekuil11:03:12

I wonder if there was metadata attached, actually..

nivekuil11:03:23

yup.. the doc was being returned from a pathom resolver which attaches metadata to it

jarohen14:03:55

ah, good find. we've had similar issues with metadata on objects in the past - Nippy serialises metadata, which then impacts our content hashing. I'll look out the history/context of that issue and see what we can do

jarohen14:03:59

IIRC we had to maintain this behaviour as it'd impact the hashes of existing documents persisted in the document store

nivekuil21:03:27

well, as it stands then the doc hashes would be different depending on which api the doc was submitted through

nivekuil21:03:02

though.. I didn't think there was a way for crux to modify the doc store aside from appending and evicting

jarohen19:03:04

have pushed a change to master that means that submit-tx just calls through to submit-tx-async + deref 🙂

👍 3
jarohen18:03:23

@U797MAJ8M: (WIP) a first pass at making Crux's transaction submission more async-friendly: https://github.com/juxt/crux/pull/1476

❤️ 3
nivekuil23:03:22

amazing stuff! I had just complained to refset that submit-tx-async was still blocking too 🙂

nivekuil23:03:46

the doc store hanging change also seems very nice.. my understanding is that q/entity will now allow for missing docs (leaving it up to the user to handle), but only the indexer requires fetch-docs to succeed or blow up? that would be the perfect time (turning up a new node) to discover and handle any data loss

Tuomas08:03:04

What am I missing/misunderstanding here? I have an entity with at least 2 attributes and I can find it using the first one, but not the second one.

(crux/q
   (crux/db node)
   '{:find [(eql/project e [:person/ssn :union.membership/number])]
     :where [[e :person/ssn v]]})
=> #{[{:person/ssn "1234", :union.membership/number 1234}]}

(crux/q
   (crux/db node)
   '{:find [(eql/project e [:person/ssn :union.membership/number])]
     :where [[e :union.membership/number v]]})
=> #{}

Tuomas08:03:03

Just in case this should work

(-> "deps.edn"
      slurp
      read-string
      :deps)
=> {juxt/crux-core #:mvn{:version "21.02-1.15.0-beta"},
 juxt/crux-lmdb #:mvn{:version "21.02-1.15.0-alpha"},
 org.clojure/data.csv #:mvn{:version "1.0.0"},
 io.randomseed/phone-number #:mvn{:version "8.12.16-1"},
 ch.qos.logback/logback-classic #:mvn{:version "1.2.3"}}

;start-node config file
(-> "resources/config.edn"
      slurp
      read-string)
=> #:crux{:index-store {:kv-store {:crux/module crux.lmdb/->kv-store, :db-dir "/tmp/lmdb/index"}},
       :document-store {:kv-store {:crux/module crux.lmdb/->kv-store, :db-dir "/tmp/lmdb/document"}},
       :tx-log {:kv-store {:crux/module crux.lmdb/->kv-store, :db-dir "/tmp/lmdb/tx"}}}

jarohen09:03:40

thanks - could you also post the full document you've submitted?

Tuomas09:03:04

(crux/q (crux/db node) '{:find [e] :where [[e :person/ssn a]] :full-results? true}) => #{[{:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}]}

Tuomas09:03:30

Also here is the entity history if it helps

[{:crux.tx/tx-time #inst "2021-03-17T06:37:20.712-00:00",
  :crux.tx/tx-id 67,
  :crux.db/valid-time #inst "2021-03-17T06:37:20.712-00:00",
  :crux.db/content-hash #crux/id "2cebc1f95bc91a64ff325d56d23fe84c51a7462e",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T06:40:39.739-00:00",
  :crux.tx/tx-id 72,
  :crux.db/valid-time #inst "2021-03-17T06:40:39.739-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T06:41:31.887-00:00",
  :crux.tx/tx-id 73,
  :crux.db/valid-time #inst "2021-03-17T06:41:31.887-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T06:44:49.968-00:00",
  :crux.tx/tx-id 74,
  :crux.db/valid-time #inst "2021-03-17T06:44:49.968-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:32:49.860-00:00",
  :crux.tx/tx-id 83,
  :crux.db/valid-time #inst "2021-03-17T07:32:49.860-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:34:45.882-00:00",
  :crux.tx/tx-id 84,
  :crux.db/valid-time #inst "2021-03-17T07:34:45.882-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:35:33.747-00:00",
  :crux.tx/tx-id 85,
  :crux.db/valid-time #inst "2021-03-17T07:35:33.747-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:40:04.698-00:00",
  :crux.tx/tx-id 89,
  :crux.db/valid-time #inst "2021-03-17T07:40:04.698-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:42:56.245-00:00",
  :crux.tx/tx-id 93,
  :crux.db/valid-time #inst "2021-03-17T07:42:56.245-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}]

jarohen10:03:37

thanks 🙂 fwiw it works for me starting a fresh node, so there's something else here for us to figure out.

jarohen10:03:40

let's try removing the eql/project from the situation - what do these two return?

(crux/q
   (crux/db node)
   '{:find [e]
     :where [[e :person/ssn v]]})

(crux/q
   (crux/db node)
   '{:find [e]
     :where [[e :union.membership/number v]]})

Tuomas10:03:55

I have additional entities in the db now, but when I tried those queries I got empty result set on latter.

(crux/q
   (crux/db node)
   '{:find [e]
     :where [[e :person/ssn v]]})
=> #{[#uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7"] [#uuid "4e3d2554-80b1-4a0a-abe6-5b8dfabe6d3a"]}

(crux/q
   (crux/db node)
   '{:find [e]
     :where [[e :union.membership/number v]]})
=> #{[#uuid "4e3d2554-80b1-4a0a-abe6-5b8dfabe6d3a"]}

; the missing entity
(crux/entity (crux/db node) #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7")
=> {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}
Seems like just that entity is bugged on that one attribute

Tuomas10:03:48

I am just learning how to use crux and did my first temporal query to demonstrate the queries from past. This is so cool

(crux/q
   (crux/db node {:crux.tx/tx-time #inst "2021-03-17T07:42:56.245-00:00"})
   '{:find [e]
     :where [[e :person/ssn v]]})
=> #{[#uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7"]}

(crux/q
   (crux/db node {:crux.tx/tx-time #inst "2021-03-17T07:42:56.245-00:00"})
   '{:find [e]
     :where [[e :union.membership/number v]]})
=> #{}

jarohen10:03:23

> This is so cool 🙂 I can't immediately see why that's not returning you a result, though. If it's a non-sensitive dataset, would you mind sending your query indices through to <mailto:[email protected]|[email protected]>? (`/tmp/lmdb/index`, from your config above)

👌 3
jarohen13:03:26

thanks 🙂 sorry to be a pain, could you send through the other two directories in there too? (`/tmp/lmdb/document` and /tmp/lmdb/tx) (I thought it'd be something in the indices, but turns out that's not enough to narrow it down)

Tuomas04:03:36

Yes I'll send them

Tuomas04:03:25

And you're not a pain. I feel fortunate to have come across a bug so I can help make crux better!

Tuomas05:03:00

fyi, i've built up 2,15GB of query indices, 127MB of transaction log and 449MB of documents. Do the older indices work or do you need the new ones?

jarohen09:03:13

not bad 🙂 the older indices did show the problem but not enough information to figure out the cause, if you still have those around that'd be great!