looks like that call should have been added to both, yep 😕 will add it in. the api/conform-tx-ops is to convert Java types into Clojure types - the impact of not doing it should (as far as I can see) be limited to differing behaviour if Java Maps/Lists are submitted. does that match with what you're seeing?

seems like it must be the case, but the error was baffling: crux.IllegalArgumentException: invalid tx-op: Unfreezable type: class jdk.internal.net.http.HttpClientFacade and the :op it showed was a totally innocuous looking crux.tx/put with a legit doc (clojure map, no weird java objects) and a valid time. no idea why calling it seems to help or how a HttpClientFacade could have gotten in there.btw, could conform-tx-ops be written without the (map vec)? as in

(defn conform-tx-ops [tx-ops]   (mapv (fn [tx-op]           (mapv #(if (instance? Map %) (into {} %) %)                 tx-op))         tx-ops))

I wonder if there was metadata attached, actually..

yup.. the doc was being returned from a pathom resolver which attaches metadata to it

ah, good find. we've had similar issues with metadata on objects in the past - Nippy serialises metadata, which then impacts our content hashing. I'll look out the history/context of that issue and see what we can do

IIRC we had to maintain this behaviour as it'd impact the hashes of existing documents persisted in the document store

well, as it stands then the doc hashes would be different depending on which api the doc was submitted through

though.. I didn't think there was a way for crux to modify the doc store aside from appending and evicting

have pushed a change to master that means that submit-tx just calls through to submit-tx-async + deref 🙂

@U797MAJ8M: (WIP) a first pass at making Crux's transaction submission more async-friendly: https://github.com/juxt/crux/pull/1476

amazing stuff! I had just complained to refset that submit-tx-async was still blocking too 🙂

the doc store hanging change also seems very nice.. my understanding is that q/entity will now allow for missing docs (leaving it up to the user to handle), but only the indexer requires fetch-docs to succeed or blow up? that would be the perfect time (turning up a new node) to discover and handle any data loss

Just in case this should work

(-> "deps.edn"
      slurp
      read-string
      :deps)
=> {juxt/crux-core #:mvn{:version "21.02-1.15.0-beta"},
 juxt/crux-lmdb #:mvn{:version "21.02-1.15.0-alpha"},
 org.clojure/data.csv #:mvn{:version "1.0.0"},
 io.randomseed/phone-number #:mvn{:version "8.12.16-1"},
 ch.qos.logback/logback-classic #:mvn{:version "1.2.3"}}

;start-node config file
(-> "resources/config.edn"
      slurp
      read-string)
=> #:crux{:index-store {:kv-store {:crux/module crux.lmdb/->kv-store, :db-dir "/tmp/lmdb/index"}},
       :document-store {:kv-store {:crux/module crux.lmdb/->kv-store, :db-dir "/tmp/lmdb/document"}},
       :tx-log {:kv-store {:crux/module crux.lmdb/->kv-store, :db-dir "/tmp/lmdb/tx"}}}

thanks - could you also post the full document you've submitted?

(crux/q (crux/db node) '{:find [e] :where [[e :person/ssn a]] :full-results? true}) => #{[{:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}]}

Also here is the entity history if it helps

[{:crux.tx/tx-time #inst "2021-03-17T06:37:20.712-00:00",
  :crux.tx/tx-id 67,
  :crux.db/valid-time #inst "2021-03-17T06:37:20.712-00:00",
  :crux.db/content-hash #crux/id "2cebc1f95bc91a64ff325d56d23fe84c51a7462e",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T06:40:39.739-00:00",
  :crux.tx/tx-id 72,
  :crux.db/valid-time #inst "2021-03-17T06:40:39.739-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T06:41:31.887-00:00",
  :crux.tx/tx-id 73,
  :crux.db/valid-time #inst "2021-03-17T06:41:31.887-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T06:44:49.968-00:00",
  :crux.tx/tx-id 74,
  :crux.db/valid-time #inst "2021-03-17T06:44:49.968-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:32:49.860-00:00",
  :crux.tx/tx-id 83,
  :crux.db/valid-time #inst "2021-03-17T07:32:49.860-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:34:45.882-00:00",
  :crux.tx/tx-id 84,
  :crux.db/valid-time #inst "2021-03-17T07:34:45.882-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:35:33.747-00:00",
  :crux.tx/tx-id 85,
  :crux.db/valid-time #inst "2021-03-17T07:35:33.747-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:40:04.698-00:00",
  :crux.tx/tx-id 89,
  :crux.db/valid-time #inst "2021-03-17T07:40:04.698-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}
 {:crux.tx/tx-time #inst "2021-03-17T07:42:56.245-00:00",
  :crux.tx/tx-id 93,
  :crux.db/valid-time #inst "2021-03-17T07:42:56.245-00:00",
  :crux.db/content-hash #crux/id "9f530a74ef94d8b58027e2b90525875b27f8e8a7",
  :crux.db/doc {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}}]

thanks 🙂 fwiw it works for me starting a fresh node, so there's something else here for us to figure out.

let's try removing the eql/project from the situation - what do these two return?

(crux/q
   (crux/db node)
   '{:find [e]
     :where [[e :person/ssn v]]})

(crux/q
   (crux/db node)
   '{:find [e]
     :where [[e :union.membership/number v]]})

I have additional entities in the db now, but when I tried those queries I got empty result set on latter.

(crux/q
   (crux/db node)
   '{:find [e]
     :where [[e :person/ssn v]]})
=> #{[#uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7"] [#uuid "4e3d2554-80b1-4a0a-abe6-5b8dfabe6d3a"]}

(crux/q
   (crux/db node)
   '{:find [e]
     :where [[e :union.membership/number v]]})
=> #{[#uuid "4e3d2554-80b1-4a0a-abe6-5b8dfabe6d3a"]}

; the missing entity
(crux/entity (crux/db node) #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7")
=> {:person/ssn "1234", :person/first-name "tuomas", :union.membership/number 1234, :crux.db/id #uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7", :person.branch/id #uuid "e8653f50-a1bc-4881-bdd2-504b5de4a333"}

I am just learning how to use crux and did my first temporal query to demonstrate the queries from past. This is so cool

(crux/q
   (crux/db node {:crux.tx/tx-time #inst "2021-03-17T07:42:56.245-00:00"})
   '{:find [e]
     :where [[e :person/ssn v]]})
=> #{[#uuid "1a23bcbf-f0be-492b-8ced-d254ebb60df7"]}

(crux/q
   (crux/db node {:crux.tx/tx-time #inst "2021-03-17T07:42:56.245-00:00"})
   '{:find [e]
     :where [[e :union.membership/number v]]})
=> #{}

> This is so cool 🙂 I can't immediately see why that's not returning you a result, though. If it's a non-sensitive dataset, would you mind sending your query indices through to <mailto:[email protected]|[email protected]>? (`/tmp/lmdb/index`, from your config above)

thanks 🙂 sorry to be a pain, could you send through the other two directories in there too? (`/tmp/lmdb/document` and /tmp/lmdb/tx) (I thought it'd be something in the indices, but turns out that's not enough to narrow it down)

Yes I'll send them

And you're not a pain. I feel fortunate to have come across a bug so I can help make crux better!

fyi, i've built up 2,15GB of query indices, 127MB of transaction log and 449MB of documents. Do the older indices work or do you need the new ones?

not bad 🙂 the older indices did show the problem but not enough information to figure out the cause, if you still have those around that'd be great!