xtdb 2020-07-19 | Slack Archive

Eric Ihli19:07:47

(set-log-level! 'crux.tx "warn")

  (let [tx (crux/await-tx
            conn
            (crux/submit-tx conn [[:crux.tx/put {:crux.db/id :foo :email ""}]]))
        committed? (crux/tx-committed? conn tx)]
    [committed? tx])
  ;; => [false #:crux.tx{:tx-time #inst "2020-07-18T23:05:00.005-00:00", :tx-id 18}]

  (def tx-log (crux/open-tx-log conn 0 true))
  ;; => #'com.owoga.template.dev/tx-log

  (iterator-seq tx-log)
  ;; => ({:crux.tx/tx-id 1,
  ;;      :crux.tx/tx-time #inst "2020-07-19T18:47:01.525-00:00",
  ;;      :crux.api/tx-ops
  ;;      ([:crux.tx/put {:crux.db/id :foo, :email ""}])}
  ;;     {:crux.tx/tx-id 4,
  ;;      :crux.tx/tx-time #inst "2020-07-19T18:54:12.882-00:00",
  ;;      :crux.api/tx-ops
  ;;      ([:crux.tx/put {:crux.db/id :foo, :email ""}])}
  ;;     {:crux.tx/tx-id 5,
  ;;      :crux.tx/tx-time #inst "2020-07-19T18:54:26.138-00:00",
  ;;      :crux.api/tx-ops
  ;;      ([:crux.tx/put {:crux.db/id :foo, :email ""}])})

  (crux/q (crux/db conn) '{:find [e]
                           :where [[e :crux.db/id _]]
                           :full-results? true})
  ;; => #{[nil]}

  (crux/q (crux/db conn) '{:find [e]
                           :where [[e :crux.db/id :foo]]})
  ;; => #{[:foo]}

  (crux/entity (crux/db conn) :foo)
  ;; => nil

I'm very confused by the sequence above. I first noticed that I wasn't able to retrieve some documents I submitted. I did an await-tx and tx-committed? to see if that is where the error was and I noticed it returning false. Then I realized my query wasn't actually returning an empty result. It was returning a nil value for full-results? true. And, I could find the eid if I queried specifically for the eid. But if I used crux/entity with the eid, I got nil. I then looked at the transaction log and saw the exact same document that is failing now was successfully committed 3 times previously. @taylor.jeremydavid, I set the the log level to warn as you suggested above, but I don't get any output when I run crux/submit-tx. I do get general log output, so I don't thing it's my logging config that is the issue (but it is possible).

12:23:13.020 [crux-tx-consumer] INFO crux.hash - Using java.security.MessageDigest for ID hashing.
...
12:33:30.347 [crux-tx-consumer] DEBUG crux.memory - :pool-allocation-stats {:allocated 786432, :deallocated 0, :in-use 786432}

Eric Ihli19:07:01

Ughhhh. I made an unfortunate mistake by accidentally deleting my data directory. On recreation, it is working fine. I assume my data directory was somehow corrupted and it would have been nice to have a copy of it around for further debugging.

refset13:07:39

I don't get why the transaction function warning wouldn't be firing [0]. Do you see any other warnings in the log? [0] https://github.com/juxt/crux/blob/1b2a05fde2b701a408b8ae5f84bdb5a9270aab0f/crux-core/src/crux/tx.clj#L199

refset13:07:10

Your issue with crux/entity not returning anything sounds like your haven't got :crux.standalone/event-log-dir configured and have restarted the node - can you post your topology?

Eric Ihli16:07:25

(crux/start-node {:crux.node/topology '[crux.standalone/topology
                                          crux.kv.rocksdb/kv-store]
                    :crux.kv/db-dir (str (io/file storage-dir "db"))}))

Eric Ihli16:07:34

Regarding the transaction function warning not firing: it was firing after I recreated the data directory and ran the same code, so I'm guessing when I was experiencing the issue the line with that log/warn wasn't getting hit.

refset16:07:00

ahh, yes that would make sense. Transaction functions should only get run once ever for standalone mode (whether they succeed or fail). You need to configure a :crux.standalone/event-log-dir directory to fix resolve your issues. Not having it configured means only your indexes are persisted, which means you can't meaningfully "restart" the node as all the source data is lost

🙏 3

2020-07-19

Channels