Fork me on GitHub
#xtdb
<
2020-07-10
>
Eric Ihli01:07:52

I just changed my deps.edn from 20.0.5... to 20.0.6... and lost everything in my database (RocksDB). Is that expected? It's just a toy project so not a big deal. But it was unexpected. I didn't think version bumping would have any effect on data. Worst I thought could happen would maybe get some errors thrown if there was a breaking change.

☝️ 3
🙏 3
Jacob O'Bryant05:07:11

I also ran into this. I've been holding off on upgrading in production until I can test if this is also an issue when using a postgres backend.

jarohen09:07:23

Hi - no, not expected beyond the 'index version bump' that we talked about in the release notes. Your golden stores (tx-log and document-store) should be unaffected by the release - deleting the KV store should then cause it to reindex using the new index structure.

jarohen10:07:13

Happy to pair up if it's unclear, understand that you'd want to be sure about what you're deleting (and certainly if Crux isn't behaving this way) - let me know 🙂

Jacob O'Bryant11:07:21

Deleting the kv store still doesn't cause crux to re-index for me. Does the configuration below look correct? I also tried this with 20.07-1.10.0-beta-SNAPSHOT and got the same results.

$ tree
.
├── deps.edn
└── src
    └── foo
        └── core.clj
$ cat deps.edn
{:aliases {:old {:extra-deps
                 {juxt/crux-core    {:mvn/version "20.05-1.8.4-alpha"}
                  juxt/crux-rocksdb {:mvn/version "20.05-1.8.4-alpha"}}}
           :new {:extra-deps
                 {juxt/crux-core    {:mvn/version "20.06-1.9.1-beta"}
                  juxt/crux-rocksdb {:mvn/version "20.06-1.9.1-beta"}}}}}
$ cat src/foo/core.clj
(ns foo.core
  (:require
    [crux.api :as crux]))

(defn -main [cmd]
  (let [node (crux/start-node
               {:crux.kv/db-dir "crux-db"
                :crux.node/topology '[crux.standalone/topology crux.kv.rocksdb/kv-store]
                :crux.standalone/event-log-dir "crux-event-log"
                :crux.standalone/event-log-kv-store 'crux.kv.rocksdb/kv})]
    (crux/sync node)
    (case cmd
      "prn" (prn (crux/entity (crux/db node) :foo))
      "tx" (crux/await-tx node
             (crux/submit-tx node [[:crux.tx/put {:crux.db/id :foo
                                                  :test "hello"}]])))
    (.close node)))
$ clj -A:old -m foo.core tx
$ clj -A:old -m foo.core prn
{:crux.db/id :foo, :test "hello"}
$ ls
crux-db  crux-event-log  deps.edn  src
$ rm -r crux-db/
$ clj -A:new -m foo.core prn
nil

jarohen11:07:52

Hey @U7YNGKDHA, thanks for these repros, very helpful 🙂 I've now identified what's causing this and hope to have a snapshot version released shortly, will keep you posted

jarohen14:07:45

I've released a fix as 20.07-1.9.2-beta-SNAPSHOT. The bug was introduced as part of the index changes in 1.9.0, and caused Crux >=1.9.0 to not be able to read standalone tx-logs from <1.9.0. (Kafka and JDBC were unaffected) Fortunately, the data in the <1.9.0 tx-logs wasn't corrupted in any way - the fix in this snapshot makes those transactions visible again. When you start a Crux node at this snapshot, you will see an IndexVersionOutOfSyncException - this is expected for upgrades over an index version bump. As above, you'll need to stop your node, delete your :crux.kv/db-dir and restart your node, which will cause Crux to re-index from its golden stores. Sorry for causing this issue, and thanks again for your patience - it's much appreciated

👍 3
Vincent Cantin06:07:07

I ran into a problem today. My transaction function threw an exception, then crux.api/entity was no longer returning my entities. Restarting the REPL did not solve the problem, but deleting the DB's store and restarting the REPL did.

Vincent Cantin06:07:13

I did not try to reproduce the problem, deadline is coming.

jarohen10:07:33

I'd be interested to see a repro for this one as and when you've got a moment - obviously understand you've got plenty of other demands on your time 🙂

Vincent Cantin07:07:21

I found in the tests of Crux many Datalog features (`and`, or, or-joint, transformations with variable binding, etc ..) which are not listed on the documentation of the website. Are those documented somewhere else?

Vincent Cantin07:07:51

It's not that I need this documentation, it's to know if it makes sense that I start a small gist to list those hidden features.

jarohen10:07:15

These are as yet undocumented because they're still quite experimental. On the understanding that they could change at any time, the best place to see how these work is in the crux.query-test at https://github.com/juxt/crux/blob/master/crux-test/test/crux/query_test.clj, particularly the datascript-test-* tests.

jarohen14:07:45

I've released a fix as 20.07-1.9.2-beta-SNAPSHOT. The bug was introduced as part of the index changes in 1.9.0, and caused Crux >=1.9.0 to not be able to read standalone tx-logs from <1.9.0. (Kafka and JDBC were unaffected) Fortunately, the data in the <1.9.0 tx-logs wasn't corrupted in any way - the fix in this snapshot makes those transactions visible again. When you start a Crux node at this snapshot, you will see an IndexVersionOutOfSyncException - this is expected for upgrades over an index version bump. As above, you'll need to stop your node, delete your :crux.kv/db-dir and restart your node, which will cause Crux to re-index from its golden stores. Sorry for causing this issue, and thanks again for your patience - it's much appreciated

👍 3
jarohen15:07:23

We've released v1.9.2 this afternoon - it's a bugfix release for the issue raised above, related to problems upgrading standalone Crux instances from <1.9.0. This issue was introduced as part of the index changes in 1.9.0 to reduce disk space and improve ingest/query performance. (Kafka and JDBC are unaffected, as are users upgrading directly from <1.9.0 to 1.9.2) Full details of the bug and the fix are in the release notes: https://github.com/juxt/crux/releases/tag/20.07-1.9.2 - we'd advise anybody running 1.9.0 or 1.9.1 to upgrade as soon as is practical. Particularly, you'll need to delete your :crux.kv/db-dir in order to re-index your Crux nodes, for the index-version bump in 1.9.0. There're a number of other smaller fixes included - we'll get in touch with the individual reporters shortly. Thanks to all involved for helping us to track it down, as well as your patience and understanding

👍 18