Hello! :) Yes you can definitely migrate without losing data. There have been a couple of changes since then that you might need to account for though: 1) eviction no longer works at a document level 2) put and delete ops with a "valid time end" argument behave differently If you weren't using either of those features then you should be okay to migrate without doing any manual transformations. The simplest method is to read the successful transactions out into a file via tx-log and then submit them back into the new jdbc node. Worth noting that this won't preserve transaction times. I don't have a code snippet handy right now but I'll give it a go tomorrow!

You can validate it by checking the attribute-stats output matches across both versions

hey! thanks for the help. It's exclusively put txs so it's fine. It's also a toy project which just happened to run for quite a while now and I would be sad losing all it had collected.

what I did was

(let [data (crux/tx-log sys (crux/new-tx-log-context sys) nil true)]
      (clojure.pprint/pprint data ( "data.edn")))

and with a bit of hacking putting it back:

(doall
      (for [tx (clojure.edn/read-string (slurp "data.edn"))]
        (let [tx' (:crux.api/tx-ops tx)
              tx'' (mapv (fn [[tx _ data]] [tx data]) tx')]
          (when (seq tx'')
            tx''
            (crux/submit-tx sys tx'')))))

I have just a small offset in attribute stats which I'm not sure why is happening but it's an increase so it's probably fine 🙂

from

{:kino.album/artist-ids 38,
 :kino.album/release-date 36,
 :kino.track/artist-ids 53,
 :kino.album/images 108,
 :kino.album/total-tracks 36,
 :kino.album/name 36,
 :kino.track/number 47,
 :type 1,
 :kino.play/played-at 50,
 :kino.play/user-id 50,
 :kino.user/refresh-token 1,
 :kino.play/track-id 50,
 :kino.artist/name 42,
 :kino.track/explicit 47,
 :kino.track/name 47,
 :crux.db/id 176,
 :display_name 1,
 :kino.track/album-id 47}

I'm getting to

{:kino.album/artist-ids 41,
 :kino.album/release-date 39,
 :kino.track/artist-ids 55,
 :kino.album/images 117,
 :kino.album/total-tracks 39,
 :kino.album/name 39,
 :kino.track/number 49,
 :type 1,
 :kino.play/played-at 50,
 :kino.play/user-id 50,
 :kino.user/refresh-token 1,
 :kino.play/track-id 50,
 :kino.artist/name 41,
 :kino.track/explicit 49,
 :kino.track/name 49,
 :crux.db/id 180,
 :display_name 1,
 :kino.track/album-id 49}

in any case I'm not going crazy over this as I can see my data are there

thanks so much again! Take care!

btw I just tried the above with the existing version but I guess it will work when I upgrade too

Nice! That difference is weird though...if you're happy to live not knowing, then no problem, but equally I can definitely help get to the bottom of it 🙂

I don't see why that script would be doing anything weird

well, I thought I could ... but know having someone willing to help I cannot 🙂 Should I grab the tx logs for a single key like :kino.track/name and see the diff before and after? Or is there a more efficient way to tackle this?

Not sure if the following makes sense:

the status for the following attr is:

:kino.track/name 49

but this:

(count
      (crux/q
        (crux/db (-> system :db :db))
        '{:find [e]
          :where [[e :kino.artist/name ?]]}))

gives 37

not sure if this is expected

also, checking the history for each of those docs:

(->>
      (crux/q (crux/db (-> system :db :db))
        '{:find [e]
          :where [[e :kino.artist/name ?]]})
      (map first)
      (map #(crux/history sys %)))

also gives me a single history records for each one of those - I guess that expected as this is after the import

Ahhh, that's even weirder then! Good idea to check that the history of each only contains one entry. It's possible that there's something we've changed with ingestion that's caused attribute-stats to go haywire, like a race condition of some kind (it's run async after the main index-tx thread runs). I will try to repro on a vanilla data set.

Please note this is on the older version

I'll give this a go too on the latest version

hey again! I gave it a go 20.04-1.8.1-alpha and attribute-stats were consistent this time. Before and after the export and it matched the db key counts I did. This is with jdbc backend tho. Will try now with rocks.

{:crux.node/topology '[crux.standalone/topology
                                                  crux.kv.rocksdb/kv-store]
                            :crux.kv/db-dir "data"}

everything looks all right with this config as well, so I guess we can close the file on this one 🙂

as this was a fun process I'm thinking of writing a blog post. Someone may find this useful.

Cool, okay well I'm glad you're happy :) and a blog post would be awesome! We'll crack on with some more rigorous migration tooling also 😊