This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-04-09
Channels
- # announcements (1)
- # architecture (20)
- # aws (22)
- # babashka (41)
- # beginners (191)
- # chlorine-clover (66)
- # cider (19)
- # clj-kondo (54)
- # cljs-dev (15)
- # cljsrn (78)
- # clojars (1)
- # clojure (148)
- # clojure-android (4)
- # clojure-europe (7)
- # clojure-gamedev (15)
- # clojure-germany (6)
- # clojure-losangeles (46)
- # clojure-nl (23)
- # clojure-survey (3)
- # clojure-uk (46)
- # clojurescript (24)
- # conjure (21)
- # cursive (21)
- # data-science (11)
- # datomic (5)
- # events (2)
- # fulcro (28)
- # garden (32)
- # joker (2)
- # kaocha (6)
- # lambdaisland (4)
- # mount (2)
- # off-topic (11)
- # pathom (10)
- # pedestal (13)
- # re-frame (7)
- # shadow-cljs (15)
- # spacemacs (21)
- # specmonstah (1)
- # wasm (1)
- # windows (1)
- # xtdb (37)
Hi! I'm on an older version of crux (19.07-1.1.1-alpha) using standalone config. Can I switch to latest version with the jdbc backend without losing data? Is there a way to migrate? Thanks!
Hello! :) Yes you can definitely migrate without losing data. There have been a couple of changes since then that you might need to account for though:
1) eviction no longer works at a document level
2) put and delete ops with a "valid time end" argument behave differently
If you weren't using either of those features then you should be okay to migrate without doing any manual transformations. The simplest method is to read the successful transactions out into a file via tx-log
and then submit them back into the new jdbc node. Worth noting that this won't preserve transaction times.
I don't have a code snippet handy right now but I'll give it a go tomorrow!
You can validate it by checking the attribute-stats output matches across both versions
hey! thanks for the help. It's exclusively put txs so it's fine. It's also a toy project which just happened to run for quite a while now and I would be sad losing all it had collected.
(let [data (crux/tx-log sys (crux/new-tx-log-context sys) nil true)]
(clojure.pprint/pprint data ( "data.edn")))
(doall
(for [tx (clojure.edn/read-string (slurp "data.edn"))]
(let [tx' (:crux.api/tx-ops tx)
tx'' (mapv (fn [[tx _ data]] [tx data]) tx')]
(when (seq tx'')
tx''
(crux/submit-tx sys tx'')))))
I have just a small offset in attribute stats which I'm not sure why is happening but it's an increase so it's probably fine 🙂
{:kino.album/artist-ids 38,
:kino.album/release-date 36,
:kino.track/artist-ids 53,
:kino.album/images 108,
:kino.album/total-tracks 36,
:kino.album/name 36,
:kino.track/number 47,
:type 1,
:kino.play/played-at 50,
:kino.play/user-id 50,
:kino.user/refresh-token 1,
:kino.play/track-id 50,
:kino.artist/name 42,
:kino.track/explicit 47,
:kino.track/name 47,
:crux.db/id 176,
:display_name 1,
:kino.track/album-id 47}
{:kino.album/artist-ids 41,
:kino.album/release-date 39,
:kino.track/artist-ids 55,
:kino.album/images 117,
:kino.album/total-tracks 39,
:kino.album/name 39,
:kino.track/number 49,
:type 1,
:kino.play/played-at 50,
:kino.play/user-id 50,
:kino.user/refresh-token 1,
:kino.play/track-id 50,
:kino.artist/name 41,
:kino.track/explicit 49,
:kino.track/name 49,
:crux.db/id 180,
:display_name 1,
:kino.track/album-id 49}
btw I just tried the above with the existing version but I guess it will work when I upgrade too
Nice! That difference is weird though...if you're happy to live not knowing, then no problem, but equally I can definitely help get to the bottom of it 🙂
well, I thought I could ... but know having someone willing to help I cannot 🙂 Should I grab the tx logs for a single key like :kino.track/name
and see the diff before and after? Or is there a more efficient way to tackle this?
(count
(crux/q
(crux/db (-> system :db :db))
'{:find [e]
:where [[e :kino.artist/name ?]]}))
(->>
(crux/q (crux/db (-> system :db :db))
'{:find [e]
:where [[e :kino.artist/name ?]]})
(map first)
(map #(crux/history sys %)))
also gives me a single history records for each one of those - I guess that expected as this is after the import
Ahhh, that's even weirder then! Good idea to check that the history of each only contains one entry. It's possible that there's something we've changed with ingestion that's caused attribute-stats to go haywire, like a race condition of some kind (it's run async after the main index-tx thread runs). I will try to repro on a vanilla data set.
hey again! I gave it a go 20.04-1.8.1-alpha
and attribute-stats were consistent this time. Before and after the export and it matched the db key counts I did. This is with jdbc backend tho. Will try now with rocks.
{:crux.node/topology '[crux.standalone/topology
crux.kv.rocksdb/kv-store]
:crux.kv/db-dir "data"}
everything looks all right with this config as well, so I guess we can close the file on this one 🙂