This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-01-30
Channels
- # bangalore-clj (1)
- # beginners (104)
- # boot (207)
- # cider (173)
- # cljs-dev (157)
- # cljsjs (1)
- # cljsrn (51)
- # clojure (196)
- # clojure-berlin (1)
- # clojure-chicago (1)
- # clojure-italy (4)
- # clojure-new-zealand (1)
- # clojure-nl (1)
- # clojure-russia (28)
- # clojure-spec (17)
- # clojure-uk (73)
- # clojured (13)
- # clojurescript (110)
- # core-async (4)
- # datascript (25)
- # datomic (92)
- # editors (1)
- # emacs (157)
- # events (4)
- # hoplon (16)
- # klipse (74)
- # lein-figwheel (10)
- # leiningen (2)
- # lumo (13)
- # off-topic (78)
- # om (3)
- # om-next (3)
- # onyx (14)
- # protorepl (1)
- # re-frame (17)
- # reagent (23)
- # remote-jobs (1)
- # ring-swagger (33)
- # schema (2)
- # slack-help (3)
- # spacemacs (7)
- # testing (1)
- # yada (7)
Morning
Morning
Ah, HMRC, how I love giving you even more money
I really wish accountants would do their jobs properly
Morning all
Good morning
to be fair, the HMRC web experience is quite a lot better than it was just a few years ago
still frustrating that the data they provide needs correcting - but if you enter your data wrong they can penalise you
@yogidevbear you mean you didnt use Alternative Facts to get your HMRC rebate... I should have mine next week 🙂
This discussion reminded me to give them a ring... but the tape on the phone said if it isn't dead-line related call back later.
For anyone interested in graph databases: https://neo4j.com/book-graph-databases/?utm_source=GPPC&gclid=CJPHu8mc6tECFW6x7QodDZ0DAQ
anyone know anyone who could tell me about battle-tested strategies for migrating large cassandra databases ? @otfrom ?
when we did it in hecuba (to solve some partitioning problems) we just wrote some code to copy from one table to the other IIRC
schema migration
our current hacky approach has a distinctly limited lifespan as our data size increases...
ideally i want to be able to move data to a new cluster, migrate the new cluster, point the api at the new cluster and then move any recent changes from the old cluster to the new cluster
so there is no downtime and no risk of a schema migration borking production
hmm... sounds like you want to write new data to both clusters while you do the migration
but that is me just trying to persuade tcoupland acron elise_huard and jasebell of the beauty of my architecture
haha, most of it is on kafka... some not - but i'm happy with making code changes
if the messages are coming in on kafka then you might be able to run a 2nd version of your inserter that puts the code in the right place while you migrate and then turn off the old one
it sounds like you'd want to keep track of offsets at time of migration, to know which bits you'd need processing after everything is functional again
or have all the messages consumed from that point and pushed onto another queue
or even if it was all in kafka at some point and your kafka stuff was archived in something like s3
i can have a complete history in kafka... might have some problems with ordering consistency between old & new databases
tcoupland I did similar to process today's data and then union it with the state produced by the previous day's job to create the new state
well if you have the whole history, then you can reconsume it into the new cassandra
I think I like spark from s3 as it means that my parallelism for the re-processing doesn't need to be the same as the normal daily processing
food for thought - thanks all !
o, one other thought would be to look at getting casandra to start logging it's updates. Then you can load the new one from a back up, catch up by using the log, do a bit of a comparison to make sure everythings cool, then flip over.
yeah, i've got two thoughts at the moment - one to log app-level mutation descriptions and apply those to the new copy as soon as the schema migrations have run, and two is to log cassandra-level mutation descriptions along with their timestamps, which i can easily mod my c* client to do, and do similar