This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-08-14
Channels
- # admin-announcements (3)
- # alda (1)
- # beginners (12)
- # braveandtrue (4)
- # cider (9)
- # cljs-dev (109)
- # cljsrn (6)
- # clojars (4)
- # clojure (40)
- # clojure-japan (5)
- # clojure-russia (10)
- # clojurescript (42)
- # cursive (2)
- # datomic (6)
- # hoplon (3)
- # luminus (3)
- # melbourne (1)
- # om (4)
- # om-next (1)
- # onyx (41)
- # proton (2)
- # protorepl (1)
- # quil (1)
- # re-frame (6)
- # respo (1)
- # testing (1)
@lucasbradstreet: interesting onyx-datomic/read-log question for you. how might i read from TWO database logs, and process the txes from both as a single stream ... in combined, transacted order?
Boy. That's kind of a tough one
i'm starting an epic task to refactor our whole production database, to rebuild everything according to our current best understanding -- correct schema, eliminate unwanted data, etc -- and i also want to combine two databases. to rebuild a db, you need to do it in transacted order, if you want to keep the time data inherent in datomic dbs
my first idea is to stream one database and query the second for any txes between the current and previous txes for the streaming db
walk the first and query the second, basically
was just wondering if it would be possible to plex two input streams some other way
It’s possible to order the txes with windows, but I think you would have to keep the whole DB in memory as we don’t have a disk state store implementation yet
that's actually possible. the whole db is under 8gb
Highstorm git:(rebuild-db) ✗ du -hs ~/Datomic/data/db 8.1G /Users/robert/Datomic/data/db
and i only really need the tx-log which is some fraction of that
e.g. DB-1 read-log -> window-task, DB-2 read-log -> window-task, ensure that window-task orders the tx-es correctly, and use a trigger to write out to the new db
actually if you do the windows / refinements correctly you wouldn’t even need to hold the whole DB in memory
because you can flush out every now and again when you’re sure you won’t be getting any txes out of order
how quickly do you think i can sort ±58 million transactions?
Unsure, but I think if you do it right you will only need to be sorting X at a time
I would be interested in this job - I think others could find it pretty useful for rewriting their history
yes. like i said, it's an epic, because i've got a big list of tech debt to undo
what’s cool about it is you can just leave it running while your main system continues to transact
at some point they’ll be basically in sync, then you take down your main system, let it finish, then swap over
most of the work will be in filtering txes and writing convertors for found tx shapes
yeah! i'll definitely unpack that with you at some point
i want it as a job because i want to be able to reuse it -- being able to rewrite whole-history regularly buys a hell of a lot of flexibility
Cool, it’d be nice if you could use a different transactor to transact the new DB 😕
e.g. being able to re-partition data or shard data
why can't i?
nothing i'm aware of prevents that
My intuition was that the transactor will write its own location out to your storage, and it would be hard to read from one DB which uses one transactor and write out to storage using another
I’ve never tried something like that though so it was only intuition
would definitely want a separate transactor for the final production run, because it's far cheaper to restore big changes to DynamoDB
they're completely separate connections
the only commonality is that they're both accessed by the same JVM process
peers actually find the transactor via storage 🙂
ok cool, makes sense. I was just a bit worried about whatever magic datomic does to figure out the transactor
connect to storage, get primary txor ip, connect, download live index, ready
since they might be on separate dbs it’s probably fine because it’ll just look at that DB instead
it uses storage and heartbeats written to storage to coordinate HA failover
thanks for listening -- i'm off for now. i'll definitely share what i have when it's ready 🙂
Great 🙂