This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-05-22
Channels
- # aws (5)
- # bangalore-clj (3)
- # beginners (84)
- # boot (31)
- # carry (1)
- # cider (6)
- # cljsrn (7)
- # clojure (188)
- # clojure-conj (1)
- # clojure-dev (3)
- # clojure-dusseldorf (8)
- # clojure-greece (10)
- # clojure-italy (7)
- # clojure-nl (3)
- # clojure-russia (45)
- # clojure-spec (5)
- # clojure-uk (64)
- # clojurescript (83)
- # clr (9)
- # core-async (45)
- # cursive (55)
- # datomic (8)
- # devops (1)
- # emacs (7)
- # graphql (1)
- # hoplon (3)
- # jobs (1)
- # jobs-discuss (37)
- # lein-figwheel (3)
- # leiningen (4)
- # luminus (6)
- # off-topic (5)
- # om (2)
- # onyx (33)
- # pedestal (13)
- # re-frame (9)
- # reagent (28)
- # rum (15)
- # specter (4)
- # unrepl (38)
- # vim (3)
it doesn't look like http://www.onyxplatform.org/docs/user-guide/latest/ covers Checkpointing (aside from a few references). should it?
@devth We cover it a little bit under the ABS section, but yep, it should have a dedicated section. One of the last tasks is to document that before release 0.10. http://www.onyxplatform.org/docs/user-guide/latest/#_asynchronous_barrier_snapshotting
btw i understand that ABS syncs checkpoints to S3, is this correct ? and if so, how do i configure it ?
@lmergen that’s right. Configuration via these keys
hmm linking to searches isn’t working, but search for s3 in http://www.onyxplatform.org/docs/cheat-sheet/latest/#/peer-config
via the search bar at the top
and configure http://www.onyxplatform.org/docs/cheat-sheet/latest/#peer-config/:onyx.peer/storage
ah i see zookeeper is used by default. i assume it's contained within the tenancy id ?
cool. i’m trying to figure out how i can use this to “snapshot” my aggregates as well (use this epoch?), so that i can rebuild aggregates based on these snapshots
i’m not exactly sure about the guarantees onyx gives me — the checkpointed!
call is probably not atomic for all inputs and outputs at the same time eh (meaning i cannot use the epoch
there)
ah wait! i just noticed that this, in fact, is consistent — while the ABS is snapshotting, input is temporarily stopped, correct ?
@lmergen yes, each task will stop, take a snapshot of the state, then resume processing while pushing the snapshot to S3 asynchronously
Yeah, the model makes it really easy to take consistent snapshots without slowing things down too much.
so all i have to do is persist the epoch in my aggregate store as well, and i can just recover aggregate state as well
or rather, bring it back in the state as it was at the moment the last barrier was injected
Right, yes, when it recovers it will tell you the epoch that it’s recovering to, so you just have to load the corresponding epoch
note that it’s an epoch and replica version combination
since the epoch resets each time the cluster changes.
ideally it’d be a single number which is the combination of each, but I was worried about overflowing a 64 bit long if I combined them
Yeah, it would just be easier on users if they only had to ever worry about one increasing number.
e.g. A is a stream that I want to enrich with stream B and they both share a primary key
You currently need to build your own aggregation to do this.