This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-06-27
Channels
- # bangalore-clj (2)
- # beginners (37)
- # boot (16)
- # cider (17)
- # clara (4)
- # cljs-dev (351)
- # cljsrn (16)
- # clojure (219)
- # clojure-belgium (4)
- # clojure-dev (3)
- # clojure-france (2)
- # clojure-italy (24)
- # clojure-russia (23)
- # clojure-spec (55)
- # clojure-switzerland (3)
- # clojure-uk (89)
- # clojurescript (121)
- # cursive (2)
- # datomic (29)
- # devops (2)
- # graphql (8)
- # hoplon (15)
- # immutant (5)
- # lein-figwheel (4)
- # liberator (3)
- # luminus (18)
- # off-topic (9)
- # om (6)
- # onyx (31)
- # pedestal (48)
- # precept (9)
- # re-frame (19)
- # reagent (63)
- # ring-swagger (69)
- # robots (1)
- # slack-help (14)
- # spacemacs (12)
- # sql (2)
- # test-check (4)
- # unrepl (28)
- # untangled (5)
- # yada (3)
i'm on onyx 0.9.15 and seeing occasional significant segment processing delays - some small number of segments (sourced from onyx-kafka) seem to go missing for 30s or so, and then turn up and get processed
also, it only seems to happen when peers have been running for a while
does that ring any bells with anyone ?
i was wondering that too - not sure how to investigate though
cool - i haven't been running metrics - i'll hook it up and see
@mccraigmccraig Almost definitely replays kicking in.
i'm not sure why replays would be kicking in though - my fns are logging an id for the segments in question immediately, and the delayed invocation is the first mention i'm seeing in logging
Metrics will tell the story.
@michaeldrogalis I am starting to look at a local fs check point implementation. A couple questions, do I need to implement all the multi methods in checkpoint and also assuming I get something is there a good way to test it?
@camechis Yes - all the checkpoint multimethods need to get implemented for alternate storage. Are there ones you’re looking at that potentially seem S3 or ZooKeeper specific?
All you’d have to do is change out the configuration to use your new backend and run the full test suite to get a base line. Jepsen testing it would be a good idea too. https://github.com/onyx-platform/onyx-jepsen
We caught a bunch of problems early with the S3 implementation doing that.
Yeah, all of these are required https://github.com/onyx-platform/onyx/blob/0.10.x/src/onyx/checkpoint.clj
Hard to test jepsen with a FS based state store, since all nodes would need everything to recover
Yeah, i have never used jepsen before so i don’t know a whole lot about it. Also i know this impl is probably a very limited use case. Although maybe it could be used for a physical deployment that is writing to some type of NFS share
Ah, I figured it the storage was distributed. Doesn’t buy you much then.
yeah, i just want to get this impl done so we can move to 0.10 for our minimal use case right now
Can’t remember whether it has enough of the API implemented, but you could try something sneaky and run the FakeS3 gem and connect to that endpoint
In the good or bad way?
Cool, that should save you some work.
Haha anytime.