This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # aatree (9)
- # admin-announcements (2)
- # alda (4)
- # announcements (2)
- # beginners (87)
- # boot (218)
- # braid-chat (14)
- # cbus (2)
- # cider (19)
- # cljs-dev (17)
- # cljsjs (1)
- # cljsrn (5)
- # clojure (84)
- # clojure-android (1)
- # clojure-czech (8)
- # clojure-ireland (3)
- # clojure-madison (20)
- # clojure-poland (22)
- # clojure-russia (54)
- # clojure-sanfrancisco (1)
- # clojurescript (81)
- # clojurewest (4)
- # community-development (94)
- # conf-proposals (5)
- # core-async (199)
- # css (3)
- # cursive (68)
- # datavis (2)
- # datomic (23)
- # dysphemism (138)
- # editors (7)
- # hoplon (8)
- # jobs (8)
- # jobs-discuss (7)
- # ldnclj (2)
- # liberator (6)
- # off-topic (32)
- # om (200)
- # omnext (2)
- # onyx (88)
- # proton (58)
- # re-frame (14)
- # reagent (1)
- # ring-swagger (26)
- # yada (14)
hey lucas another question about lifecycle error handling. we submit to yeller in a task that receives stuff from exception? true flow conditions. should we submit to yeller from the lifecycle one too?
@robert-stuttaford: I think in the lifecycle case it at least logs to timbre error, so should be picked up by yeller
the fn in lifecycle can side effect and then return one of those 3 kws right? :restart :kill :and-the-other-one
Hmm, if you never got the yeller timbre appender you could be missing out on a lot of onyx exceptions that you want to know about
it’s the current event context (basically has everything the whole task needs)
ok. our segments each have the full system config in it, which i need to reach into to know whether to yell or not - how would i get to it?
why do you need the segments? the exception could’ve been caused by anything - e.g. loss of connection in the input plugin
Ah seems a bit weird for that to be in the segments. I think you should inject that into the event map
See the standard-out appender which has its own standard-out-logger output fn
If you don’t want to use yeller-timbre-appender, you can just put the the timbre call in your custom fn
i’ve just searched through the likely onyx projects to use yeller and couldn’t find any examples - sorry, but could you link me to one, please?
the jepsen test I created randomly kill -9s a random subset of node’s onyx jars
I’m looking into using Onyx to do notification digests and trying to understand windowing better. Would it be feasible to have 10–20k concurrent session windows? Would each require a virtual peer?
@joshg: I think it depends on your cluster topology. Windows are durably persisted to BookKeeper for fault tolerance, but they are maintained in memory between sync calls. Number of windows doesnt map to number of peers. Peers can have arbitrarily many windows depending on which task they're working on
You could have 1 peer working on, say, 1 task that does notification handling, which has 20,000 windows. That has to fit in memory for the machine. Does that help?
Yes, that’s exactly what I needed to know. Thanks! So if they didn’t fit in memory on a single machine, I would have to partition them into separate tasks, which could get allocated to other cluster members.
Stick with the number of tasks that makes sense for your domain. Add more peers. You can have multiple peers working on the same task. Onyx knows how to partition work across peers sensibly.
e.g. If you had 3 tasks and 9 peers, Onyx's Balanced task scheduler (the one most people use), will allocate 3 peers to task A, 3 to task B, and 3 to task C. You can use a Percentage based task scheduler, too.
We'd recommend running BookKeeper in standalone mode in production for performance reasons, but for development/general playing around you can run it in embedded mode - which is what you're probably doing now. Check your peer config - there's a line in there enabling it to run directly on the peer.
@joshg: Sure! Yeah it's a wonderful idea. The DataFlow paper came out at just the right time. A lot of our API comes from there.
Out of curiosity, what are the advantages of using BookKeeper over Kafka for a replicated log?
@joshg: That's a good question. BookKeeper behaves a little differently than Kafka, even thought they look very similar. BK's abstraction only lets a single process write to a ledger, what Kafka calls at topic (sort of). When that process stops writing, no one else is allowed to write to the ledger. That helps with a lot of potential concurrent writer problems. Kafka also doesn't have a transactional writer, which is what is holding Samza back from having exactly-once semantics.
That makes sense. I know that they’re working on transactional messaging for Kafka, but I’m not sure how close it is to being done.
@joshg: we’d like a Kafka state log implementation, but until they support transactional messaging it’s not really a priority for us