This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-02-12
Channels
- # aatree (9)
- # admin-announcements (2)
- # alda (4)
- # announcements (2)
- # beginners (87)
- # boot (218)
- # braid-chat (14)
- # cbus (2)
- # cider (19)
- # cljs-dev (17)
- # cljsjs (1)
- # cljsrn (5)
- # clojure (84)
- # clojure-android (1)
- # clojure-czech (8)
- # clojure-ireland (3)
- # clojure-madison (20)
- # clojure-poland (22)
- # clojure-russia (54)
- # clojure-sanfrancisco (1)
- # clojurescript (81)
- # clojurewest (4)
- # community-development (94)
- # conf-proposals (5)
- # core-async (199)
- # css (3)
- # cursive (68)
- # datavis (2)
- # datomic (23)
- # dysphemism (138)
- # editors (7)
- # hoplon (8)
- # jobs (8)
- # jobs-discuss (7)
- # ldnclj (2)
- # liberator (6)
- # off-topic (32)
- # om (200)
- # omnext (2)
- # onyx (88)
- # proton (58)
- # re-frame (14)
- # reagent (1)
- # ring-swagger (26)
- # yada (14)
hey lucas another question about lifecycle error handling. we submit to yeller in a task that receives stuff from exception? true flow conditions. should we submit to yeller from the lifecycle one too?
@robert-stuttaford: I think in the lifecycle case it at least logs to timbre error, so should be picked up by yeller
But you may want to just in case?
i never got the yeller timbre appender working
so, i’m going to add it myself
the fn in lifecycle can side effect and then return one of those 3 kws right? :restart :kill :and-the-other-one
Hmm, if you never got the yeller timbre appender you could be missing out on a lot of onyx exceptions that you want to know about
I think you may want to just fix that instead
event in the handler is the segment, right?
hmm, perhaps not
it’s the current event context (basically has everything the whole task needs)
ok. our segments each have the full system config in it, which i need to reach into to know whether to yell or not - how would i get to it?
why do you need the segments? the exception could’ve been caused by anything - e.g. loss of connection in the input plugin
to get the system config to read the yeller on/off flag
ah heck let me try the timbre thing again
Ah seems a bit weird for that to be in the segments. I think you should inject that into the event map
via a lifecycle
Happy to help with the timbre appender
how would individual task fns read the event map?
they only receive the segment
i thought about putting it in the event map as well
i see how
:onyx/params
Yeah, that’s the way to go
pity. means we can’t use defnk for task fns
oh well. faster code is better than easier to read code, at scale
why not? defnk supports multiple params no?
hmm. maybe they added it since i last checked
just confirming that your fix is in 0.8.9?
fix is in forthcoming 0.8.10
I’m just running a couple more jepsen runs and will release it
ah. that will explain why i’m still struggling
Oh, which fix do you mean?
the one you fixed in onyx-datomic/read-log
Ah yes. Will be releasing that with the “can’t rejoin onyx/id” issue
btw, I just tested out custom logging here
https://gist.github.com/lbradstreet/41fcd627a46d7c474cd2#file-custom-logging-clj-L19
ok. phew. then i don’t have to worry
See the standard-out appender which has its own standard-out-logger output fn
If you don’t want to use yeller-timbre-appender, you can just put the the timbre call in your custom fn
into standard-out-logger ?
let me try the yeller-timbre-appender again
because it’ll only yell on errors, right?
i’ve just searched through the likely onyx projects to use yeller and couldn’t find any examples - sorry, but could you link me to one, please?
I don’t have any examples
oh, right!
But yes, just rename standard-out-logger to something else
and then modify the function it calls to send to yeller
and also change the min-level to whatever you want it to yell on
cool. i’m going to try both approaches. thank you Lucas!
Good luck
Will have 0.8.10 for you in 30 minutes or so
wonderful. take your time. thank you!
the jepsen test I created randomly kill -9s a random subset of node’s onyx jars
and starts them again
it managed to find your issue pretty quickly
our initial fix was bad too
Just waiting on a couple more runs and we’ll be good
i’m about to test the error handling by killing datomic mid-run
@lucasbradstreet: should i try with 0.8.10-alpha2?
Yep, that should be identical to 0.8.10
I’m just waiting for CI to finish on alpha2
ok. i’m out of brains. we’ll do this on monday
Alrighty. 0.8.10 is releasing now and will be ready for you on Monday :)
I’m looking into using Onyx to do notification digests and trying to understand windowing better. Would it be feasible to have 10–20k concurrent session windows? Would each require a virtual peer?
@joshg: I think it depends on your cluster topology. Windows are durably persisted to BookKeeper for fault tolerance, but they are maintained in memory between sync calls. Number of windows doesnt map to number of peers. Peers can have arbitrarily many windows depending on which task they're working on
You could have 1 peer working on, say, 1 task that does notification handling, which has 20,000 windows. That has to fit in memory for the machine. Does that help?
Yes, that’s exactly what I needed to know. Thanks! So if they didn’t fit in memory on a single machine, I would have to partition them into separate tasks, which could get allocated to other cluster members.
Stick with the number of tasks that makes sense for your domain. Add more peers. You can have multiple peers working on the same task. Onyx knows how to partition work across peers sensibly.
e.g. If you had 3 tasks and 9 peers, Onyx's Balanced task scheduler (the one most people use), will allocate 3 peers to task A, 3 to task B, and 3 to task C. You can use a Percentage based task scheduler, too.
We'd recommend running BookKeeper in standalone mode in production for performance reasons, but for development/general playing around you can run it in embedded mode - which is what you're probably doing now. Check your peer config - there's a line in there enabling it to run directly on the peer.
Its a nice convenience.
@joshg: Sure! Yeah it's a wonderful idea. The DataFlow paper came out at just the right time. A lot of our API comes from there.
Out of curiosity, what are the advantages of using BookKeeper over Kafka for a replicated log?
@joshg: That's a good question. BookKeeper behaves a little differently than Kafka, even thought they look very similar. BK's abstraction only lets a single process write to a ledger, what Kafka calls at topic (sort of). When that process stops writing, no one else is allowed to write to the ledger. That helps with a lot of potential concurrent writer problems. Kafka also doesn't have a transactional writer, which is what is holding Samza back from having exactly-once semantics.
That makes sense. I know that they’re working on transactional messaging for Kafka, but I’m not sure how close it is to being done.
I'm pretty excited for it to come out. Lots of great use cases for it
@joshg: we’d like a Kafka state log implementation, but until they support transactional messaging it’s not really a priority for us