This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-10-27
Channels
- # aws (3)
- # aws-lambda (8)
- # bangalore-clj (1)
- # beginners (155)
- # boot (13)
- # cider (88)
- # cljs-dev (3)
- # cljsrn (9)
- # clojure (120)
- # clojure-india (1)
- # clojure-italy (2)
- # clojure-norway (1)
- # clojure-romania (2)
- # clojure-russia (41)
- # clojure-spec (4)
- # clojure-uk (34)
- # clojurescript (68)
- # core-logic (16)
- # cursive (11)
- # data-science (9)
- # datomic (19)
- # dirac (6)
- # duct (20)
- # emacs (7)
- # events (2)
- # figwheel (4)
- # fulcro (12)
- # graphql (1)
- # hoplon (68)
- # klipse (1)
- # leiningen (7)
- # lumo (11)
- # off-topic (9)
- # onyx (114)
- # pedestal (4)
- # protorepl (15)
- # re-frame (60)
- # reagent (3)
- # ring (18)
- # shadow-cljs (15)
- # spacemacs (82)
lucas / michael: Could either of you talk briefly about how updating code interacts with jobs, when to use resume points, etc. For instance, if I fix a bug in my business logic that doesn't change the onyx job definition, is best practice just killing all my peer java processes, redeploy a new jar, and just turn the peers back on, keeping the same job running throughout? Are the only times I should be using the resume-points is if I actually need to change the structure of my onyx job definition? Are there any timeouts or anything to be aware of with how long a job can stay in zk without peers, does it stay frozen indefinitely?
Will write up an answer in the AM - Ill get these all under the FAQ, they're good questions.
specifically, it appears that after a :submit-job
, i'm not receiving anything else anymore
i know for a fact that a job is running, but my process that's monitoring the onyx log doesn't receive anything else anymore after submit-job
There won’t necessarily be any other messages unless something changes about the job
The submit job message is enough for all of the peers to know what to do and start the job
You have to actually apply the log entries
Via the same log application method as the peers will use
just looking up a good example
http://www.onyxplatform.org/docs/user-guide/0.9.10-beta2/#_subscribing_to_the_log
old version since I got to the docs via google 😛
Same deal in 0.11 http://www.onyxplatform.org/docs/user-guide/0.11.x/#_subscribing_to_the_log
applying the log will basically start communicating with the peers directly, to figure out the events that are not stored in the onyx log ?
basically, the current cluster state is a state machine which is built by applying the log entries in turn. This ensures that every node in your cluster has the same view of the cluster
applying the log just applies the log command to the state machine
to materialize the view of the cluster
if you don’t apply the log entries, you won’t know what the current cluster state is
yeah, it does happen in the peer-group, but if you want to get a view of the cluster from outside this has to take place somewhere else.
since it’s easier for you to just do it yourself than reach into the peer group
no worries, good luck. Looking forward to seeing what you come up with. We have some similar code to do log -> job status in the DB. We tried to keep the flow one way otherwise it gets very tough to reason about.
essentially log playback -> DB for the job status
then if you want to change the job status, send a kill-job command to the cluster
and wait for that to go back through the playback flow to get the new status
Cool, hope it turns out well. Onyx needs something like this
hit me up if you have any more questions
im looking at this: https://github.com/onyx-platform/onyx/blob/0.11.x/src/onyx/log/commands/submit_job.clj#L62
since the allocations is not explicitly part of the submit log entry, i assume that allocation is always deterministic and as such is correctly reproduced in all replicas. is this correct ?
Yes it’s completely deterministic.
The only exception /can/ be if you switch from one onyx version to another. When we migrate to new onyx versions we just figure out what the end submit/kill job results are and migrate the jobs that aren’t killed. That way we don’t need it to play back exactly the same
Is this just for interest to understand how it’s all working?
Ok cool :). Just making sure since I thought I might not understand your problem well enough
Definitely good to understand what’s going on :)
Yeah it is a bit heh
It’s not so crazy once you understand it but it’s kinda novel for it to all work like that
We had to set a seed on the scheduler so it would make the same decisions everywhere
This is what we mean when we called onyx masterless, although we do still lean on zookeeper for the rest, so it’s not completely masterless
One nice thing with this is that you can find out the state of the cluster at any time, by looking at the replica state at that time
onyx-dashboard allows you to step through it, which can be useful when debugging
Zzz time
@eriktjacobsen Ill repaste your question down here:
lucas / michael: Could either of you talk briefly about how updating code interacts with jobs, when to use resume points, etc. For instance, if I fix a bug in my business logic that doesn't change the onyx job definition, is best practice just killing all my peer java processes, redeploy a new jar, and just turn the peers back on, keeping the same job running throughout? Are the only times I should be using the resume-points is if I actually need to change the structure of my onyx job definition? Are there any timeouts or anything to be aware of with how long a job can stay in zk without peers, does it stay frozen indefinitely?
The first thing to get a mental picture of is the division of behavior (your concrete Clojure code and functions that do real things) and wiring instructions (the Onyx job that stitches all the Clojure code together at runtime). When you stand up a peer, all you've done is plugged that peer into the network to listen to the log for instructions, and possibly open up a network connection to another peer when it's going to run a task for a job. This peer has the behavior bundled into its jar - it doesnt know anything about your specific Onyx jobs. When you submit an Onyx job, you're putting those instructions about how to use your behavioral code into ZooKeeper, which goes into the log, which is eventually read by a peer. The submission of the job happens away from the cluster - these instructions are received at runtime. If you have a bug in your business logic that pertains to your code, you'll definitely need to reboot the peer with the new code. Whether you stop and start your job is a completely different matter that depends on whether or not you need to replay all your data to correct the error. It also depends on whether its okay for your domain to possible have two peers on at the same time with the incorrect and the correct code. Resume points are used for transferring state between jobs. So in the case that you do need to restart your job, but you want to keep your progress so far in regards to window contents, Kafka offsets, and so forth - you use resume points to literally resume previous progress.
There are no inherent limitations for how long a job can stay around before peers don't pick it up for work. Its simply a key in a map waiting for a peer to be scheduled on it. See the above conversation with @lmergen about how the log works.
I also gave a talk a few years ago on this called Inside Onyx that you can find on YouTube that explains the log. None of that has changed
It's a log architecture being used in a place where it's never been used before. Storm/Flink/Spark/Data flow all use architectures that rely on a centralized coordinator. To the best of my knowledge Onyx is the first to do this.
So the latest on the Media Driver Saga on GKE. We did lots of debugging to determine if the peers could communicate over UDP and that seems to all work fine. Our last attempt for the day was to set the media driver into a dedicated thread mode . Currently the idle job has been running for 10 minutes where before it wouldn’t make it past 1 minute. Any thoughts on the media driver thread mode ?
Thanks for the explanation michael, clears things up
If it's surviving that's indicative of the peer being noisy wrt to resource usage. Dedicated thread mode for Aeron will do just that
@camechis if dedicated mode is making it better it makes me wonder if this job is complex/large and is opening a lot of channels. It would also partially explain why it’s working in single peer mode.
@camechis how many peers in this job?
How many nodes?
Ok that isn’t the cause then
The media driver and the peers might just be fighting for resources then
i have 5 physical nodes in my cluster with about 30 gigs available in mem and not a whole lot of CPU being used since everything is pretty idle
@lucasbradstreet, @camechis shared with me that they're individually limited through K8s resource constraints, more than our test env.
Via limits?
Yes, where we provide a single upper bound on both containers, they are providing individual bounds though.
That should be enough machine resources at least, though the limits might apply. I dunno.
Limits for CPU are not strict either, they're a scaling factor in relation to other containers running on the node.
I think you should get a flight recording of the peers and of Aeron
Yes, one sec
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.rmi.port=1099
-Djava.rmi.server.hostname=127.0.0.1
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
Using those JVM opts, you can kubectl port-forward
to 1099
, then run Mission Control to get a flight recorder file.I should say you should only do this for testing/non permanent prod purposes only, as flight recorder is commercial in jdk8. It’s opened up in OpenJDK/jdk9 thankfully
It’s really great
I can't commit to anything for at least next 2 months, however, I'm curious are you guys actively working on spec integration? I see https://github.com/onyx-platform/onyx-spec but it looks like focused on speccing onyx itself rather than integrating it with jobs. It seems like an analogous data structure to workflow that lists specs for messages passing between two components, something in catalog to say what specs a task accepts and emits, and being able to put a spec directly into a flow condition instead of having to wrap it in an opaque predicate function would be really nice and would also let you generate diagrams / documentation for your job that actually shows the types of data flowing between each node. Thoughts?
@eriktjacobsen I had considered that before Spec came out. I think that would make a really nice supporting library - it adds an additional checker to your runtime