This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # admin-announcements (3)
- # announcements (1)
- # aws (3)
- # beginners (41)
- # boot (109)
- # braid-chat (2)
- # braveandtrue (5)
- # cider (11)
- # cljs-dev (38)
- # cljsjs (15)
- # cljsrn (5)
- # clojure (87)
- # clojure-berlin (16)
- # clojure-ireland (1)
- # clojure-japan (10)
- # clojure-madison (3)
- # clojure-nl (3)
- # clojure-poland (6)
- # clojure-russia (115)
- # clojure-sg (1)
- # clojurebridge (35)
- # clojured (8)
- # clojurescript (36)
- # core-async (24)
- # cursive (18)
- # datavis (1)
- # datomic (27)
- # dirac (22)
- # editors (1)
- # emacs (3)
- # events (19)
- # hoplon (149)
- # ldnclj (7)
- # lein-figwheel (1)
- # luminus (1)
- # off-topic (70)
- # om (196)
- # onyx (63)
- # parinfer (155)
- # proton (36)
- # re-frame (69)
- # reagent (2)
- # ring (2)
- # ring-swagger (1)
- # slack-help (4)
- # spacemacs (9)
- # testing (11)
@robert-stuttaford, or anyone: Do you have any ClojureScript talent on your team? I have an interesting, small-ish scoped project idea that would be beneficial Onyx users trying to visualize the throughput of their cluster.
@michaeldrogalis: I’ve written a fair few reagent webapps. Would be happy to contribute, but my domain knowledge of Onyx is limited
@lsnape @robert-stuttaford It should be pretty straight-forward. I think the trickiest part is making the UI look appealing. So basically, Onyx has a strictly ordered log of all coordination messages in the cluster. By subscribing to the log (the API lets you do this through a core.async channel), you can learn things about the cluster at any particular point in time. For example, you can learn that at time T, Job
:j1 has Peers X, Y, and Z allocated to its tasks. Being able to visualize that would be huuugely useful for people who are trying to tune their clusters for performance.
There are two things that we can do that almost no other project can, by nature of it having a log. 1 - we can play the log forwards and backwards and show how peers move between tasks as you add or remove jobs. That'd be neat so you can intuit what the scheduler is doing through visualization. 2 - here's the real kicker.. We can call into a metrics serve and display the throughput for each peer at any time
You can think of having a forwards/backwards timeline somewhere in a web UI. As you go forwards in time, you get to see the concrete implications on performance as your cluster topology changes. I think this would be amazing to have.
i think you should draw some diagrams of how this data might be represented
if you can produce that, and cogent apis for a client to call, i’m sure the bit in the middle can be made quite easily!
I can do that The inspiration that we had was from BtrPlace: http://www.btrplace.org/play/ On "Use Case", select Vertical Elasticity, then "Solve", then go to the bottom and click the Play Button.
You can see animations of how the VMs move between servers. If we could do that, plus display metrics per machine.. Holy moly
There's so much potential in Onyx because of how its designed. The bottleneck is creativity and developer time 😛
Getting throughput from a metrics server would just be the beginning. You could imagine that you could talk to Yeller's API and inline exceptions, too. @tcrayford
Oh man, we could even overlay a heat map sort of deal and show where the bottleneck in the job is in terms of latency.
Excellent. Ill put together some materials tonight. We can collaborate right in here.
Any preference on cljs setup? We’ve been used Reagent and Petrol internally and that works really well. Would like to have a play with Om Next too
@robert-stuttaford: yeah part drool but MixRadio shut down on Monday so I still also need to find a job
it’s a real bummer. Incredible team, working environment. Investment deal collapsed at the last minute
i really enjoyed the AWS tooling talk one of your colleagues (or maybe you) gave recently
Anyway, this sounds exciting. Let me know when you’ve got it spec’d out a bit more and i’ll give you a hand
Wonderful, will do. For minimum pain, I should be able to make a Docker container for ZooKeeper with a preloaded Onyx log. We could do the same for a metrics server.
Having day dreams about being able to display arbitrary performance graphs on a machine-by-machine basis with this tool. Someone snap me out of it. 😛
onyx jobs not starting with no exceptions or log messages telling you why! 😁
But yes, I can see how that'd be frustrating as a new user. Hence, this tool.
We have a patch out to attach arbitrary metadata to jobs that get logged with normal messages now, btw. Its pretty sweet - you can name your jobs and what-not.