Fork me on GitHub
#onyx
<
2016-08-29
>
brandoff00:08:26

@michaeldrogalis Was thinking of writing a k-means|| implementation on top of it but wondering if it's worth it if next release will make it easier, as @mlimotte mentioned,

michaeldrogalis00:08:54

@brandoff I'm admittedly not familiar with how to most efficiently implement that, or what primitives would be required. If local iteration is what you need, Onyx can do that now. If records need to efficiently flow through repeated tasks, you'll want to wait.

michaeldrogalis00:08:03

Gotta run, happy to answer questions when I'm back.

brandoff00:08:52

Thanks 🙂

aaelony03:08:45

@brandoff wouldn't you be using a window of data, and running kmeans on each window separately?

robert-stuttaford11:08:59

@lucasbradstreet i don't suppose you've coordinated ext aeron and an onyx jar with systemd yet, have you?

lucasbradstreet11:08:49

Afraid I haven’t. We created scripts that are started up by s6 in alpine linux on Docker, but no systemd yet

robert-stuttaford11:08:45

ok, cool 🙂

robert-stuttaford11:08:50

i'm pretty close, i think

robert-stuttaford11:08:13

this appears to be two hosts both running a task set to max-peers 1

robert-stuttaford11:08:04

i've just made sure they're all using the same tenancy-id -- they are (it's based on our git sha)

robert-stuttaford11:08:22

i've also made sure that the mechanism that decides which instance will submit jobs ensures just a single instance (it does)

robert-stuttaford11:08:03

what do you think could cause metrics from two instances for a single task like that?

lucasbradstreet11:08:37

You didn't end up splitting your jobs up, right? I remember some discussion about two jobs both with read-log tasks (sanity question)

robert-stuttaford11:08:39

i'm summing by host

robert-stuttaford11:08:56

is it possible for input tasks to report these metrics from multiple hosts?

robert-stuttaford11:08:05

no, it's all on one read-log

lucasbradstreet11:08:36

Ok. Next mostly likely was that you had submitted the job twice, but it sounds like that isn't the case either

robert-stuttaford11:08:53

actually, this explains why we sometimes see 50ms flat lines and 100ms flatlines; multiple instances

robert-stuttaford11:08:03

oh, certainly submitted more than once, but killed as well

robert-stuttaford11:08:32

i've totally forgotten how to ask ZK what it's currently got. can you aid me on that, or should i RTM?

lucasbradstreet11:08:10

You mean for the current Onyx allocation? Easiest would be the dashboard if you can get it up

robert-stuttaford11:08:03

ok. i'll give that a go

robert-stuttaford11:08:18

there's not perhaps a shell command i can issue to ZK directly?

robert-stuttaford11:08:23

if i know the tenancy id

lucasbradstreet11:08:48

Not really because you have to play back the Onyx log using our code to actually get a view of the replica back

lucasbradstreet11:08:21

There will be a web service you can switch on that will make this trivial in the next release

robert-stuttaford11:08:37

what version will that be?

robert-stuttaford11:08:42

we're still on 0.9.6

robert-stuttaford11:08:04

be keen to try it out

robert-stuttaford11:08:54

wow. some pretty big fixes since 0.9.6. will defo get upgraded

aengelberg17:08:00

I'm about to work with exception handling with flow conditions in Onyx, so I'm reading the docs for clues. The docs say :flow/thrown-exception? true set in a flow condition will cause that flow to be activated in a failure case. But it does not say whether the flow condition will ONLY be activated in a failure case. i.e. does a flow condition with :flow/thrown-exception? true also get called with successful segments?

michaeldrogalis18:08:27

@aengelberg It does not get called for successful segments, no.

aaelony23:08:51

question regarding the onyx-dashboard... The Job Visualization panel is very nice, is there a way to make it shrink or scroll to see the entire workflow when the workflow doesn't fit in the allotted space?

smw23:08:02

Hey, where are the docs for how to actually provision an onyx cluster in production? Run against mesos, etc?

Travis23:08:46

It'd kind of up to you to figure it out based on your platform but the it's not a lot to it

smw23:08:06

How do cluster members discover each other?

gardnervickers23:08:15

We don't have anything public for Mesos right now but the onyx-Twitter-sample has some kubernetes manifests. It's fairly straight forward.

gardnervickers23:08:20

Through Zookeeper

Travis23:08:37

If using mesos I can help

smw23:08:46

Awesome. Thanks.

smw23:08:58

I have a big mesos cluster sitting here, so I’d love to reuse it.

Travis23:08:45

I having been deploying on mesos for a month or so now. Only thing I haven't worked out is how I want to manage jobs

Travis23:08:10

The Twitter example although kubernetes can still help a lot

smw23:08:48

Is there no persistent server endpoint? I guess that would lose the masterless bit?

smw23:08:56

So you just launch a job against marathon or something?

Travis23:08:18

Yeah, deploy the peers through marathon and the jobs

Travis23:08:43

Could also possibly use chronos or metronome as well for job deployment

smw23:08:25

Thank you 🙂

michaeldrogalis23:08:28

@smw You can deploy a web server in lib-onyx to track the log and act as if it were a master. https://github.com/onyx-platform/lib-onyx#replica-http-server

smw23:08:09

awesome!

smw23:08:11

Thank you