Fork me on GitHub
#onyx
<
2016-01-29
>
robert-stuttaford05:01:21

i think we’re very, very close

robert-stuttaford05:01:47

everything boots up - i see a running aeron and onyx processes, and then we get the above exception when submitting a job

robert-stuttaford05:01:06

at least, i think it’s when we submit a job, because the onyx service starts up without error

robert-stuttaford05:01:58

{:onyx.messaging/impl :aeron :onyx.messaging/peer-port 40200 :onyx.messaging/bind-addr "localhost" :onyx.messaging.aeron/embedded-driver? false :onyx.messaging/allow-short-circuit? false :onyx.peer/job-scheduler :onyx.job-scheduler/balanced :zookeeper/address "localhost:2181"}

robert-stuttaford05:01:19

(ns cognician.highstorm.aeron-media-driver (:gen-class) (:require [clojure.core.async :refer [chan <!!]] [clojure.tools.logging :as log]) (:import [uk.co.real_logic.aeron Aeron$Context] [uk.co.real_logic.aeron.driver MediaDriver MediaDriver$Context ThreadingMode])) (defn -main [& args] (MediaDriver/launch (MediaDriver$Context.)) (log/info "Launched the Media Driver. Blocking forever...") (<!! (chan)))

robert-stuttaford05:01:49

-Daeron.client.liveness.timeout=50000000000 -Daeron.threading.mode=SHARED

robert-stuttaford05:01:56

version 0.8.6 of onyx

robert-stuttaford05:01:08

what could the NPE mean?

lucasbradstreet05:01:08

Hi @robert-stuttaford. I'll have a look into it shortly

lowl4tency05:01:02

Good morning )

lucasbradstreet06:01:33

Looking into it now

lucasbradstreet06:01:02

Alright, I think I know what’s gone wrong

lucasbradstreet06:01:27

I’ll push out a new version shortly

robert-stuttaford07:01:19

oh, wow. i thought we were doing something wrong! what’s your hypothesis, Lucas?

lucasbradstreet07:01:18

We’re cleaning up the aeron directory on shutdown, but this fails when AeronPeerGroup didn’t start an embedded driver

lucasbradstreet07:01:28

It’s probably relatively harmless because it happens on shutdown

lucasbradstreet07:01:58

I’m releasing 0.8.7-alpha1 now. If you could give it a go when it’s released and let me know whether all is good, I’ll release 0.8.7

robert-stuttaford07:01:21

thank you, for your quick response, Lucas! @lowl4tency will do exactly that right away

lowl4tency07:01:19

lucasbradstreet: thank you, I will notify you when it will be done

lucasbradstreet07:01:56

Cool, onyx core is released

lowl4tency07:01:21

[org.onyxplatform/onyx "0.8.7-alpha1"] correct?

lucasbradstreet07:01:46

hmm, I’m having a problem pulling it down from clojars

lucasbradstreet07:01:33

Ah, sorry, it released the snapshot build before the alpha

lucasbradstreet07:01:40

So it’s still in the process of releasing the alpha

lowl4tency07:01:29

Okay, jsut updating codebase, waiting release and ready to build

lucasbradstreet07:01:58

Blah. Circle is kinda sucking atm. You can try out the snapshot build, 0.8.7-20160129.072333-3

lucasbradstreet07:01:00

It’s exactly the same code

lowl4tency07:01:02

argh, my bad

lowl4tency07:01:26

sh failed "lein do build-prod," Retrieving org/onyxplatform/onyx/0.8.7-SNAPSHOT/onyx-0.8.7-20160129.072333-3.pom from clojars Retrieving org/onyxplatform/onyx/0.8.7-SNAPSHOT/onyx-0.8.7-20160129.072333-3.jar from clojars SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/maven/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/maven/repository/org/slf4j/slf4j-nop/1.7.12/slf4j-nop-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]

lowl4tency07:01:43

ah nope, I used the version correct

lowl4tency07:01:48

lucasbradstreet: ^

lucasbradstreet07:01:03

You were using 0.8.6 prior to this?

lucasbradstreet07:01:37

was lein do build-prod on CI or your own machine?

lowl4tency07:01:15

Trying it locally

lucasbradstreet07:01:27

Eek. Looks like we upgraded to clojure 1.8.0 again

lucasbradstreet07:01:49

how the heck did that happen

lucasbradstreet07:01:07

Try 0.8.7-20160129.075502-4

lucasbradstreet07:01:13

I’m not sure what the problem is though

lucasbradstreet07:01:05

Must have been a bad rebase or something 😕

lowl4tency08:01:22

Locally works

lowl4tency09:01:07

don't see exceptions atm

lucasbradstreet09:01:41

I’ll push out an official release soon

lowl4tency09:01:08

Might be Robert will confir, better simple_smile

lucasbradstreet09:01:20

Yeah, I’ll wait until you give the all clear

robert-stuttaford11:01:59

@lucasbradstreet: we’re not getting that aeron error any more. i think you’re safe to proceed

lucasbradstreet11:01:40

Cool, I’m doing another alpha release just to iron out release kinks first

tcrayford12:01:52

@lucasbradstreet: typo "over a variety stimuli." in the triggering docs, should be "over a variety of stimuli."

lucasbradstreet12:01:10

Will fix that up now

tcrayford12:01:38

I'm kinda looking into using onyx for yeller's backend btw. Still just reading docs and thinking right now though

lucasbradstreet12:01:10

Cool. Even if you don’t decide to use it, I’d be interested in your thoughts once you’ve given it a proper look

tcrayford12:01:21

a thing I'm curious about: what kind of performance/etc stuff do you see out of windowing/aggregation? There's not too much in the docs right now that seems to give me a high level picture of how that stuff is going to play out in prod (e.g. say I want to compute some stats (relatively small amounts of data) over a stream of 1 million events a second, grouped into 1000 buckets, over fixed windows of: 1 minute, 1 hour, 1 day)

lucasbradstreet12:01:35

Currently we’d get pretty killed here. We can manage about 6000-7000 segments per second peer core for a basic grouping task with changelog outputting to BookKeeper

tcrayford12:01:51

http://www.onyxplatform.org/get-started/#onyx-starter should link to the starter project as well as the git clone command (I just wanna read right now, don't want to download or run code)

lucasbradstreet12:01:56

We need to spend more time optimising windowing/aggregations - we’ve been focusing on correctness

tcrayford12:01:02

that seems fair simple_smile

lucasbradstreet12:01:41

Good idea in the get-started

tcrayford12:01:55

the playing I've been doing with samza has it performing just fine for that kind of throughput, btw. Not even a scratch. Of course the systems are designed very very differently

lucasbradstreet12:01:48

Interesting. From the benchmarks I’d seen we were within a few multiples of them

lucasbradstreet12:01:53

Is it checkpointing at all?

lucasbradstreet12:01:03

Thanks. Unfortunately there are heaps of dead links on the site 😕

tcrayford12:01:11

I'll keep on calling them out

lucasbradstreet12:01:21

It’s a translation issue

lucasbradstreet12:01:31

We need a way to convert the links

lucasbradstreet12:01:49

I guess we should switch from relative paths in the md files

tcrayford12:01:55

samza checkpoints the offset it's reached to kafka every few seconds, along with streaming the state of the local k/v store into there

tcrayford12:01:15

(assume that's what the question is around checkpointing)

lucasbradstreet12:01:23

Yup, that’s what I was asking

lucasbradstreet12:01:06

How many cores got you to about 1M/sec with Samza?

tcrayford12:01:30

uh, 16 I think

tcrayford12:01:46

that was somebody else's benchmark on a slightly different problem, but it's somewhat similar

tcrayford12:01:04

yeller's current processing only does 10s windowing, and it hits 1M/sec/core

tcrayford12:01:46

that thing could spit out it's results into onyx though - that thing trims down the stream into 200-300 events/s tops, and it can for sure handle the current (and the peak load)

tcrayford12:01:36

(yeller has… somewhat high throughput requirements. Most of the time shit is actually waaay lower, but when somebody's site goes down hard and sends a 500 for every http req they get, or they break their background jobs, yeller has to process everything without drops)

lucasbradstreet12:01:52

For aggregations like counts, where previous updates are completely superseded by later updates, I have some ideas for how to optimise the checkpointing

tcrayford12:01:52

"The embedded server is currently the recommended approach to running BookKeeper along side Onyx. This will be re-evaluated in the beta release of Onyx 0.8.0." (http://www.onyxplatform.org/docs/user-guide/aggregation-state-management.html) isn't 8.0 already out?

lucasbradstreet12:01:22

I always forget to make those fixes to the docs afterwards

tcrayford12:01:37

Exactly Once Side-Effects Exactly once side-effects resulting from a segment being processed may occur, as exactly once side-effects are impossible to achieve. Onyx guarantees that a window state updates resulting from a segment are perfomed exactly once, however any side-effects that occur as a result of the segment being processed cannot be guaranteed to only occur once.

tcrayford12:01:45

exactly once goes against FLP, right?

tcrayford12:01:53

it's literally physically impossible

tcrayford12:01:06

please to not break the laws of physics in your docs

tcrayford12:01:16

oh, you say that

tcrayford12:01:25

sorry, I copied the whole paragraph whilst only reading the header 😕

tcrayford12:01:52

ah, this paragraph needs a similar thing:

tcrayford12:01:54

Exactly Once Data Processing Exactly once data processing is supported via Onyx's filtering feature.

lucasbradstreet12:01:36

With a link to the side effects section?

tcrayford12:01:40

you're using bootstrap right? use http://getbootstrap.com/components/#alerts to call out caveats like that

tcrayford12:01:17

well, "Exactly once data processing is supported via Onyx's filtering feature" is breaking the laws of physics again 😉. You're either "at most once" or "at least once"

lucasbradstreet12:01:30

Yeah, but it’s all generated from the .md files in the code base. I haven’t looked at the code that generates the page - that was MD’s doing. I’m sure we have a way to do it.

tcrayford12:01:49

most markdown stuff can have just raw html shoved into it 😉

lucasbradstreet12:01:52

So what we mean by that is

lucasbradstreet12:01:12

If we see a value with ID 1, we will only make the change to the aggregation state machine once

lucasbradstreet12:01:45

For example, for a counting aggregation, if we see a segment with ID 1 twice, we will only count it once

lucasbradstreet13:01:35

We can’t guarantee that any side-effects from a trigger that are performed on the results are only performed once

lucasbradstreet13:01:06

that’s what we mean by “data processing”. If you have a better suggestion I’d be happy to change that

tcrayford13:01:39

hmm. Let me give it some thought and get back to you

tcrayford13:01:55

maybe something about "ignoring duplicate messages" instead of the "exactly once" phrasing

lucasbradstreet13:01:17

What about Exactly Once Aggregation?

tcrayford13:01:29

that's better

lucasbradstreet13:01:44

or Exactly Once Aggregation Updates

tcrayford13:01:52

I gotta ask about failure cases here I think

lucasbradstreet13:01:20

It’s a tough one. We did debate whether to call it exactly once and settled on calling it that with proper qualifications

lucasbradstreet13:01:29

Everyone else just claims exactly once without proper qualifications

tcrayford13:01:35

is the checkpointing of the storage of that key in bookeeper transactional with the storage of the aggregation?

lucasbradstreet13:01:47

Yes, stored in the same write

tcrayford13:01:57

and bookeeper is just paxos under the hood right?

lucasbradstreet13:01:36

Yeah, it’s built on top of ZK and guarantees in order writes

lucasbradstreet13:01:03

Earlier today I added a task to my phone to add all the failure modes as a doc. Really need to get to that.

tcrayford13:01:26

meta feedback about doc layouts whilst I notice it: you can (and should) have versioned docs (e.g. http://bookkeeper.apache.org/docs/r4.3.1/, or http://www.postgresql.org/docs/9.4/static/row-estimation-examples.html)

tcrayford13:01:47

(especially if your docs are in the same repo as the code, which it sounds like they are?)

lucasbradstreet13:01:59

Yeah, we need to get to that too. At the moment we’re kinda content to just keep pushing people forward, but we have to do this at some point.

tcrayford13:01:26

I gotta run, but yes, would love to see something about failure modes. I'm happy to review it once you have something (email me: <mailto:[email protected]|[email protected]>)

lucasbradstreet13:01:35

Awesome. Would appreciate that

lucasbradstreet13:01:39

Thanks and catch you

lucasbradstreet13:01:17

Yeah, that’ll be good to get some ideas

lucasbradstreet13:01:25

We want to do that stuff right.

tcrayford13:01:29

onyx's window/aggregation stuff is a distributed database on many levels 😉

tcrayford13:01:33

awesome, good to hear

lucasbradstreet13:01:39

That’s why we’re jepsen testing it 😛

robert-stuttaford13:01:29

readme driven debugging simple_smile

lvh21:01:11

So, we’re doing a thing and I’m not 100% sure how to map it properly to Onyx, but I have some thoughts. Here’s a high-level diagram. We’re trying to do data analysis on a pretty big firehose (syslog, bro, and a few commercial monitoring tools for… a lot of machines). The live data kafka -> onyx -> dashboard and onyx -> cloud files part makes sense; that one’s mostly covered by the standard docs. The stuff I’m more concerned about is queries going back into the onyx cluster, for two reasons: 1. they combine cloud files (historical) data and… in-memory data I suppose? 2. the queries aren’t terribly complex but they operate over quite a bit of data I think that means that we adjust the workflow at runtime, but the catalog entry is fixed; and the catalog entry has a thing that takes an EDN structure describing a query (I think)

lvh21:01:38

I don’t know if any of that makes sense, and what “in memory as a source” looks like. I think that should probably just be another onyx plugin and some clojure data structures until further notice. The hypothesis is that interest is long-taily, and that I care much more about current data than historical data, and with a little luck I can fit most of the current data I care about in memory.

lvh21:01:03

I currently have 5x 512GB RAM boxes, which … may be enough hopefully? At least it ought to be for now; I can get bigger ones from OCP if need be.

michaeldrogalis21:01:21

@lvh: That looks sound. Onyx by itself doesn't do any storage. I'm a little unclear about what you meant when you say "querying back into the Onyx cluster". That typically moves into user application design.

michaeldrogalis21:01:56

How much data are you looking to handle btw? I know you said you have an exceptionally large data set, so it wouldnt at all surprise me if we hit scaling problems moving into those kind of numbers.

michaeldrogalis21:01:52

Something to clarify though is that once you deploy an Onyx job, its effectively immutable. That is, if you deploy a job with workflow [[:a :b] [:b :c]], you can't change that job to be something else.

lvh21:01:37

michaeldrogalis: whoa, that was a poor choice of words on my part

michaeldrogalis21:01:41

Sometimes there's some confusion about runtime at the point of job construction, and runtime at the point of job execution. The former is totally flexible for change - the latter is not.

lvh21:01:44

what I mean is: runtime defined queries

michaeldrogalis21:01:55

Oh, yup. You're all good then simple_smile

lvh21:01:58

Is there a way to “tap” a running workflow? So, I have a -> b -> c -> d, and I’d like to see what a new function f does with the outputs of b/d, ideally without breaking the stream (the kafka inputs do not stop ever)

lvh21:01:22

is there like a draining mode or something? because I probably care about segments in transit being persisted at the end

michaeldrogalis21:01:46

You can use lifecycles to attach that kind of thing. What exactly do you mean by draining mode?

michaeldrogalis22:01:21

@lvh: Also, what is "bro"? simple_smile

michaeldrogalis22:01:10

Ah - nevermind, looked it up. I can infer why your firehose is gigantic.

lvh22:01:08

A terribly named security monitor named before “bro” meant something else.

michaeldrogalis22:01:24

Any guess on how many events/s you want to push it to?

lvh22:01:21

michaeldrogalis: so, for the test deployment probably less than 1E6/s, but it shouldn’t take too long to get there

lvh22:01:45

michaeldrogalis: bro can trigger pcap (as in network capture), which will probably have to go straight to archive

lvh22:01:59

virtually all the interconnects are 10gbe, often with good reason 😉

michaeldrogalis22:01:42

Easy enough for test