The explanation of Storm guarantees is really interesting. I remember reading an explanation but it did not really click back then.

I love writing non-user facing functions and using Australian english names 😛

we do the same with British naming here

what garbage collector does the onyx recommend/use for the JVM in general?

G1GC is a pretty safe choice. I think it has similar properties to what you're using now. I tend to err on the side of safety/minimising pauses which it does

we're using concurrent mark sweep, i wonder if that's a wise choice, thinking of trying G1GC rather, especially since our heap size is 5G

It's probably going to be pretty similar, but yup, it's probably a good choice. If you have a memory leak or are hitting your limits you're going to have a bad time whichever GC you use though

true

good news; I have navigated the powers that be sufficiently well to go to clojure/west

woo!

@michaeldrogalis: at 8:30 in https://youtu.be/YlfA8hFs2HY?t=8m30s you talk about a collection of behaviors and the execution spec/jobs which are deployed at two different times. Where could i got to find out more? What would be an example of each? Assuming a job is the catelog, workflow, windows, lifecycles, etc... Then what is the "collection of behaviors". Im trying to solidify my understanding because i want to pitch to present the idea to my company. Also because its awesome.

@drewverlee: The job is just a datastructure, a normal nested EDN map. When Mike talks about this being divorced from behavior, he means that the actual code is not included in the datastructure. The datastructure (execution map/job from the talk) is just there to describe to computation.

It’s most easy to think of the collection of behaviors as all of the code that you’ve written to process each step your data goes through, and the execution spec / job is the description of how you want that code you wrote applied to your data.

Now the reason this is significant is because once you have data describing your computation, you can modify it with the normal tools you use every day. And tooling can quickly be built to generate these execution-maps/jobs far easier when it’s just data than if it was data+code.

@lvh: Hooray! Looking forward to meeting you.

@drewverlee: What @gardnervickers said

@gardnervickers: thanks. In trying to connect this back to what i learned in the learn-onyx repo. I have a workflow, a catalog, some lifecycles, windows, flow conditions, triggers. Would it be fair to say these things are the execution spec/job? Meanwhile the function i pull in to do the processing say :onxy/fn :my-function is the an example of 'a behavior'

Exactly correct

oh great!

Yup, everything that runs through onyx.api/submit-job is the job.

@drewverlee: Is your company currently doing some kind of distributed activity? I can give you a good analogy for a lot of related systems.

@michaeldrogalis: were doing a lambda architecture. Kafka to storm and hadoop with opentsdb. Im not intimately involved in that end. I only started their 3 weeks ago.

@drewverlee: Got it. Sounds like it'd be an interesting spot to try it.

Im going to put together some of the things i have learned into a blog and a talk as a way to try and give back to you guys for all your sharing.

@drewverlee: Thanks, I really appreciate that. Makes it really rewarding for us 😄

Awesome @drewverlee!