@drewverlee It's also best to make the change on a new branch and submit the PR onto upstream's "dev" branch I think.

I have a question regarding deployment. I'm coming from Storm, where you could bundle an uberjar for each topology you submit. Can I do the same thing with Onyx, that is deploy an uberjar for each job I want to submit. Or does all the code for the jobs I want to submit have to be in the same uberjar? If the latter, how does this affect streaming jobs that are in progress when a new deploy occurs?

@jeremy With Onyx, you create an uberjar that contains all of the functions that will be pointed to by keywords in your job. The data structure that represents the job doesn’t go in the uberjar — think of the uberjar as a collection of functions. The job is the instruction set for how to assemble those functions at runtime

When you redeploy, you’re only upgrading the functions themselves, so if all of your function signatures still resolve correctly, you can redeploy without any issues to a streaming job.

And if not, I'd need to restart all the affected jobs?

@jeremy Correct.

Or, more likely, modify the jobs to fit the changes you made to your functions.

So then it's possible to do a deploy while keeping existing jobs running, as long as the api stays the same?

or do they all restart during a deploy?

@jeremy The former.

There’s no centralized coordinator doing restarts - if there are enough peers online, they’ll self-schedule themselves to work on available jobs.

@jeremy Also relevant - job data is stored in ZooKeeper, not in the jars themselves. That’s the piece that let’s jobs ensure restarts of peers.

Ah, ok. (Sorry if I'm just repeating what you've said, but I want to make sure I understand.) Deploying an uberjar to the peers means restarting them with the new code (there's no hotswapping), and the job picks up where it left off based on the state in zookeeper. is that correct?

No problem, happy to clarify. Onyx’s deployment model is pretty different.

@jeremy Yes, that’s an accurate description.

Interestingly enough, I’ve wanted to try hot-reloading code for a while just for fun. It’s achievable with the usual Clojure nrepl tool chain.

It’d be neat to be able to control an Onyx cluster’s behavior from the repl. Not sure if it’s a good idea - but definitely interesting.

How Onyx confines external state to the lifecycles and keyword->var lookups might actually make it pretty simple to get right too.

thanks!

yeah, it would be cool to see if hot-reloading works

Onyx isn’t doing anything fancy there, so we should get the standard Clojure guarantees around how it usually works.

so no multi-methods, got it. haha

@michaeldrogalis I've just watched the powderkeg talk today. Have you seen it? (regarding controlling onyx from repl)

@yonatanel I haven’t, no.

they know when a new var was defined in repl and send the new jar to the spark cluster, or something like that. I don't know if that's considered hotswapping. https://www.youtube.com/watch?v=OxUHgP4Ox5Q

Spinning up a new jar isn’t quite what I had in mind. Think more what happens in Emacs when you do eval-sexpr.