so, i’m considering separating different code versions using different onyx tenancy id’s, but i’m a but uncertain whether this is a useful pattern. intuitively, it seems like a good idea, so that you can reason about an entire tenancy as a single version of compatible code (i.e. when you have more than 1 job running), and to make migrations easier. i’m also using different kafka topics for different versions of my code, so this appears to make a lot of sense as well (so that different tenants / code versions don’t share the same topics either). would there be any disadvantages to this approach ? am i overengineering this ?

@lmergen that’s exactly how we use it.

@lucasbradstreet hah okay, thats great to know

i’m on the right train of thought then

We’ve been taking to using our docker image tag + an increasing number as our tenancy id (so we can bump a tenancy id, without changing our code).

You will at least need to bump your tenancy id when you upgrade your onyx version anyway, because we can’t guarantee interoperability between onyx versions.

yes, exactly, so that entire tenancies are in fact immutable — that makes things a lot easier imho

and immutable in an automated way as well

i’m usually just using the git describe result as a version id

made the most sense to me

Yep, then we have some code that migrates from the old tenancy to new tenancy via resume points

Yep, git sha is also good

👍

that’s exactly what i thought when i saw the resume points

i thought about using kafka offsets manually, but the resume points seem to be more flexible

Yep, plus if you use any kind of aggregation, you can carry over that state in a way that will be consistent against the offsets that you use via resume points.

yes, i was under the impression resume points were designed for the CQRS snapshotter concept ?

Essentially to allow for migrations that will allow you to maintain the exactly once / consistency guarantees. Pretty similar concept to CQRS snapshotter, yes.

great, thanks for the info

Hey guys, I’m trying to get started with Onyx, using a commander pattern. But something’s not working (https://github.com/onyx-platform/onyx-commander-example/issues/3).

Anything obvious I’m missing?

@twashing https://github.com/twashing/simple-kafka-onyx-commander/blob/master/src/clojure/com/interrupt/streaming/core.clj#L26 are you sure you want those things to be defs ?

that wil run at compile time

@twashing Thanks for the report! I’ll check this out when I’m back — about 4 hours.

@twashing From a very quick read, I’d check to make sure that your Kafka topic is being set up and torn down correctly. I had a lot of problems when I wrote that example with reusing the same topic name on Docker compose. I never figured out exactly what was going on, but the consumer wasn’t reliably picking up new messages - completely excluding Onyx.

Will investigate more later.

@michaeldrogalis Hmm, looks like you’re right. I put in an extra function , to three-consume-from-topic. And indeed there’s nothing there. https://github.com/twashing/simple-kafka-onyx-commander/blob/master/src/clojure/com/interrupt/streaming/core.clj#L55

There’s also a kafka-tools container that should help troubleshoot the issue. https://github.com/twashing/simple-kafka-onyx-commander/blob/master/docker-compose.yml#L38-L49

I’m gonna fiddle with it some more myself, and see what I missed.

@twashing Okay, keep me posted. I’ll dive in more if that doesn’t end up being the issue

@lmergen: > yes, exactly, so that entire tenancies are in fact immutable — that makes things a lot easier imho > >[2:13] >and immutable in an automated way as well Bingo, this is exactly what we’re settled on. It works great.

We also use versioning to track which code built which Onyx job and deployed to a particular tenancy, so we can reproduce problems all the way from the top of the architecture to the bottom if something goes bad.