Fork me on GitHub
#onyx
<
2017-04-23
>
lmergen09:04:31

so, i’m considering separating different code versions using different onyx tenancy id’s, but i’m a but uncertain whether this is a useful pattern. intuitively, it seems like a good idea, so that you can reason about an entire tenancy as a single version of compatible code (i.e. when you have more than 1 job running), and to make migrations easier. i’m also using different kafka topics for different versions of my code, so this appears to make a lot of sense as well (so that different tenants / code versions don’t share the same topics either). would there be any disadvantages to this approach ? am i overengineering this ?

lucasbradstreet09:04:09

@lmergen that’s exactly how we use it.

lmergen09:04:34

@lucasbradstreet hah okay, thats great to know

lmergen09:04:43

i’m on the right train of thought then

lucasbradstreet09:04:00

We’ve been taking to using our docker image tag + an increasing number as our tenancy id (so we can bump a tenancy id, without changing our code).

lucasbradstreet09:04:31

You will at least need to bump your tenancy id when you upgrade your onyx version anyway, because we can’t guarantee interoperability between onyx versions.

lmergen09:04:55

yes, exactly, so that entire tenancies are in fact immutable — that makes things a lot easier imho

lmergen09:04:04

and immutable in an automated way as well

lmergen09:04:29

i’m usually just using the git describe result as a version id

lmergen09:04:34

made the most sense to me

lucasbradstreet09:04:36

Yep, then we have some code that migrates from the old tenancy to new tenancy via resume points

lucasbradstreet09:04:42

Yep, git sha is also good

lmergen09:04:13

that’s exactly what i thought when i saw the resume points

lmergen09:04:32

i thought about using kafka offsets manually, but the resume points seem to be more flexible

lucasbradstreet09:04:02

Yep, plus if you use any kind of aggregation, you can carry over that state in a way that will be consistent against the offsets that you use via resume points.

lmergen09:04:42

yes, i was under the impression resume points were designed for the CQRS snapshotter concept ?

lucasbradstreet09:04:29

Essentially to allow for migrations that will allow you to maintain the exactly once / consistency guarantees. Pretty similar concept to CQRS snapshotter, yes.

lmergen09:04:58

great, thanks for the info

twashing13:04:23

Hey guys, I’m trying to get started with Onyx, using a commander pattern. But something’s not working (https://github.com/onyx-platform/onyx-commander-example/issues/3).

twashing13:04:43

Anything obvious I’m missing?

lmergen16:04:04

that wil run at compile time

michaeldrogalis17:04:07

@twashing Thanks for the report! I’ll check this out when I’m back — about 4 hours.

michaeldrogalis17:04:48

@twashing From a very quick read, I’d check to make sure that your Kafka topic is being set up and torn down correctly. I had a lot of problems when I wrote that example with reusing the same topic name on Docker compose. I never figured out exactly what was going on, but the consumer wasn’t reliably picking up new messages - completely excluding Onyx.

michaeldrogalis17:04:00

Will investigate more later.

twashing20:04:44

@michaeldrogalis Hmm, looks like you’re right. I put in an extra function , to three-consume-from-topic. And indeed there’s nothing there. https://github.com/twashing/simple-kafka-onyx-commander/blob/master/src/clojure/com/interrupt/streaming/core.clj#L55

twashing20:04:02

There’s also a kafka-tools container that should help troubleshoot the issue. https://github.com/twashing/simple-kafka-onyx-commander/blob/master/docker-compose.yml#L38-L49

twashing20:04:24

I’m gonna fiddle with it some more myself, and see what I missed.

michaeldrogalis21:04:03

@twashing Okay, keep me posted. I’ll dive in more if that doesn’t end up being the issue

michaeldrogalis21:04:01

@lmergen: > yes, exactly, so that entire tenancies are in fact immutable — that makes things a lot easier imho > >[2:13] >and immutable in an automated way as well Bingo, this is exactly what we’re settled on. It works great.

michaeldrogalis21:04:48

We also use versioning to track which code built which Onyx job and deployed to a particular tenancy, so we can reproduce problems all the way from the top of the architecture to the bottom if something goes bad.