Fork me on GitHub
#onyx
<
2017-12-18
>
asolovyov13:12:02

hey! Would you be willing to accept changes to onyx-kafka which (optionally) commits offset to Kafka? Without that our process breaks, so I'm hoping you would accept that so that we don't need to maintain fork 🙂

fdserr14:12:44

Hi! Is there a hot code reload story with Onyx on a local setup? Expecting to reload jobs/fns, not the catalog... Pointers much appreciated, thanks!

niamu16:12:53

@asolovyov You can set Kafka consumer options to enable that like so: :kafka/consumer-opts {"enable.auto.commit" true}

lucasbradstreet17:12:39

@asolovyov that should work. What do you need the commits for?

michaeldrogalis17:12:37

@fdserr It's just a jar -- standard nrepl reloading story for Clojure applies 🙂

michaeldrogalis18:12:55

4 years everyone! Fun times.

asolovyov19:12:39

@lucasbradstreet at least for our operations guys for monitoring 🙂

lucasbradstreet19:12:00

Right, was wondering if it was for that. That’s a good use case for it.

lucasbradstreet19:12:46

You can also look at lag_Value from onyx’s metrics. It’ll tell you how far a peer is behind the head of the partitions, though it does aggregate multiple partitions into one value.

asolovyov19:12:04

riight, that one is also interesting

asolovyov19:12:32

also when we make some changes to our jobs we usually make a new release of the project, which means a new tenancy id

asolovyov19:12:57

which means checkpoints are left in old tenancy, so I think easiest way for us is to use kafka offsets for some time

asolovyov19:12:07

not sure that's the best practice right now though...

lucasbradstreet19:12:31

Ah, I see. Resume points is usually the way of picking those back up.

lucasbradstreet19:12:53

Resume points will allow you to pick them up from a diff tenancy.