Fork me on GitHub
#onyx
<
2017-02-24
>
acron11:02:36

Question of the day: How can we exert back-pressure when using onyx-kafka?

lucasbradstreet11:02:19

@acron with onyx 0.9, the main backpressure knob is to modify onyx/max-pending on the input task

lucasbradstreet11:02:47

With onyx 0.10, backpressure is more natural and shouldn't require as much tweaking

acron11:02:19

@lucasbradstreet You may have just saved my bacon -- thanks! Do you happen to know the default?

lucasbradstreet11:02:33

Default is 10000

lucasbradstreet11:02:21

I'd point you to the cheat sheet but I'm not sure it'll display it still since it's showing 0.9 (I think it will display it as deprecated)

michaeldrogalis15:02:39

@acron Notably backpressure isn’t specific to any plugin. That’s especially true with 0.10. 0.9’s abstractions leaked a bit.

acron15:02:21

@michaeldrogalis That explains why it's an onyx namespace'd key 🙂 So input tasks are expected to respect that? Also, in the case of onyx-kafka, does it keep messages in memory until they're ACKd? or will it go back to Kafka to retrieve msgs that fail to ACK?

michaeldrogalis15:02:27

@acron Onyx core polls input storage for messages, so they don’t really have a say in the matter.

michaeldrogalis15:02:14

In 0.9, onyx-kafka will keep unack’ed messages in memory, but won’t advanced its offset checkpoint until the lowest saved record has been fully acked — to answer what I’d expect your question will be, there should be no message loss in the event of a failure.

michaeldrogalis15:02:31

Same goes for 0.10, expect messages aren’t stored in memory. We do a hard rollback.

acron15:02:52

Thanks for clarifying -- actually, the issue we're tracing is OOM. Just trying to get a handle on where things are being kept. It's a fairly straightforward kafka->s3 but it could be that s3 output isn't writing fast enough, causing Kafka to store a lot of msgs over time (we're on 0.9)

michaeldrogalis15:02:46

@acron Ah, got it. Input plugins have an upper limit on the number of messages they’ll buffer - controlled by :onyx/max-pending.

acron15:02:16

@michaeldrogalis Awesome, yeah, we're going to try tuning that and see what happens

michaeldrogalis15:02:57

@acron I’d recommend turning on FlightRecorder to try and figure out which part is inflating before you change much.

acron15:02:49

That's a good idea. We're trawling heap dumps at the moment and strings show 90% 😿

gardnervickers15:02:37

@acron Are you running in Docker?

acron15:02:16

@gardnervickers Yes, on Mesos

michaeldrogalis15:02:18

@acron Invest in good monitoring tools 🙂

gardnervickers15:02:36

Are you explicitly setting the max heap for both your peer JVM and the media driver JVM?

acron15:02:28

@gardnervickers Yes, we are - we've adopted and adjusted the onyx template

gardnervickers16:02:23

If you’re allocating shm space for your container, I believe mesos takes that out of your container’s memory allocation too so make sure you’re budgeting for that.

tcoupland16:02:05

Aye, we've dropped the xmx to .4 of available, media driver .2, allocated 2G in total. Just fiddling with the code abit to reduce waste, but suspect the max pending will be the real answer

gardnervickers16:02:43

Gotcha, is it the Mesos OOM killer or the JVM complaining about being OOM?

tcoupland16:02:07

It's the JVM one for now

tcoupland16:02:30

Theorising that it's backing up as we're outputting to s3

tcoupland16:02:39

So, the Kafka plugin is reading in 10000 msgs, and the graph to s3 can't clear them out fast enough

acron16:02:50

(approx. 1MB per message)

acron16:02:19

You love people using your tools in weird ways, right? ;D

gardnervickers16:02:49

Heh it’s definitely an edge case 😄

tcoupland16:02:24

Is there a metric for pending messages? That wld be a handy thing to track :)

gardnervickers16:02:07

Yup, there’s the pending_messages_count metric.

acron16:02:45

Excellent

gardnervickers16:02:45

I’m not 100% on the state of onyx-metrics right now wrt/ Onyx 0.10.x though

acron16:02:47

We're still on 0.9

michaeldrogalis16:02:54

I know you’re probably super busy and stuff, but I’d recommend jumping to 0.10 when you can. It’s not just a matter of better perf, it’s faaaar better tested. It’s seeing a lot of field action in Pyroclast, plus much more of it is property-tested.

camechis16:02:43

Oh that's interesting, might look at switching myself then

acron16:02:24

Sounds good

michaeldrogalis16:02:27

We’re going to stay on in beta for about 4 more weeks. There are 1 or 2 more things I’d like to get into the 0.10 release, but it’d be good to coast for a bit while folks switch over.

jasonbell21:02:52

Is there anything I need to setup on metrics for 0.10? I keep getting No method in multimethod 'apply-log-entry' for dispatch value: :signal-ready on startup.

lucasbradstreet21:02:51

@jasonbell hmm, that’s odd. Metrics should be automatically reported via JMX without anything on your part. You can also use onyx-peer-http-query as a prometheus endpoint https://github.com/onyx-platform/onyx-peer-http-query#route-2

lucasbradstreet21:02:08

You’re seeing that signal-ready exception on peer startup?

jasonbell21:02:56

I'm deploying on Docker so is there another port that needs opened up?

lucasbradstreet21:02:58

I think I can see what could be happening. The replica is showing your log version as 0.9.15. It should be throwing an error to say that the versions are incompatible and you should use a new :onyx/tenancy-id. Maybe it’s not performing that check because it’s a beta

jasonbell21:02:25

Ah right, cool. I'll try that. Thanks for the prompt response.

lucasbradstreet21:02:03

You will want to open up the port that onyx-peer-http-query is open on, so that you can get at the health/metrics endpoint, but these errors aren’t related to your metrics setup

jasonbell21:02:51

So I need to add

:onyx.query/server? true
 :onyx.query.server/port 8080
to my peer config as well then.

lucasbradstreet21:02:20

That’s right, and (:require [onyx.http-query]) in your peer entry point ns

jasonbell21:02:38

okay, thanks

jasonbell21:02:30

@lucasbradstreet got the basics in, just need to tweak docker things.

Insufficient usable storage for new log of length=50332096 in /dev/shm
thanks for your help this evening, much appreciated.

gardnervickers21:02:15

You’ll need to allocate some space for Aeron at /dev/shm, preferable memory.

jasonbell21:02:15

I usually set the --shm-size via the mesos deployment json, normally 512m, can ramp it up if needs be.

lucasbradstreet21:02:26

The shm size requirements have increased. I think we will need to recommend reducing the buffer sizes by default

jasonbell21:02:36

What is your default now? And which buffers to reduce?

lucasbradstreet21:02:43

16*1024*1024 bytes = 16MB, controlled via java property aeron.term.buffer.length

lucasbradstreet21:02:46

it must be a power of 2

lucasbradstreet21:02:01

The way that Aeron uses the buffers is 3*term.buffer.length per connection, on each the subscriber and publisher side. This is required for each task to task connection, so you can see how it could add up.

lucasbradstreet21:02:31

I’m going to have to run some tests before we release to make sure that the defaults can scale OK with only 1GB.

jasonbell21:02:24

okay, I'll sort that out over the weekend

jasonbell21:02:30

much appreciated, time to get some rest

lucasbradstreet21:02:45

Have a good weekend

jasonbell21:02:06

@lucasbradstreet you too sir