Fork me on GitHub
#onyx
<
2017-01-19
>
rc114005:01:36

@michaeldrogalis @mariusz_jachimowicz if you want feel free to guide/prod me in the direction that you are thinking regarding the new action and I can attempt to contribute the change

rc114010:01:54

also is it required to shutdown a peer group manually when closing/shutting down an app or is that closed by the virtue of no peers being around (the peers get shutdown first)

rc114011:01:15

thanks , nice and concise

michaeldrogalis16:01:16

@rc1140 That is working as designed. The peer group initializes shared resources for the virtual peers.

asolovyov20:01:38

Hey all! I've got a weird problem and I'm absolutely unsure how to tackle it: There is this seemingly usual job, which reads message from Kafka, makes a bunch of writes to Postgres and ElasticSearch, and then writes a message to another topic in Kafka. There are two of those jobs, and one works flawlessy. The other, though, is strange: it never updates its offset in Kafka, and processes same messages (I presume a batch?). Just over and over again, Any thoughts on what can cause that?

lucasbradstreet21:01:49

@asolovyov it sounds like messages aren’t getting fully acked and are getting retried

lucasbradstreet21:01:06

If you have onyx-metrics setup, you can check whether that job is seeing retries

asolovyov21:01:24

@lucasbradstreet oh, right, I'll try to add that to my charts

asolovyov21:01:27

I don't know, it looks like no retries: https://monosnap.com/file/KTJSxYL7WYNZJ0VoSuguQGKlc6OH6u also, from time to time it writes a bunch of feedbacks, but read throughput has weird pattern

michaeldrogalis21:01:36

@asolovyov Is the misbehaving job running on the same hardware with the same networking rules as the correct one?

asolovyov21:01:28

even allocated on some node right now

asolovyov21:01:47

I actually have an idea

asolovyov21:01:22

not sure why there are no retries, but serialization functions for kafka are different - I'll try to unify them and see what happens

asolovyov21:01:27

they are very similar

michaeldrogalis21:01:09

Cool, yeah keep us posted.

asolovyov21:01:04

btw, I have feeling that I asked that once before already, but still - why is throughput of input task is so much higher than throughput of output task?

michaeldrogalis21:01:14

@asolovyov Either because your job is stripping off segments before they each the output by using flow conditions, or your metrics aren’t reporting correctly.

asolovyov21:01:24

Interesting. So I have this job sending emails and metrics look like this: https://monosnap.com/file/DY6PyR5rF7q294781gQwN3O29QzRKu I'm pretty sure I send all emails I have to 🙂

asolovyov21:01:48

read-email is an input task reading from kafka, send-email is an output task to onyx-http

lucasbradstreet21:01:16

It’s also possible that you have one input peer, and multiple function/output peers, and you’re not summing the multiple peers throughputs

asolovyov21:01:41

that's my problem I guess :))

michaeldrogalis21:01:07

Metrics are great and absolutely essential - but be sure they’re not lying to you 🙂

asolovyov21:01:29

and it seems problem is a bit deeper and I guess my riemann config is to blame

asolovyov21:01:42

because in grafana I do sum...

asolovyov21:01:49

ok, I'll investigate that, thanks 🙂

lucasbradstreet21:01:20

Yeah, it’s either that you’re you’re filtering somewhere

lucasbradstreet21:01:31

But probably riemann

asolovyov21:01:49

heh, seems my guess about serialization function wasn't right 😞 it's a bit late here, will continue my search tomorrow 😕