i’m trying to understand what exactly is going on here. after running an ETL task for a long time, I’m getting a kafka exception that it “could not locate any kafka brokers to connect to” — with recoverable? set to true after that, it appears as if onyx is restarting the task, but it never fully recovers

https://gist.githubusercontent.com/solatis/a3f99410891c1b1624569f93f5e66d13/raw/783af4762621c6a3c3f447ff58b76d33a27ed1f7/Onyx%2520kafka%2520recoverable

that’s the log there

shouldn’t that recoverable? be restartable? btw ?

i’m trying to understand what exactly is going on here

i think that the connection to kafka temporarily broke

recoverable is checked in the handle-exception lifecycle here https://github.com/onyx-platform/onyx-kafka/blob/0.10.x/src/onyx/plugin/kafka.clj#L336

which will allow the peer to restart and try to continue the job

aha, but, as i far as i can see from my logs, it’s not really restarted ?

or that’s a manual process

Ah, it looks like it’s happening during initialization, which is not something we allow it to be restarted from. I thought we had resolved that issue though.

That’s a bug for sure

i’m still running beta14

fwiw

anything i can do to be of help here ?

I’m logged an issue here https://github.com/onyx-platform/onyx/issues/789. I thought this was resolved, but I am going to have to track down whether it was fixed or not.

this is something that has affected us previously, so I will try to track it down asap.

👍

Yeah we definitely hit and fixed this in the last 8 weeks. We’ll get it figured out ASAP, sorry about this one.

no problem, this is not in production yet — it’s a poor dev server that’s under immense load all the time

i suspect i hit this bug because the system was swapping

causing kafka to grind into a halt

@lmergen Ahh, okay. Thanks for the report anyway, it’s high priority for us regardless 🙂

that’s great