Actually this happens even if I have a single process running

The tasks are just repeating over and over again

at high speed

@spangler: See the docs: https://github.com/onyx-platform/onyx/blob/0.8.x/doc/user-guide/architecture-low-level-design.md Your messages are failing to complete, and are being replayed for fault tolerance.

Your segments are failed to be fully processed.

So ... I see the tasks finish

How could they fail, when the tasks themselves finish?

You're seeing "Stopping Task LifeCycle for blahblah" and all your data is coming back to you?

Yes, I see all the "Stopping Task LifeCycle ..." but the :done never fires

then they repeat over and over

Tasks will restart themselves if they encounter an exception. Maybe is throwing an exception on shutdown?

I noticed in your catalog you set :force-reset? to true, which would cause a full reload of your Kafka stream, so thats not surprising

Wait, so why would you ever set :force-reset? to true then?

When does it cause a full reload?

From the beginning of that topic?

because I create the topic for that job

Would it reread the entire set of messages for that topic?

That would explain it I guess....

It's a helpful option if you want to reprocess the entire message stream, if, say, you had a bug in your code that caused a running aggregate to be incorrect, and you need to see the entire stream from the beginning to correct it.

Okay, but would it be triggered during the job?

Like I say, the topic is created with the job

No.

So that's not it then

I suspect your job is failing to shutdown cleanly, and Onyx is resurrecting your job, and its starting from the beginning again, because thats what the configuration is set to do.

Where do I detect it failing to shutdown cleanly? How can I ensure it does?

I am not getting exceptions anywhere

It just at a certain point starts performing the same tasks at high speed

I'd try stripping down your job piece by piece and finding the part that's causing the problem.

Ah -- one thing... Hold on a sec.

We gave you that SNAPSHOT release yesterday to fix the :data key problem, but we knew that under some particular conditions, it swallowed exceptions. I fixed the problem entirely yesterday and pushed up a new SNAPSHOT.

Move to the latest, you'll probably see your exception there. Sorry - I forgot you moved to the edge.

Okay, here we go: FATAL [onyx.messaging.aeron.publication-manager] - Aeron write from buffer error: java.lang.IllegalArgumentException: Encoded message exceeds maxMessageLength of 2097152, length=5716465

So

How do I set the Aeron write buffer size

And thank you for not swallowing exceptions

Yeah, sorry about that. That was a bug in Timbre we found a while back

I havent seen that error before. You shouldnt have to configure Aeron like that.

How big of a message are you sending?

It says there

5.7 MB

I've not tested Onyx with messages that large before. I'd look at Aeron and see what you need to configure to allow you to do that.

They're very particular about what they offer by default to achieve such high performance.

Hi, is there deadlock detection within Onyx?

I submitted two jobs . the first one is working correctly . Onyx and kafka share the same Zookeeper .

after submitting the second job , I had zookeeper bad version error and everything hangs

No deadlock detection. That's weird that you get a zookeeper error. Are you using the same group-Id for zookeeper checkpointing?

You should probably use a different group-Id between the two jobs

first one uses kafka. second one doesnot

Ah, hmm

Do you have enough peers for both jobs?

yes

16 peers

Is the zookeeper error showing up in the ZK logs or are they in the onyx logs?

Onyx log

My vpn failed

Ahhhh

Using ZK over high latency / unreliable connections can suck

Yeah

Do messages have to be cleaned up?

If I want to to reset my cluster

Other than cleaning up ask

zookeeper

Not usually. You should be able to rejoin on the same id and the jobs should be auto killed. Let us know if you hit any cases where they aren't

Ok

Okay @michaeldrogalis @lucasbradstreet I have identified this error:

FATAL [onyx.messaging.aeron.publication-manager] - Aeron write from buffer error:  java.lang.IllegalArgumentException: Encoded message exceeds maxMessageLength of 2097152, length=2112492

So I need to raise that buffer size

But also, once it goes off onyx fails from this point on

Which does not seem very fault tolerant!

Can we catch this exception rather than failing entirely?

It should try to reset the publication after that point

Create an issue and I’ll look into whether a single big message can take it down

Hmm... would that be why my tasks just spin forever from that point on?

What does "reset the publication" mean?

What’s probably happening is that message never makes it through

Then after the pending-timeout it retries

same result etc

So the job never finishes

That seems like undesirable behavior

Well, what’s the alternative? Complete the job even though one of the messages never made it through?

The bet is that the error is transitory

Yeah, stop the job

Or at least let the user catch it?

Yeah, we do that for most things

and handle it?

The problem is that the error is in the messaging layer

As it is, they spin forever and I need to kill everything

It’s kinda hard to give you hooks there

Does that prevent us from catching it?

I need to be able to handle that exception, but it is closed to me

I think these are the settings you’re looking for

"-Daeron.mtu.length=16384" "-Daeron.socket.so_sndbuf=2097152" "-Daeron.socket.so_rcvbuf=2097152" "-Daeron.rcv.buffer.length=16384" "-Daeron.rcv.initial.window.length=2097152"

I don’t think you need to handle that exception, you need to reconfigure aeron

so you can send big messages

It seems like a significant error state that, once triggered, is unrecoverable

I will up the size

Create an issue for this anyway. I’ll definitely have a think about it

I can see what you’re saying

but there has to be a better way

Okay!

Cool, thank you @lucasbradstreet

I think the only thing we could really do is kill the job

Aeron will never be able to send the message

Right

That would be good behavior, then the app would not freak out and need killing

and I could fix it and redeploy without customers being totally distressed ; )

The thing to do here is to configure Aeron to handle messages as large as you can possibly send for your application. Onyx rebooting the job is desirable behavior. It's trying to get the work done at any costs. It has no idea that its going to process the same message again - it cannot make that assumption. We have :onyx/restart-pred-fn to conditionally kill the job when there's a task-level error, but we can't do much if there's a networking link error. We presume its transient and will be resolved.

If the job restarting is a problem even after you've adjusted the size, you're going to need to design in a defense of a rebooted job. Can't do much else atm, and it feels out of scope to try to go there.

I could design defensively, if I was able to tell from inside my code that something went wrong. The problem is the exception occurs outside of the scope of the task, so I can't handle it or respond to it

Its a hard thing to handle even if you could, because there's no good way to ask for the size of an object's byte count in the JVM

Is there some way to detect that a particular task is throwing that messaging exception (one we know is not transient) and not retry that task?

Or provide configuration for that?

The closest thing I can think to do is catch that specific exception type and do a string comparison for that one, particular error. Feels gross, but might be the right move

Are there other IllegalArgumentExceptions that you do want to retry?

It seems like an IAE will always be a permanent failure

That is not a network issue, it is a code issue

ie: not transient

Its transient if your tasks are non-deterministic, e.g. if they're doing a database look up and sending back results.

I'd say for now reconfigure Aeron to a very high limit, Ill think on it in the mean time

Okay, thanks for checking it out

And yes, there are IAE error we retry on purpose. Np

Okay, here is a new one I haven't seen before:

clojure.core.async/thread-call/fn                async.clj:  434
   onyx.peer.task-lifecycle/launch-aux-threads!/fn       task_lifecycle.clj:  449
                onyx.peer.pipeline-extensions/fn/G  pipeline_extensions.clj:    4
                   clojure.core/-cache-protocol-fn         core_deftype.clj:  554
java.lang.IllegalArgumentException: No implementation of method: :ack-segment of protocol: #'onyx.peer.pipeline-extensions/PipelineInput found for class: onyx.peer.function.Function

Any idea what that is about?