@asolovyov

:onyx.peer/subscriber-liveness-timeout-ms 
 :onyx.peer/publisher-liveness-timeout-ms 

The default is 10 seconds, I've had to extend it out to 30 seconds for more involved jobs.

@jasonbell that's interesting, thanks, I'll look into that; it's certainly not set right now

no problem at all

Hey all! I wonder how onyx serializes segments to aeron. Does each segment become separate aeron message or could multiple segments be grouped into one aeron message? I'm asking because I want to understand where "Encoded message exceeds maxMessageLength" problem might be coming from. We're don't seem to have segments larger than 2mb

Aeron term buffer determines your segment size

16mb term buffer / 8 = max segment size

which in this instance is 2mb

if you adjust the aeron.term.buffer.length in the java options (using -D) then you can extend that

thanks, I'll try that. Still not sure where such big segment is coming from, though.. want to get to the bottom of it

@jetmind Segments are the user-facing items that you deal with in your task/aggregation/flow-condition functions. These are batched together using :onyx/batch-size.

yeah, I understand that. Just wanted to figure out how they correspond to the underlying messaging layer. So, it theory one batch of segments could be sent through the wire as single Aeron message?

Yes, the Aeron offer is a batch of segments https://github.com/onyx-platform/onyx/blob/5e011d2cadea05bc0e66ca32fd34fd80ba14f323/src/onyx/plugin/messaging_output.clj#L110-L116

So if all your segments are 2kb+ in size, then a :onyx/batch-size of 100 of them would put you over the 2MB limit.

We have onyx/batch-size set to 10 but I think it’s possible for some of our segments to exceed 200kb+, so probably that’s what puts us over the limit

thanks for clearing that out for me!

@gardnervickers I know the max Aeron term buffer is 1GB, has anyone tried segment sizes of 128mb?

Not that I’m aware of

Cool. I'll let you know how it goes then 🙂