Fork me on GitHub
#onyx
<
2017-11-10
>
jholmberg01:11:14

This may be a naive question but I'm gonna ask anyways. I've got JSON documents that are pretty big (ranging from 1K to ~16MB in size at times). With documents that large, what affect will that have on Aeron or Peers as they pass segments between each other as tasks process them?

lucasbradstreet01:11:20

16MB is going to be tough, as you’ll need 16*8*3 MB connections between the peers (one for each the publisher and the subscriber)

lucasbradstreet01:11:49

So you’re going to start getting pretty big in terms of how big your buffers are going to need to be, since each peer to peer connection will require one of those

jholmberg01:11:00

Ok, that's good to know. Seems like that wouldn't play too nice with the other jobs we're running. Most of the docs in the other jobs are around 1-10k as @camechis and I have been fiddling with buffers. Sounds like these docs may not be right for an onyx job possibly.

lucasbradstreet01:11:47

Yeah, or you may be better off putting the documents somewhere and passing around the reference to them.

jholmberg01:11:07

That was a thought I had. Pass a claim check pointing to where the docs would be located in google cloud storage maybe

lucasbradstreet01:11:53

Right. That’ll be especially beneficial when most fns aren’t even doing anything with the document, since you’d normally be paying a serialisation cost on every message

lucasbradstreet01:11:10

The other thing I’m open to is having a configurable channel size between two task types, so you could choose to allow bigger messages strategically

jholmberg01:11:13

Huh, that would be kinda cool actually. I'll talk it out with @camechis but my guess is for now, we'll look into our options first. Onyx is a perfect fit for the other stuff we're doing so naturally I go to Onyx first for data processing.

lucasbradstreet01:11:52

Yeah, you can also try to collapse down multiple tasks into one, where appropriate.

lucasbradstreet01:11:45

But yeah, passing around 16MB documents in streaming is something you need to think through

jholmberg01:11:09

yeah that's a big honkin' doc for streaming

stephenmhopper22:11:25

Hi, I haven't been tracking much with Onyx recently (been working on other projects). But I saw that 0.12 is underway. I was poking around the docs and couldn't find the "what's new" summary. Where is it located?