Fork me on GitHub
#onyx
<
2017-07-24
>
lmergen07:07:42

which of the configuration options influences the aeron service call timeout ? trying to make this error go away on a heavily loaded dev environment: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns

lucasbradstreet07:07:13

I'm off to bed, and can't look up the exact property that will correspond to that message right now, but the timeout will definitely be listed in https://github.com/real-logic/aeron/wiki/Configuration-Options#media-driver-options

lucasbradstreet07:07:41

You can configure it via a Java property. Look for one that will correspond to that many ns

lmergen11:07:44

ok, thanks!

wildermuthn13:07:52

Hey all! I went through the onyx challenges this weekend, and very grateful for the tutorials. Very helpful! I have a few questions about best-practices and patterns RE: input tasks. I have a concrete use case for a personal project, which is input JSON from a long-polling http request. It looks to me like the use of a crontab to submit a job would fit best with the lein template app, but I have also seen talk about jobs that never stop (with what I assume is an :in task that is on some kind of trigger/timer). What’s the best way to have a long-poll type of input task?

michaeldrogalis15:07:35

@wildermuthn The long-polling story is a little bit awkward because Onyx is expecting that input plugins are communicating with a medium that durably stores results and can recover. IMO doing something like long polling then dumping into Kafka, then feeding that into Onyx would do well.

michaeldrogalis15:07:02

Is the idea that each job ought to execute over a batch of long-polled results? Trying to connect what you’re saying with Crontab.

wildermuthn15:07:58

Right, so I make a large amount of http requests to an external API every 30s, and want to process all the resulting data with an onyx job.

wildermuthn15:07:14

You’re saying, perhaps run those http requests outside of onyx, and put it into Kafka?

michaeldrogalis15:07:55

Yeah. Those two activities sound pretty distinct. Kafka might not be the right storage choice now that I have a little more insight.

michaeldrogalis15:07:11

You could plop them into S3 and use the S3-file reader plugin to handle them.

wildermuthn15:07:32

Ok, but the idea is the data should go somewhere durable so any peer can grab from it? Gotcha.

michaeldrogalis15:07:13

Yeah, mostly so that if there’s a failure of any kind, you’ll be able to recover the original data.

wildermuthn15:07:16

In terms of the crontab, I’m looking to understand the best way to submit a job to the peers, whether it should be submitted and then hang until data is avaiable (via Kafka/S3), or whether something should trigger the job externally (like a crontab)

michaeldrogalis15:07:21

It’ll just be plain easier for debugging from a systems perspective, too

wildermuthn16:07:35

@michaeldrogalis For pyroclast’s roaming, can arbitrary functions be included in the service composition?

wildermuthn16:07:38

or, is it restricted to the functions provided by the roaming sdk?

wildermuthn16:07:57

ok cool. thanks

stephenmhopper17:07:14

So, I'm getting this error in one of our deployed environments on application startup: Exception type: io.aeron.exceptions.RegistrationException. Exception message: Insufficient usable storage for new log of length=50332096 in /dev/shm (shm)

stephenmhopper17:07:33

I'm starting the app using with-test-env because we've currently no need to run Onyx in distributed mode

stephenmhopper17:07:50

It works great locally, but apparently this box lacks sufficient space on /dev/shm

stephenmhopper17:07:28

I tried setting :onyx.log/file "/var/log/onyx.log" in both my env-config and peer-config. Is that ignored for with-test-env? What should I do?

lucasbradstreet17:07:50

@stephenmhopper how many peers are you running on this node? If you have lots of peers connecting to each other, you’ll find that they each need a buffer in /dev/shm and you use up your space pretty quickly

lucasbradstreet17:07:34

@stephenmhopper you can either increase /dev/shm’s size, or you can decrease the term buffer size, e.g. -Daeron.term.buffer.length=4194304

stephenmhopper17:07:34

There's 8 logical peers

lucasbradstreet17:07:52

Hmm, that seems odd that you’d run out of space that quickly. Are you using docker?

lucasbradstreet17:07:54

I’m guessing so

stephenmhopper17:07:17

Yeah, it's a docker container

stephenmhopper17:07:35

It has plenty of other space on other partitions though. Is there a way to use one of those instead?

lucasbradstreet17:07:39

Try docker’s --shm-size option

lucasbradstreet17:07:59

Default is 64MB I think, which is pretty much never going to work

stephenmhopper17:07:18

yeah, I'll set it to a higher value, thank you!

stephenmhopper17:07:39

also, I'm reading from and writing to AWS SQS queues and I'm getting permissions issues for my app profile

stephenmhopper17:07:52

I'm not sure if it's just a red herring though because of the aforementioned /dev/shm issue

stephenmhopper17:07:26

I'm giving the profile ["sqs:SendMessage*","sqs:ReceiveMessage","sqs:DeleteMessage*","sqs:ChangeMessageVisibility*"] does it need any other permissions on the queues?

lucasbradstreet17:07:48

I’m not sure. I haven’t really looked into the permissions for a while

lucasbradstreet17:07:04

Can you let us know how you go there? Once you figure it out I’ll add it to the docs

stephenmhopper17:07:07

@lucasbradstreet So, the --shm-size option fixed the first error. But now I'm still getting the AWS error about either not having permissions or specifying the wrong queue name. I'll let you know what I find