@greywolve Right, so did you have to add 10 corresponding peers? I'm guessing it's running on your own machine alone?

The solution is to increase the timeout

This can often pop up during long GCs too.

yup, added the peers, batch timeout? our max pending is also set pretty low

If you look at the notes listed in https://github.com/onyx-platform/onyx-jepsen/blob/master/README.md, you could use similar settings to those. Maybe reduce the timeout a bit. I think it's set to 60s

Err 50s

How many peers total now? 25-30? All running on your lone machine?

39 😛 , ahh right the aeron timeout itself

Hah that's a lot of tasks for one machine to handle. No wonder you're hitting issues.

I'd consider rationalising/fusing some where it makes sense

haha, ok then i'm just being silly i guess

good point

btw, do you think running two datomic read-log tasks, in separate jobs, is wise? i actually thought that was causing my issues, and ended up merging the 2nd job into the first, to share a read-log, and transact

does that matter?

It'll add a bit of load because you're reading multiple times, but you get the advantage of decoupling the jobs, which can help a lot with retries

Given how many tasks you have in your job I would definitely consider doing it

If you do make sure they're not both check pointing to the same key

i actually had it like that originally, then i thought this aeron issue was due to that, lol. thankfully i kept the other service as is, so i'll just revert back

yup i made sure of that

Cool. I think it makes sense to split them again. It's easier to unit test/reason about too

I think it's a good idea to still increase the timeout and switch to shared mode media driver

will do, you mean the stand alone driver? even for local dev?

I think I'd switch to shared mode in both the stand alone driver (prod) as well as for you local dev

Dedicated can achieve better throughputs but at the cost of burning CPU and your throughput is rather low for what Aeron can handle

We've switched to shared mode by default

awesome, i'll try it out, thanks

You're welcome

feel the learn

@lsnape: I think I’d benefit from having a quick look at your onyx.log first

@lucasbradstreet: wrt the issue I just posted on google groups: https://groups.google.com/forum/#!topic/onyx-user/6s7VNT6iloM I'm going to wipe my onyx.log and submit the offending job again...

Ok, sounds good.

Hmm, wow it doesn’t get very far at all

I start the system with 4 peers. Like I said, the sample job runs fine but modified one does not

Do you have more than 4 tasks in the new job?

no, there's only input and output

:workflow [[:read-messages :write-output]]

my kafka topic is configured with 8 partitions, but I think that's irrelevant

I have a feeling that you’re submitting the job to a different ZK cluster than the peers are looking at

or with a different onyx/id

It seems like the peers are just waiting around and never see the submit-job

Though it does seem to be written successfully to ZooKeeper

so :kafka/zookeeper is set to the prod zookeeper instance

the peers are looking at local zk

that should be fine I think

so why might the onyx id be different. I'll take another look..

:onyx/max-peers 8, :onyx/min-peers 8,

That is concerning, given that you only have 4 peers

ah yes, your right. I'll change number of peers to 8. I think I've run it with 8 peers before though and had the same problem. Will try again now though

I believe the scheduler will see the submit-job entry and not schedule it because there aren’t enough peers yet

You’ll need more than 8, you’ll need at least 1 for each task (except for those that set a minimum, which set a minimum). So in your case I think it would be 9

okay something has happened! The messenger buffers now start and I get exceptions further downstream

Great. Progress

ah it's complaining about a missing symbol. This kind of stuff is expected. I should be alright from here I think

You may also be interested in our new template (currently it’s only a snapshot release). We’re still iterating on it, but it uses some of the latest best practices. https://github.com/onyx-platform/onyx-template/tree/feature/new-idioms

oh nice, I'll check it out

Cool. Sounds like you’ll be good from here then. Feel free to come back with any other issues

Thanks for your help :thumbsup:

You’re welcome

You hit a common problem that we’d like a validator to deal with when deving locally. It doesn’t make sense to throw an exception in prod because you might just be waiting for more peers to come up, or other jobs to finish.

yeah, so I guess that would be a case of scanning the workflow and catalog to find the minimum number of peers required to run the job?

And checking it against the current number that are running in the current peer coordination replica. It would only be useful in dev

@lsnape: I will have some updates for that template in about an hour, hopefully it’ll make it a little clearer

@gardnervickers: awesome. I aim to give it a whirl this afternoon

@lsnape: you may need to lein install it manually and use lein new onyx-app proj-name —snapshot to make it work

grr, annoyingd that it converts - - to --

huh

thanks again @lucasbradstreet , your suggestions made my machine handle both services it seems - no starvation.

👍