My onyx dashboard is struggling to load all the log events after about #1300. Is there a way to trigger some kind of log compaction?

Update: it "aw, snap!"ed on me

update: I figured it out: onyx.api/gc

I have another question: I'm trying to use the balanced scheduler, but I'm seeing (I think) the same behavior as the greedy scheduler. There are 30 virtual peers started up. I'm submitting two instances of a job with 13 different tasks each. They both submit according to the onyx dashboard. But the peers don't recompute their task distribution (the second one doesn't start) until the first one is killed. What's the best way to debug this? Am I correct that that is unexpected behavior?

Reg. Documentation, thinking out loud, will it be better to start off with “user guide” (users of the onyx, API docs, how-to), “contributor guide” (internal architecture, nitty-gritties, design), “admin guide” (deployment, performance) etc ..

That sounds good. User guide section should include links to learn-onyx

@aengelberg: You can confirm the number of vpeers each job needs with: https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/test_helper.clj#L33

I'd expect both jobs to be running if 30 peers is enough to cover both with a balanced job scheduler.

@jmv: Did you end up tracking down what was going on?

@michaeldrogalis: From your response above ^^^. If you had a job that needs 15 peers and you launched 2 of those. Does that mean you need 30 peers?

@camechis: Yup, that is correct

I thought so I just wanted to confirm

thnx @dignati

The balanced job scheduler worked as expected on a totally fresh zookeeper. Something weird must have happened with the onyx log state and having used the greedy scheduler previously.

@aengelberg: Yeah, you can't switch job schedulers per tenancy.

@michaeldrogalis: not yet, but i'm narrowing it down. i will update when i figure out what's going on

@vijayakkineni: How are you running Aeron?

@gardnervickers: Its a very basic configuration {:onyx.peer/job-scheduler :onyx.job-scheduler/greedy :onyx.messaging/impl :aeron :onyx.messaging/peer-port 40200 :onyx.messaging/bind-addr “localhost"}

I mean in-process with regards to Onyx our on its own

I dont have an external aeron, i am following onyx starter template .

What does all-new mean in the context of flow conditions? I'm reading http://www.onyxplatform.org/docs/user-guide/latest/flow-conditions.html ... also, why do predicate function signatures need four parameters - a context map, the old segment, the new segment, and the collection of all new segments produced from the old segment?

I'm attempting my own flow condition function, but I'm not understanding what the appropriate function signature should be...

@aaelony: all-new = the collection of all new segments returned by the function. e.g. (fn [x] [x x x]). If your function only returns a single segment, it rolls it up into one a collection for all-new. Useful for when predicates need to make decisions based on the relationship between an old segments and all new segments together

hmmm, I basically have a mixed bag of event types, and I have a function that tells me if this is an event-type that I care about. My function returns true if I care about it...

what should all-new be in my case?

if you have a task f(x)->y then you’ll see [event x y [y]]

@aaelony: Onyx invokes the predicate function, you don't supply those parameters.

ah okay, that helps @michaeldrogalis

and the predicate logic that returns a boolean acts on the event, not x or y...

@aaelony: You probably just want to make your pred function check a key against the new-segment.

From the sounds of what you described

that's what I see in the docs, but I am not groking that

Would recommend some the onyx-examples to get the hang of it

okay, will do

also, is maxMessageLength something I can set for Aeron? I'm seeing

16-Aug-09 15:13:48 ip-10-63-208-42 FATAL [onyx.messaging.aeron.publication-manager] - Aeron write from buffer error:  java.lang.IllegalArgumentException: Encoded message exceeds maxMessageLength of 2097152, length=24667975

maybe this is a java Aeron thing...

please ignore -- I think that if I correct my flow condition, these payloads will be smaller and the message length will fit.

I also found this...

"-Daeron.mtu.length=16384" "-Daeron.socket.so_sndbuf=2097152" "-Daeron.socket.so_rcvbuf=2097152" "-Daeron.rcv.buffer.length=16384" "-Daeron.rcv.initial.window.length=2097152"