@lucasbradstreet: I notice you've no license on https://github.com/lbradstreet/onyx-visualization; any plans on adding one? 🙂

oops. Will do. Thanks!

Are you thinking of using it for something?

Done. Made it MIT so people can easily use it for whatever they want

@lucasbradstreet: Thanks - yes, I intend to use/adapt it 🙂

Cool, for something user facing/dashboard/etc?

Yep

Feel free to steal it completely, or send us improvements

Thanks; I'd cooked up some graphviz style ones but then I recalled this

the css is still a little wonky with the task display

On a different subject entirely, what would cause this?

16-07-11 10:32:22 acron-ubuntu INFO [onyx.messaging.aeron.publication-pool:79] - Creating publication at: {:channel "", :stream-id 1, :acker-id 18557, :peer-task-id 20607}
16-07-11 10:32:22 acron-ubuntu WARN [onyx.messaging.aeron.publication-manager:47] - Writing nil publication manager, likely due to timeout on creation.

I’ve only ever seen that on resource starvation e.g. long GCs

I'm seeing this for each peer, by the looks of it

This is local testing

hmm

How many peers?

vs how many cores on your machine?

Oooh

Way more peers than cores

Yeah, it could be easy to starve the peers in that case

you can try creating a bigger batch timeout so they’re not trying to do things all the time

Ok, will try, thanks

@lucasbradstreet: Is it fair to say that I could be encountering a job size that's simply too large to run on a single machine?

Certainly

Crap

There are probably some other things you can do to tune it, but it sounds like your job would need to be tuned/restructured to run on one machine

I'd want to use flight recorder / Java mission control to figure out what's consuming all the resources

Enabling the G1GC and setting the aeron liveness timeout higher can help

Sounds like a level of tuning outside of my comfort zone 😕

Also if you're running it at the repl it'll be using bad JVM flags to go fast

Ok, I'll try jarring it

Seems fine at 15 peers but beyond that, no luck

I need a lot more than 15..

Too many tasks in your job?

Yeah, seems so...

It doesnt even attempt to start though, which is odd

Are there any you can collapse? Common candidates are where A -> B can be collapsed into a single task

Do you have more than the minimum required peers started?

e.g. one per task + min-peers (if more than one) for each task

I have n-peers 1 on everything

This is literally just to run in a clojure test

So, say 20 peers, and 20 tasks and the job won’t start?

https://www.refheap.com/7da8234968c1a6ce28327e8fb

24 tasks

Would it be :onyx.messaging.aeron/publication-creation-timeout to fix that timeout?

That might help, yeah.

peer conf?

peer-conf, yes.

🙂

Also JVM OPTS: -XX:+UseG1GC -server -Daeron.client.liveness.timeout=50000000000

Not sure why it even has to connect, since it should be short circuiting everything

I notice in my config :onyx.messaging/allow-short-circuit? false

Yea try changing that 🙂

Does that pertain?

I inherited this config so not sure why these settings are how they are

That’s mostly to test whether your configuration is right when you go multinode, but it’ll kill perf on a single host

Is there any way to figure out the :flow-to in a event parameter inside a predicate fn called by a flow condition?

@luiseugenio: I don’t think so. It’d be nice if the flow condition map was passed into the predicate, since that would help parameterise the flow conditions better. I feel like this should be possible, but it doesn’t look like it

@lucasbradstreet: , ok, thanks

@lucasbradstreet: aeron warnings gone, now just no action after Enough peers are active, starting the task..

CPU @ 100%

try making :onyx/max-pending really small on your input tasks

just to make sure it’s that you’re not pushing too many segments through and they’re being constantly retried

now would be a good time to setup metrics

Ok, thanks

Is there much in the way of debug logging?

I added :onyx.log/config {:level :debug}

a bit. you’d at least be able to see how big the batches are

But no change

hm

I would setup timbre onyx-metrics. I would also try running jmc (java mission control) and attach to the running process and profile it a bit

to see where the time is going

prob do onyx-metrics first

I did try jmc but I've not used it before and the best I could get was that it spent most if its time in the 'clojure.lang' package..

Hmm, it's doing a couple of tasks now and then stopping

This is strange

If I reduce the number of tasks it works

strange

I'm investigating something else

@lucasbradstreet: Is there any chance that including the flow condition map inside de event map to become a wishlist feature ? 😊 I’m trying to use a rule engine to decide where to go in flow conditions, and if I could call only one predicate function (without having to pass some kind of hint of :flow/from and :flow/to) would be fantastic.

@luiseugenio: It's in the Event map now. Under keys :onyx.core/task-information -> :flow-conditions.

@michaeldrogalis: Ah, not all the flow conditions, but the map that contains the specific :flow-from and flow-to this event was called. Supose I have a workflow [:a :b] [:a :c] and the flow conditions maps are: {:flow/from :a :flow/to [:b] :flow/predicate ::can-i-go? } and the same for :a -> :c. So I could call a rule engine to decide (knowing the :flow-from and :flow-to for each case the predicate is called) where to go. This could be a stupid idea, anyway, but I could change the behaviour based only in the segment data and where is it trying to go.

Ah, right. Sorry, I cant think of an easy way to get at that data yet.

I think it’d make sense to pass the specific flow condition in. We do a similar thing with windows and triggers

It’d be another breaking change though 😕

Probably a change for 0.10.0.

@michaeldrogalis: and @lucasbradstreet , ok, no problem. I’ll do it another way (passing a parameter manually on each flow condition entry and later, if this could be gotten inside the event map, I’ll change the predicate function. Thanks.

@luiseugenio: Sounds good!