Which test-helper function tells the required number of peers? The link in the guide is broken

@lucasbradstreet After spending some more time thinking about why I was hitting GC issues while trying to pump 41 million integers through Onyx using a lazy sequence, it occurred to me that although I was evaluating the sequence lazily, the whole thing was likely still available in memory in its entirety even after elements had been consumed. Is this accurate? If so, what's the best way around this?

@stephenmhopper: I'm pretty sure that the onyx seq plugin uses doseq which won't hold onto the seq head.

Yeah, I was poking around in the source and just saw that: https://github.com/onyx-platform/onyx-seq/blob/0.9.x/src/onyx/plugin/seq.clj

So now I'm back to scratching my head

What do you have your batch size set to?

Batch size is currently at 1000. Is that too big?

Should have asked how large the segments are first

😄

integer size?

Each segment is something like {:n 100}

So, they're small

@stephenmhopper I'm curious because I've just read the production instructions, are you using G1GC?

@yonatanel I'm using whatever the default is. I'm just doing local development in a REPL with the with-test-env setup, so I hadn't considered looking at the prod settings yet

Can you share the code to reproduce this?

I'll rerun with G1GC and see if I can reproduce the issue

Super small segment sizes are worst-case for not only messaging but for the GC. If you're trying to get a feel for performance, it's difficult to do so using the test environment because it differs so much from a multi-node setup.

@gardnervickers Yeah, I figured that was the case which is why I'm not too worried about this whole thing. I mean, it managed to crank through 21 million messages locally in an acceptable time frame

That said I'm sure it can be tweaked to do what you want, it just might take some extra investigation considering the JVM on your local machine offers no extra resource margins.

The user guide offers some great tips here, especially in the Performance Tuning section http://www.onyxplatform.org/docs/user-guide/0.9.10-beta2/#performance-tuning

Hey 🙂 I'm working through learn-onyx — really well done stuff 👍 —and I'm a bit stuck at 4-2. 1) The text says I should use :lifecycle/before-task-start but in that lifecycle I don't have any visibility on the segments going through the system. Is it intended that I somehow inject the atom in a place where I can access it in the task fn? 2) all the core async lifecycles are missing, I just copied them for now because I think I need them but really I have no clue. If they're needed it seems slightly outside of the scope of the excercise?

@martinklepsch Hello! Point number 1 - yes, injecting the atom is the solution. You’re right with point 2, they are needed, and it probably is slightly out of scope. The intention is to have the learner add all the lifecycles themselves, but perhaps it’s a misdirection.

Just in case you need a look at the solutions, the answers are on the answers branch, too 🙂

ah right, forgot about the answers branch, ha!

Thanks @michaeldrogalis

@martinklepsch If you read the README at all you’re already beyond most people 🙂 Alex Miller was right when he said, if you want someone to read something, write it down multiple times.

Haha, so true

Even exception messages aren’t usually enough 🙂

Question regarding :flow/predicate. From the docs it looks like I can do things like this: :flow/predicate [:and :my.ns/female? :my.ns/athlete?]. Can I do something like this too?: :flow/predicate [:not :my.ns/female?]

@stephenmhopper Yes

See this example: https://github.com/onyx-platform/onyx-examples/blob/0.9.x/flow-predicate-composition/src/flow_predicate_composition/core.clj#L44

Cool. Thank you.