What happens with the scenario that the batch-write on the output plugin fails in terms of segment acking and retrying? Would that mean that the retry-segment will be called for all the root segments in the batch?

Yes, that's right

Does it use the bookkeeper in the background to achieve this?

It keeps the pending segments in memory and depends on the input source being replayable

What makes an input plugin replayable?

The plugin's ability to read from the input source more than once

yeah ok, so the plugin takes care of it.

thx

Yup. No worries

http://www.onyxplatform.org/docs/src/onyx/messaging/aeron_media_driver.clj

http://www.onyxplatform.org/docs/user-guide/latest/scheduling.html > For example, if you have 2 jobs - A and B, you’d give each of this percentage values - say 70% and 30%, respectively. If you had 100 virtual peers running, 70 would be allocated to A, and 30 to B. If you then added 100 more peers to the cluster, job A would be allocated 140 peers, and job B 30.

Seems that there should be >and job B 60

I’m interested in understanding how :batch-functions work as described here https://github.com/onyx-platform/onyx/blob/cf577d415a9b3f4016cfbf8675fb751a1671bb8d/doc/design/proposals/batch_primitives.md#batch-functions

Is there an example somewhere?

Batch primitives were cancelled in favour of better streaming techniques. We may pick them up again in the future

Ah ok, good to know. I thought i was missing something

I'm seeing 'Stopping task lifecycle' messages in the Onyx log, but my plugin doesn't seem to be seeing the seal-resource call

Curious - do 'input' jobs not get seal-resource?

I believe the idea is for seal-resource to be called to close off the output stream when the job is done e.g. in the case of core async write a :done to the channel

Ok, fair enough

I'm having a hard time understanding how best to 'close' my reader. I'd like to use a sentinel like ':done' but I end up just passing :done along the workflow, which isn't what I want, clearly. Usually, during read-batch I am returning {:onyx.core/batch [state]} where state is (t/input id message) - what should I return/do after I see that sentinel?

If I return nothing then onyx.api/await-job-completion hangs in my test

You need to make sure drained? returns true after there are no more messages left and all messages have been acked

drained? isn't being called

In fact, drained is only called when I return the :done key

https://github.com/onyx-platform/onyx-redis/commit/0b99a15e4f5195c71137dabeaeea91c08bbb09a0#diff-9ffbb5da032fbe15170ae3442fff900dL113

I realise this code is old, but I'm using it for inspiration

Oh I think you need to return done too

I believe the code works fine, we just dropped read support because we didnt want folks using it thinking that it could checkpoint/recover.

@gardnervickers: Yes, it does work fine, but my downstream workflow is receiving :done.

Is your downstream task receiving the done or is it being put on the medium by the output plugin?

It's weird, I have a my reader, then a function, then an output async channel

The async channel is returning me :done

I'm adding logging now to the function to see if that sees the :done

I've moved some lifecycle stuff around and the problem has disappeared now...but for a while I was definitely getting :done on an output async chan

Is :done used as a sentinel globally?

If my reader returned :foo would the effect be the same?

Yeah, you should get it on a core.async output channel, if you're using the core.async plugin

Nope, it needs to be :done

Ok, thanks again guys. The mist continues to lift, slowly but surely.

Any time

@rasom: Can you please open an issue on GitHub for those things? Happy to fix them if I can find them later.

@michaeldrogalis: ok

@rasom: thanks for the report, we definitely appreciate them

Is onyx a reasonable choice for a continuous integration style app? It's not computationally intense, but the work has many steps with a complex dependency tree.

@dg: Yeah, I think it would do well in that domain. The Windowing features should help a lot. We're missing first class iteration right now, but the workarounds are reasonable, and we expect to land iteration within 6 months.

Cool, thanks. Didn't want to dive in too deep if someone here said, "no it's just for big data processing, dummy"

Hehe, not at all. Happy to answer any questions, including beginner material.