hi all this may be a silly question , but if i want a task to receive a batch of segments , all i should need to do is set the batch-fn property to try right ?

the basic idea is that we are processing segments and then collecting them to be emailed , but the batch task isnt receiving a vector of segments , rather its just recieving individual segments

@rc1140: yep, that's right. Batch fn is what you need. Just make sure you return your data in the way described by the docs

See https://github.com/onyx-platform/onyx/blob/0.9.x/test/onyx/peer/batch_function_test.clj as an example

cool , let me setup a small demo to test with , possibly an issue with code or workflow

i assume the fact that we are reading from and writing to kafka shouldnt change anything

Yup. End result should be the same

is there a way to have an undefined batchsize , i.e just continue sending segments until timeout

so the issue atm is that it if i create a spy function as a test (as defined in the docs) and set it up as a batch function it only prints out single empty values until i kill the job , I have checked the value from the previous task is a vector of maps but the amount of items printed in the spy functions are way more than returned by the previous step

also the segment printed in the spy function is a "()" [i think thats an empty list]

seems like anytime i setup a function as a batch function is recieves empty (or rather 0 length segments)

@michaeldrogalis Have you had any legal issues using Eclipse or GPL license libs in Pyro?

@yonatanel Haven’t, nope.