Fork me on GitHub
#onyx
<
2016-03-31
>
otfrom08:03:10

any good examples of writing size/time windowed output to s3?

jasonbell09:03:38

Spooky, I was about to ask the same question.

lucasbradstreet10:03:07

We don’t have any samples of that, but you could implement it pretty easily with triggers: e.g. https://github.com/onyx-platform/onyx-examples/blob/0.9.x/fixed-windows/src/fixed_windows/core.clj#L109

jasonbell10:03:02

Thanks @lucasbradstreet

jasonbell13:03:26

Nice to see a complete suite of tools plus a learning platform, nice work indeed.

jeroenvandijk13:03:13

@otfrom @jasonbell I’m also thinking of building something that writes to S3 that way. I've started playing with Onyx just today so not sure if/when I can be of help

jasonbell13:03:36

@jeroenvandijk: thanks for letting me (and @otfrom) know. I’ve just started looking today too.

jeroenvandijk13:03:30

Wild idea: use onyx to create an http load testing tool? Sounds realistic? Maybe something like http://engulf-project.org/ would simpler to implement with Onyx. Would be a nice example application that would be rather useful

michaeldrogalis14:03:34

@jeroenvandijk: Definitely could. I had some ideas about building a simulation testing suite with Onyx. Not enough hours in the day. simple_smile

jeroenvandijk14:03:56

@michaeldrogalis: Cool simple_smile Yeah I was mostly interested in the potential of the idea. I might try it at some point as it sounds pretty useful for us. Engulf is good, but still limited and is not maintained anymore.

jeroenvandijk14:03:17

But I should probably first finish learn-onyx before fantasizing further

michaeldrogalis14:03:08

@jeroenvandijk: Wise decision. simple_smile

ckarlsen18:03:25

Should ack-segment (PipelineInput) ever receive nil as segment-id? Seems to happen with my input plugin when :onyx/pending-timeout is reached. Every pending segment just retries forever.

lucasbradstreet18:03:51

nope, it definitely shouldn’t. Which plugin are you seeing this in?

ckarlsen20:03:44

lucasbradstreet: sorry! the segment id is not nil, but the segment does not exist in pending-messages. This is a custom input-plugin. It seems like retried segments never get acked after pending-timeout is reached

ckarlsen20:03:01

dummy example

ckarlsen20:03:37

i dont understand what i'm doing wrong in my input plugin

lucasbradstreet20:03:04

It’s possible that they’ve already been retried by the time the ack-segment comes in, in which case you wouldn’t be able to dissoc it. You would ignore it in that case

lucasbradstreet20:03:54

That may not be your problem though

lucasbradstreet20:03:08

Is it possible that you’re acking slow enough that you main read-ch becomes overly full and then your retries blocked putting onto the read-ch?

lucasbradstreet20:03:13

Do they ever get read again from read-batch after a retry?

ckarlsen21:03:28

i have a logfile from that dummy example which shows exactly how it behaves

ckarlsen21:03:51

just trying to find somewhere to dump it

lucasbradstreet21:03:55

OK, sorry, I won’t be able to help much more today. It’s late and I’m off to sleep simple_smile

ckarlsen21:03:27

thanks anyway simple_smile I'll let you know if I figure it out

ckarlsen21:03:03

(<!! (async/timeout 1000)) (>!! read-ch (t/input (random-uuid) (:message segment))) (swap! pending-messages dissoc segment-id)

ckarlsen21:03:12

with a timeout all retries are acked

ckarlsen21:03:12

off to sleep myself. I'll experiment more tomorrow and would love some tips on how to implement this properly simple_smile

michaeldrogalis22:03:18

@ckarlsen: We have a new interface coming for input plugins next release. Should make writing your own I/O plugins waaay easier.

michaeldrogalis22:03:46

Will probably be out in in beta in ~8 weeks though, so worth it to write your own now since a good 80% of it will carry over.

gardnervickers22:03:04

We do have the SimpleInput plugin

michaeldrogalis22:03:56

True. That's already in 0.9.0