Hi everyone

I'm trying to write a input plugin and for the moment it's working but the poll! Function seems to be called very very frequently like many times per second

So is there a way to configure the frequency of the calls to poll!

Or am I doing something wrong in the poll! Function like returning a wrong value resulting in unnecessary polling ?

Thank you

@dwarfylenain yeah this is by design, what you can do is simply invoke a Thread/sleep command inside the poll! function as long as it is not longer than timeout-ms (last argument passed to the poll! function)

Ah

Ok I see

Thank you I will try that

i think this is the place where this poll! is invoked: https://github.com/onyx-platform/onyx/blob/0.10.x/src/onyx/peer/read_batch.clj#L39

I mean in this case it doesn't really matter to me because I'm receiving the input data as a stream so it doesn't "cost" me much to poll and see if there is new data in my buffer

yeah

i believe there was some discussion of implementing a proper backoff strategy here

But my next input plugin will actually call a rest API where there is rate limiting so

but as of now, it's not there

Yo but the sleep thing should work for me

@dwarfylenain i think the best thing to do is keep track of the rate limiting inside your poll! function, but just return nil when you're currently rate limited

when you're blocking inside the poll! function for too long, bad things could happen (since you're also blocking other lifecycle activities)

Oki I see, it makes sense, thank you

yw

Ah

Just a last question

The timeout you are talking about is the value you provide in your catalog in :onyx/batch-timeout

Or it has nothing to do with it ?

yes that's the same

https://github.com/onyx-platform/onyx/blob/0.10.x/src/onyx/peer/task_lifecycle.clj#L576

Ok, great 🙂

but, if you set it too high, it's effectively the same as just 'blocking' the thread

I digged into the docs but not really into the code itself

Ok so I just return nil except if I know I can make a “new” call the API for fresh data (like I counted that since last call enough time has passed)

yes