Fork me on GitHub
#onyx
<
2015-09-15
>
noisesmith19:09:16

I'm tracking down an odd bug in our onyx server, which may or may not be related to the platform itself. Via request ids we have determined that (rarely) a request will get handled many times. Does this sound like a known onyx issue? just fishing for clues (also I am writing a stateful logger that will write error messages if it sees the same request id twice etc. to track it down of course)

michaeldrogalis20:09:57

@noisesmith: Onyx will retry a message if it doesn't complete within the default time window of 60s.

noisesmith20:09:31

can I adjust that window? this is a big task

michaeldrogalis20:09:34

Yeah. In the input catalog entry, turn up :onyx/pending-timeout to a value in ms

michaeldrogalis20:09:49

Its tracked at the root of the workflow, which is why its configured on the input task that it comes out of

noisesmith20:09:26

OK - this bug was driving me batty (so many places this behavior could have been coming from)

michaeldrogalis20:09:39

Haha sure thing simple_smile

michaeldrogalis20:09:07

Maybe log a warning and ensure that is the case.

noisesmith20:09:20

yeah, that's an excellent idea

noisesmith20:09:36

the problem was it was rare that the issue was triggered

michaeldrogalis20:09:57

What exactly is the task doing that it exceeds 60s?

noisesmith20:09:19

thousands of networked api calls followed by a bunch of graph calculations

noisesmith20:09:25

the task needs to be more granular simple_smile

noisesmith20:09:49

but thats on the "after it works" list - or at least it was

michaeldrogalis20:09:03

Haha nice. Are you finding things to be relatively stable so far? Curious about your experience.

noisesmith20:09:30

yeah, it's definitely been the case that it's just worked after I got things set up, this has been the first hiccup

noisesmith20:09:57

and the setup issues could have gone smoother with more time reading the docs and less time randomly typing

michaeldrogalis20:09:42

Its alright. I still do the same using infrastructure I didnt write 😛 Bash away until I hit an error, then go find the docs

michaeldrogalis20:09:12

Glad to hear it though. Give me a shout when you guys move to prod. Ill crack open a celebratory beer 😛