This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # bangalore-clj (1)
- # beginners (158)
- # boot (8)
- # cider (9)
- # cljsjs (9)
- # clojure (169)
- # clojure-austin (1)
- # clojure-denmark (1)
- # clojure-dusseldorf (5)
- # clojure-italy (9)
- # clojure-losangeles (2)
- # clojure-russia (31)
- # clojure-spec (53)
- # clojure-turkiye (1)
- # clojure-uk (56)
- # clojurescript (145)
- # cursive (72)
- # datascript (4)
- # datomic (3)
- # duct (121)
- # events (9)
- # figwheel (1)
- # fulcro (46)
- # graphql (4)
- # hoplon (16)
- # jobs (1)
- # jobs-discuss (4)
- # leiningen (16)
- # lumo (5)
- # off-topic (38)
- # om (1)
- # om-next (5)
- # onyx (104)
- # parinfer (5)
- # re-frame (106)
- # reagent (1)
- # ring-swagger (3)
- # rum (1)
- # shadow-cljs (235)
- # slack-help (4)
- # unrepl (25)
- # yada (9)
Hmmm, ok. Maybe I've been thinking about it wrong. Should I, in general, always assume the possibility of duplicated state getting emitted? Or is this a special case?
Well, as a separate point, we can’t guarantee exactly once side effects since that’s impossible, so in that respect you should assume the possibility of sync being called more than once on the same state.
Ok. In that case it’s certainly possible for it to emit more than once if a node fails and it has to recover.
I would also assume that it may be called more than once when the job is completed, because you generally need that final seal to flush anything new that didn’t get triggered by the number of elements
Ok, good to know. I still have a lot to learn about what's going on behind the scenes. Thanks for your help.
I'm just afraid this stuff is a little bit too deep inside for me to comprehend what exactly is wrong 🙂
yeah. So the test is that for one of requests I want to go further than retry-params allow, which should generate an exception
I guess if the job got killed it would have finished with feedback-exception!
also, I figured out that
Thread/sleep seems to work properly - when a few requests try to back off, they don't mess with each other, which is what we need
hmm, shouldn’t have to. What do you want to happen when the exception happens?
yeah, if I call ack-fn, job finishes, but exception is never propagated... I wonder if there is a better place for propagation than throwing it in
oh wow, but I get this in my onyx.log:
clojure.lang.ExceptionInfo: Handling uncaught exception thrown inside task lifecycle :lifecycle/write-batch. Killing the job. -> Exception type: clojure.lang.ExceptionInfo. Exception message: HTTP request failed!
OK, so we setup all of our async calls in write-batch, and progress to the next state because we returned true
At some point we receive a barrier, and we have already made all the write-batch calls so all we have to do is wait for the sync
so we’re stuck in a situation where we never return true, and because we only check for an exception in write-batch it never gets thrown
so what's the right way to deal with that? do not return true in write-batch until all async requests are finished?
You could do that, or you could just check for failures in sync and completed too
do I need to track somehow that I throw an exception for correct batch or something?
Onyx appears to be propogating the exceptions properly, it just never got the chance to throw one.
This is probably the best you’ve currently got http://www.onyxplatform.org/docs/user-guide/0.12.x/#_asynchronous_barrier_snapshotting
and you end up throwing an exception even though none of the messages since the recovery should have caused it.
by the way, it seems I was getting same result with propagated exception if I called
so I wonder what's the better way - call
ack-fn to cause
write-batch or throw exception in
okay, I've pushed it, and will clean up and turn into a single commit a little bit later today
@lucasbradstreet do you think it's necessary to implement task for batch output? I've never used that and can't think of use cases.
Anyone has seen this Exception:
Lost and regained image with the same session-id and different correlation-id? And has a hint with what goes wrong?
Here are the settings for 0.12 https://github.com/onyx-platform/onyx/blob/0.12.x/changes.md
@eelke https://clojurians-log.clojureverse.org/onyx/2017-10-20.html might be useful to quote @lucasbradstreet > Essentially what happened is the aeron messaging subscriber dropped off, but then rejoined again at a later point in the stream, which is unsafe because you could lose messages. I believed it would never happen in practice, so added that check as kind of an assert. In reality, I think there are conditions like GCs where the subscriber could be booted but be rejoined with the same session-id.
Any tip to deal with concurrency on logs? - im on onyx 0.10 - trying to read a peer log.
There is 2 virtual peers (not sure about vpeers is the right term)
both are on function A, one with
data1, other with
Function A is defined as
(comp C B)
start B with data 1
start B with data 2
start C with data 2
start C with data 1
I wanna get logs just for
data1, or just for
Would recommend a log aggregator. Onyx doesn't do vpeer level logging isolation.
I'm using clojure.tools.logging... I will search about "log aggregator" and checkout onyx.log docs
If you want to be able to correlate logs to particular peer, you could inject
:onyx.core/log-prefix from the event map into one of your fn arguments. log-prefix contains all of the info about a peer/slot/task/etc
Hey @lucasbradstreet -- It looks like
onyx.plugin.null/in-calls is missing in
0.12.x. I see it in
0.11.x, though, and we were using it in version
Ah. Those calls don’t really do anything so I took them out. We actually have some functionality coming that makes the null plugin no longer necessary.
Basically we’ve added an
:onyx/type :reduce that allows you to place a
:reduce task as a leaf node without adding a plugin to it.