This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-11-09
Channels
- # bangalore-clj (1)
- # beginners (158)
- # boot (8)
- # cider (9)
- # cljsjs (9)
- # clojure (169)
- # clojure-austin (1)
- # clojure-denmark (1)
- # clojure-dusseldorf (5)
- # clojure-italy (9)
- # clojure-losangeles (2)
- # clojure-russia (31)
- # clojure-spec (53)
- # clojure-turkiye (1)
- # clojure-uk (56)
- # clojurescript (145)
- # cursive (72)
- # datascript (4)
- # datomic (3)
- # duct (121)
- # events (9)
- # figwheel (1)
- # fulcro (46)
- # graphql (4)
- # hoplon (16)
- # jobs (1)
- # jobs-discuss (4)
- # leiningen (16)
- # lumo (5)
- # off-topic (38)
- # om (1)
- # om-next (5)
- # onyx (104)
- # parinfer (5)
- # re-frame (106)
- # reagent (1)
- # ring-swagger (3)
- # rum (1)
- # shadow-cljs (235)
- # slack-help (4)
- # unrepl (25)
- # yada (9)
Hmmm, ok. Maybe I've been thinking about it wrong. Should I, in general, always assume the possibility of duplicated state getting emitted? Or is this a special case?
Well, as a separate point, we can’t guarantee exactly once side effects since that’s impossible, so in that respect you should assume the possibility of sync being called more than once on the same state.
Ok. In that case it’s certainly possible for it to emit more than once if a node fails and it has to recover.
I would also assume that it may be called more than once when the job is completed, because you generally need that final seal to flush anything new that didn’t get triggered by the number of elements
You could turn that off but you may not get the final state written
Ok, good to know. I still have a lot to learn about what's going on behind the scenes. Thanks for your help.
No worries
I'm just afraid this stuff is a little bit too deep inside for me to comprehend what exactly is wrong 🙂
Hi, sorry I didn’t get a chance to look at it yet. Lemme fire it up.
It’s stops writing in that test?
I mean calling write-batch
yeah. So the test is that for one of requests I want to go further than retry-params allow, which should generate an exception
OK, did the job get killed?
I guess if the job got killed it would have finished with feedback-exception!
also, I figured out that Thread/sleep
seems to work properly - when a few requests try to back off, they don't mess with each other, which is what we need
hmm, shouldn’t have to. What do you want to happen when the exception happens?
which would make it to a handle exception?
Just checking
yeah, if I call ack-fn, job finishes, but exception is never propagated... I wonder if there is a better place for propagation than throwing it in write-batch
?
oh wow, but I get this in my onyx.log:
clojure.lang.ExceptionInfo: Handling uncaught exception thrown inside task lifecycle :lifecycle/write-batch. Killing the job. -> Exception type: clojure.lang.ExceptionInfo. Exception message: HTTP request failed!
something weird is going on
for sure
I think I know what’s happening.
OK, so we setup all of our async calls in write-batch, and progress to the next state because we returned true
At some point we receive a barrier, and we have already made all the write-batch calls so all we have to do is wait for the sync
so onyx is going to just keep calling sync until it returns true
so we’re stuck in a situation where we never return true, and because we only check for an exception in write-batch it never gets thrown
so what's the right way to deal with that? do not return true in write-batch until all async requests are finished?
You could do that, or you could just check for failures in sync and completed too
do I need to track somehow that I throw an exception for correct batch or something?
Onyx appears to be propogating the exceptions properly, it just never got the chance to throw one.
Not really since it’ll cause it to rewind to the last barrier
so everything since the last barrier will be retried anyway
This is probably the best you’ve currently got http://www.onyxplatform.org/docs/user-guide/0.12.x/#_asynchronous_barrier_snapshotting
Not really since the best you can get is that the peer will be rebooted
so everything will be re-initialized anyway
Actually you should probably reset it
Because maybe you get some old exceptions from the previous recovery
and you end up throwing an exception even though none of the messages since the recovery should have caused it.
by the way, it seems I was getting same result with propagated exception if I called ack-fn
so I wonder what's the better way - call ack-fn
to cause write-batch
or throw exception in synced?
and completed?
?
okay, I've pushed it, and will clean up and turn into a single commit a little bit later today
@lucasbradstreet do you think it's necessary to implement task for batch output? I've never used that and can't think of use cases.
Anyone has seen this Exception: Lost and regained image with the same session-id and different correlation-id
? And has a hint with what goes wrong?
Here are the settings for 0.12 https://github.com/onyx-platform/onyx/blob/0.12.x/changes.md
@eelke https://clojurians-log.clojureverse.org/onyx/2017-10-20.html might be useful to quote @lucasbradstreet > Essentially what happened is the aeron messaging subscriber dropped off, but then rejoined again at a later point in the stream, which is unsafe because you could lose messages. I believed it would never happen in practice, so added that check as kind of an assert. In reality, I think there are conditions like GCs where the subscriber could be booted but be rejoined with the same session-id.
Any tip to deal with concurrency on logs? - im on onyx 0.10 - trying to read a peer log.
@souenzzo concurrency in what way?
There is 2 virtual peers (not sure about vpeers is the right term)
both are on function A, one with data1
, other with data2
.
Function A is defined as (comp C B)
My log:
start B with data 1
start B with data 2
start C with data 2
start C with data 1
...
I wanna get logs just for data1
, or just for data2
Would recommend a log aggregator. Onyx doesn't do vpeer level logging isolation.
So, you mean log messages written to onyx.log?
I'm using clojure.tools.logging... I will search about "log aggregator" and checkout onyx.log docs
If you want to be able to correlate logs to particular peer, you could inject :onyx.core/log-prefix
from the event map into one of your fn arguments. log-prefix contains all of the info about a peer/slot/task/etc
Hey @lucasbradstreet -- It looks like onyx.plugin.null/in-calls
is missing in 0.12.x
. I see it in 0.11.x
, though, and we were using it in version 0.10.x
.
Ah. Those calls don’t really do anything so I took them out. We actually have some functionality coming that makes the null plugin no longer necessary.
Basically we’ve added an :onyx/type :reduce
that allows you to place a :reduce
task as a leaf node without adding a plugin to it.
It’s not documented yet, so we aren’t really showing it anywhere.
Yeah, they weren’t doing anything anyway, which is why I removed them.
Sorry about that.