This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-02-20
Channels
- # aleph (19)
- # aws-lambda (8)
- # bangalore-clj (1)
- # beginners (13)
- # boot (179)
- # cljs-dev (12)
- # cljsjs (2)
- # cljsrn (6)
- # clojure (174)
- # clojure-italy (14)
- # clojure-nl (2)
- # clojure-russia (172)
- # clojure-spec (29)
- # clojure-uk (22)
- # clojurebridge (10)
- # clojureremote (1)
- # clojurescript (79)
- # cursive (46)
- # data-science (1)
- # datascript (8)
- # datomic (18)
- # defnpodcast (2)
- # emacs (9)
- # events (6)
- # hoplon (11)
- # klipse (13)
- # lein-figwheel (1)
- # leiningen (1)
- # luminus (1)
- # lumo (88)
- # numerical-computing (1)
- # off-topic (24)
- # om (33)
- # onyx (58)
- # protorepl (8)
- # re-frame (10)
- # reagent (26)
- # ring (8)
- # ring-swagger (7)
- # rum (22)
- # spacemacs (25)
- # specter (5)
- # uncomplicate (37)
- # untangled (75)
- # vim (17)
- # yada (3)
My mind has gone blank; what happens if an exception is thrown inside a task? Does the exception object get passed on?
@michaeldrogalis It wasn't clear to me whether I needed additional flow-conditions to prevent downstream tasks receiving exceptions so I've added 'if this is NOT an exception' predicates -- are they redundant?
There was also a single line in the user guide which I stared at for about five minutes, in http://www.onyxplatform.org/docs/user-guide/0.10.0-beta3/#_exceptions
This will only restrict the flow from :input-stream to :error-task when an exception is thrown
...
@acron Exceptions can’t be send to downstream tasks, only handled at the site they are thrown.
To answer your immediate question - that’s redundant, yes.
My parsing of that sentence was that the flow-conditition will only restrict the flow... when an exception, is thrown. I wanted it to read: This will restrict the flow from :input-stream to :error-task to only when an exception is thrown
Heh, np.
@acron Let me know if I can clarify further 🙂
@michaeldrogalis Thanks. Regarding my redundant flow-conditions, the one I did need to add was 'unless the segment is an exception, DO NOT forward it to :error-task' - is that standard practice?
unless
statements always bend my brain. So basically, you have a set of flow conditions, and only those with :flow/exception-thrown?
true will be considered when an exception has been thrown inside the task. All other flow entries will never be looked at.
The inverse is also true. If it’s not an exception, entries that dont specify thrown -> true are never considered
It might help to read the code on this one: https://github.com/onyx-platform/onyx/blob/0.10.x/src/onyx/flow_conditions/fc_routing.cljc#L40
The other thing you can do is run your job with onyx-local-rt so you can experiment and iterate a little quicker. Flow-logic has always been tricky for me to personally follow, before and after Onyx.
Yeah, I definitely would have preferred to get this right using a 'nicer' environment - that was a mistake for me in this particular project.
But yeah, I believe I had to specify the :error-task
in my workflow, right? So It was something like [[:in-task :foo:] [:foo :out-task] [:foo :error-task]]
@acron Yes, that’s correct.
Right, because by default, a segment is routed to all downstream tasks - flow conditions are what limits this behavior.
I think I understand your question from earlier. You were trying to say “Do I need a check for ‘good segment’ to route to out
, and a check for an exception and pass to error-task
?” If that’s what you meant, then yes
So I ended up with 2 flow conditions, 1) thrown-exception? true, :predicate is-exception? to :error-task
and 2) thrown-exception? false, predicate: [:not is exception?], to: out-task
Bingo, doing it right. You can drop :thrown-exception false
and use a (constantly true)
predicate, but thats the idea.
Since thrown-exception is false, that predicate will never be passed an exception object.
michaeldrogalis in onyx, should I try to model things in a monadic like way (exceptions and other things producing an error data structure) or an erlang like way (have things throw exceptions). From reading this it sounds like the monadic style is a bit better
@otfrom I didnt have monads in mind when designing the API for flow conditions. The primary requirement is the ability to capture exception objects and serialize them into data structures that can be passed downstream for further action.
So… Shrug 🙂
michaeldrogalis cool, that's really helpful and fits in with how I've started thinking about things like core.async flows too
@otfrom Cool, yep you’ve got it.
@michaeldrogalis or @lucasbradstreet :: I am trying ot upgrade to onyx-0.10.0.0-beta4 from and 0.9.15 based project .... the onyx-datomic plugin appears to be reading the tx-log as i expect ... however it appears that my logging function cannot see the message bodies inside of segment in an event .. in my logging lifecycle using a doseq around (:message (:onyx.core/batch event)) is nil
@hunter Can’t confirm what the key structure is inside :onyx.core/batch
at the moment, but it likely changed - there was heavy reconstruction in that part of the code base for 0.10
@michaeldrogalis ok, what's strange is if I log (keys (:onyx.core/batch event)) I get a java.lang.AssertionError: Assert failed: (invariants/short-identifiers-correct %)
@hunter the segments are no longer wrapped up in an extra record, so you don’t need to access the message key
Not sure why you’re getting that short identifiers error. That’s strange.
was the job killed?
@lucasbradstreet I am using onyx-datomic 0.10.0.0-beta4 ... I am setting :onyx.peer/storage :zookeeper
in my peer config ... I am using read-log as the input to my catalog ... when I start up my topology I see log-segments outputting as segments travel through my nodes; however only 1 to 2 transactions ever get processed and then the job just stops working. no relevant log output that I can see. I have stopped the job and resubmitted the topology many times with consistent results
it's almost as if checkpointing warms up and then something internal to the topology crashes
Logs would be helpful
Depends on how big they are but pastebin is fine
@hunter hmm. That log seems to be truncated