Fork me on GitHub
#onyx
<
2017-02-20
>
acron10:02:21

My mind has gone blank; what happens if an exception is thrown inside a task? Does the exception object get passed on?

acron15:02:14

nm, got to the bottom of it all

acron15:02:26

Enjoying flow-conditions with exception handling

acron15:02:50

@michaeldrogalis It wasn't clear to me whether I needed additional flow-conditions to prevent downstream tasks receiving exceptions so I've added 'if this is NOT an exception' predicates -- are they redundant?

acron15:02:45

There was also a single line in the user guide which I stared at for about five minutes, in http://www.onyxplatform.org/docs/user-guide/0.10.0-beta3/#_exceptions

acron15:02:57

This will only restrict the flow from :input-stream to :error-task when an exception is thrown ...

michaeldrogalis16:02:05

@acron Exceptions can’t be send to downstream tasks, only handled at the site they are thrown.

michaeldrogalis16:02:36

To answer your immediate question - that’s redundant, yes.

acron16:02:45

My parsing of that sentence was that the flow-conditition will only restrict the flow... when an exception, is thrown. I wanted it to read: This will restrict the flow from :input-stream to :error-task to only when an exception is thrown

acron16:02:10

English is hard.

acron16:02:17

Thanks for clarifying my questions.

acron16:02:45

God, even my amended sentence doesn't make sense

michaeldrogalis17:02:17

@acron Let me know if I can clarify further 🙂

acron17:02:45

@michaeldrogalis Thanks. Regarding my redundant flow-conditions, the one I did need to add was 'unless the segment is an exception, DO NOT forward it to :error-task' - is that standard practice?

acron17:02:31

(Otherwise :error-task was getting all my good segments)

michaeldrogalis17:02:43

unless statements always bend my brain. So basically, you have a set of flow conditions, and only those with :flow/exception-thrown? true will be considered when an exception has been thrown inside the task. All other flow entries will never be looked at.

michaeldrogalis17:02:59

The inverse is also true. If it’s not an exception, entries that dont specify thrown -> true are never considered

michaeldrogalis17:02:30

The other thing you can do is run your job with onyx-local-rt so you can experiment and iterate a little quicker. Flow-logic has always been tricky for me to personally follow, before and after Onyx.

acron17:02:11

Yeah, I definitely would have preferred to get this right using a 'nicer' environment - that was a mistake for me in this particular project.

acron17:02:25

But yeah, I believe I had to specify the :error-task in my workflow, right? So It was something like [[:in-task :foo:] [:foo :out-task] [:foo :error-task]]

michaeldrogalis17:02:35

@acron Yes, that’s correct.

acron17:02:56

So when a good segment comes along, it was routed to both :out-task and :error-task

michaeldrogalis17:02:48

Right, because by default, a segment is routed to all downstream tasks - flow conditions are what limits this behavior.

michaeldrogalis17:02:07

I think I understand your question from earlier. You were trying to say “Do I need a check for ‘good segment’ to route to out, and a check for an exception and pass to error-task?” If that’s what you meant, then yes

acron17:02:11

So I ended up with 2 flow conditions, 1) thrown-exception? true, :predicate is-exception? to :error-task and 2) thrown-exception? false, predicate: [:not is exception?], to: out-task

acron17:02:35

Yes, you're right

acron17:02:50

Glad I'm doing it right

michaeldrogalis17:02:52

Bingo, doing it right. You can drop :thrown-exception false and use a (constantly true) predicate, but thats the idea.

michaeldrogalis17:02:16

Since thrown-exception is false, that predicate will never be passed an exception object.

otfrom17:02:32

michaeldrogalis in onyx, should I try to model things in a monadic like way (exceptions and other things producing an error data structure) or an erlang like way (have things throw exceptions). From reading this it sounds like the monadic style is a bit better

michaeldrogalis17:02:04

@otfrom I didnt have monads in mind when designing the API for flow conditions. The primary requirement is the ability to capture exception objects and serialize them into data structures that can be passed downstream for further action.

michaeldrogalis17:02:11

So… Shrug 🙂

otfrom17:02:21

michaeldrogalis cool, that's really helpful and fits in with how I've started thinking about things like core.async flows too

otfrom17:02:35

as core.async is definitely not erlang like

otfrom17:02:00

and fits in with how i've started thinking about core.spec

michaeldrogalis17:02:53

@otfrom Cool, yep you’ve got it.

hunter20:02:08

@michaeldrogalis or @lucasbradstreet :: I am trying ot upgrade to onyx-0.10.0.0-beta4 from and 0.9.15 based project .... the onyx-datomic plugin appears to be reading the tx-log as i expect ... however it appears that my logging function cannot see the message bodies inside of segment in an event .. in my logging lifecycle using a doseq around (:message (:onyx.core/batch event)) is nil

michaeldrogalis20:02:49

@hunter Can’t confirm what the key structure is inside :onyx.core/batch at the moment, but it likely changed - there was heavy reconstruction in that part of the code base for 0.10

hunter20:02:54

@michaeldrogalis ok, what's strange is if I log (keys (:onyx.core/batch event)) I get a java.lang.AssertionError: Assert failed: (invariants/short-identifiers-correct %)

lucasbradstreet20:02:15

@hunter the segments are no longer wrapped up in an extra record, so you don’t need to access the message key

lucasbradstreet20:02:54

Not sure why you’re getting that short identifiers error. That’s strange.

lucasbradstreet20:02:28

was the job killed?

jasonbell22:02:37

Just catching up on the conversation, thanks for that. Very helpful indeed.

hunter22:02:58

@lucasbradstreet I am using onyx-datomic 0.10.0.0-beta4 ... I am setting :onyx.peer/storage :zookeeper in my peer config ... I am using read-log as the input to my catalog ... when I start up my topology I see log-segments outputting as segments travel through my nodes; however only 1 to 2 transactions ever get processed and then the job just stops working. no relevant log output that I can see. I have stopped the job and resubmitted the topology many times with consistent results

hunter22:02:27

it's almost as if checkpointing warms up and then something internal to the topology crashes

hunter22:02:37

i have onyx.log output if it would be helpful

lucasbradstreet22:02:00

Logs would be helpful

hunter22:02:07

where should i put them?

lucasbradstreet22:02:37

Depends on how big they are but pastebin is fine

lucasbradstreet23:02:27

@hunter hmm. That log seems to be truncated