Fork me on GitHub
Slack Archive

This page is not created by, affiliated with, or supported by Slack Technologies, Inc.

2017-05-05

Channels

#onyx
<
2017-05-05
>
jetmind07:05:00

hey, what could be the reason onyx throws exception like this: 

:cause nil
 :data {:original-exception :java.lang.reflect.InvocationTargetException}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message nil
   :data {:original-exception :java.lang.reflect.InvocationTargetException}
   :at [onyx.compression.nippy$fn__13691$fn__13692 invoke "nippy.clj" 33]}]
It seem to appear randomly after job works for a while and I don’t see any of my namespaces in the stack trace. Could share the whole stack trace if it helps I’m on 0.9

michaeldrogalis15:05:36

@jetmind Yep, let’s the whole trace.

michaeldrogalis16:05:10

@jetmind That one’s a brain bender. Hm. Well according to the stacktrace, what’s happening is that Onyx is trying to read the contents of a node in ZooKeeper and deserialize it with Nippy. From the looks of it.. Seems like it’s reading a serialized exception out of a znode.

michaeldrogalis16:05:28

I can think of a somewhat brute force way to figure this one out if you’re feeling patient.

michaeldrogalis16:05:48

How many Onyx tenancies are you running in ZooKeeper?

jetmind16:05:22

I think just one, but am not sure. We're using our internal release version as tenancy-id and running peers only for the most recent one

michaeldrogalis16:05:06

@jetmind If you’re handy with a repl, you can run onyx.compression.nippy/zookeeper-decompress on the contents of each znode /onyx/<tenancy>/<job id>/exception/*

michaeldrogalis16:05:24

Its pretty manual, but since the stack trace is so bare, that’s all I can think of.

michaeldrogalis16:05:04

It’s looking like Nippy can’t decompress whatever was put into that node, but I’m having trouble thinking of what value that might be.

lucasbradstreet16:05:40

My guess is that the exception was put there by your peers, and the dashboard can't deserialise it because it doesn't have the same requires/imports

jetmind16:05:17

I'm on the go right now, but I'll definitely do that when I get to the repl

lucasbradstreet16:05:34

This exception was shown by the dashboard, right?

michaeldrogalis16:05:24

Sounds about right. Are you using any custom reader literals?

jetmind16:05:13

I believe so

michaeldrogalis16:05:42

I think @lucasbradstreet’s hypothesis is a winner.

jetmind16:05:48

Thanks, will check it out

michaeldrogalis20:05:00

We’ll be finally be able to close out issue #1: https://github.com/onyx-platform/onyx/issues/1

georgek20:05:29

Thanks, the native side package is coming along as well. I should have basic tests for that monday along with more complete documentation! https://github.com/MissInterpret/onyx-native There’s enough info to get a flavor for it at least!

michaeldrogalis20:05:30

Definitely! I was just saying that ya’ll have been hammering on this for weeks. I don’t think most people have a good idea of the effort level to tackle these issues. You’re doing a great job!

georgek20:05:46

Tots, It’s exciting that we are soooo close to an end-to-end test but yep, the old saying tends to be true. Fast, Cheap, Well-built? Pick 2. lol

georgek20:05:17

thanks again for the encouragement and input!

georgek20:05:44

You are most welcome and Go Team! hee hee