Fork me on GitHub
#onyx
<
2018-06-29
>
kenny16:06:03

Can org.clojure/test.check be set to :scope "test" and org.clojure/clojurescript & org.clojure/clojure set to :scope "provided" in onyx-spec?

lucasbradstreet16:06:36

Yes, that seems fine

lucasbradstreet17:06:29

@dave.dixon looks like that’s a case where we fail fast and your container / monitored java should start back up

lucasbradstreet17:06:06

It seems to be due to initial failures connecting to Zk, and then having it fail again when it tries to start the connection. We may be able to improve this case.

sparkofreason17:06:27

It's stuck in CrashLoopBackOff in k8s, error keeps repeating. I'll check ZK.

lucasbradstreet17:06:35

Actually this part of the message suggests something else: “org.apache.zookeeper.KeeperException.create KeeperException.java: 103 org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /onyx code: -6 path: "/onyx"

lucasbradstreet17:06:09

UnimplementedException - May need to know what version of onyx and ZK you’re on.

sparkofreason17:06:43

Onyx 0.13.0.0. We're running ZK as a hosted service via CloudKarafka, I'll try to find out what's happening there. It was all working fine just an hour ago.

lucasbradstreet17:06:53

Ah weird then. Either way it definitely seems like something to do with ZK but I’m not much help past that.

sparkofreason17:06:40

Can you think of anything I might have been able to do to cause this?

lucasbradstreet17:06:50

One possibility is that we did bump the ZK client library in 0.13.0 so that we could support SSL. It’s supposed to be backwards compatible, but it’s possible that there is a bad interaction with the hosted ZK version

lucasbradstreet17:06:16

That might be preventing you from seeing the true error

lucasbradstreet17:06:22

You could try excluding org.apache.curator/curator-client and org.apache.curator/curator-framework from the onyx dependency, and provide [org.apache.curator/curator-framework "2.9.1"] in your deps to get back the old library, and try again

lucasbradstreet18:06:31

Oof. Looks like there are other deps that will need to be overridden. I’ll check with the diff again

sparkofreason18:06:53

FWIW - I'm pretty sure the previous working version was onyx 0.13.0.0 as well.

lucasbradstreet18:06:27

Yeah, I’m mostly hoping to resolve the unimplemented exception issue so we can get at the real error

lucasbradstreet18:06:44

You may need to exclude org.apache.zookeeper/zookeeper and use “3.4.10” instead

lucasbradstreet18:06:41

All my testing thus far has found the curator/zk deps we bumped to be fine, but I don’t really have any other ideas for debugging it

lucasbradstreet18:06:05

I guess you could try connecting to zk with the cli and listing /onyx and see if you’re able to poke around there

sparkofreason18:06:02

I rolled back to the container image that was working before, and it still works, so it must be something that we changed recently.

lucasbradstreet18:06:33

Ah ok. Dependencies seem the most likely cause.

sparkofreason19:06:42

BTW, the big change is that we moved to using tools-deps, which seems to change how deps are resolved.

j0ni20:06:32

hello all - I'm using a docker-compose setup based closely on the one generated by the +docker flag to the leiningen onyx template - and the config.edn is currently the one which was generated. I'm seeing a lot of DEBUG level logs from the peer container though, which this

:onyx.log/config #profile {:default nil
                             :docker {:level :info}}
looks like it should preclude

j0ni20:06:27

it's this kind of thing:

DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x1644d383a3d0001 after 1ms
which looks like it's not generated originally by timbre

j0ni20:06:25

any ideas how to turn down logging to INFO?