Fork me on GitHub
#onyx
<
2017-04-08
>
yonatanel09:04:51

Can kubernetes and mesos+marathon work together in some way? In the docs you recommend both so I wonder if I'm missing something.

lmergen13:04:19

hey, i know this is actually off-topic for this channel, but since there’s a big overlap, i’m asking the question in here anyway if that’s okay

lmergen13:04:00

(i have a question about kafka)

lmergen13:04:37

so, i downloaded the onyx-commander-example and everything seems to be working fine — tests pass, so my setup works

lmergen13:04:52

i saw that the project was using gregor, so i decided to try that out myself… but i’m getting completely weird errors with this code:

(ns kafka-test
  (:require [gregor.core :as g]))


(def kafka-zookeeper "127.0.0.1:2181")
(println "topics = " (pr-str (g/topics {:connection-string kafka-zookeeper})))
(g/create-topic {:connection-string kafka-zookeeper} "foobar-wombat" {})

gardnervickers13:04:36

@yonatanel Kubernetes can run on Mesos, however last time I checked the Kubernetes-Mesos version was pretty far behind upstream Kubernetes. Onyx can run on both platforms.

gardnervickers13:04:06

@lmergen hmm what kind of errors?

lmergen13:04:05

baiscally, the create-topic is breaking over some SINGLETON field not being recognized: https://glot.io/snippets/eoqy6ebx6d

lmergen13:04:46

it’s important to note that the (g/topics) command is properly listing the topics

lmergen13:04:51

so communication seems to work

lmergen13:04:07

but… i’m puzzled by this error, and wasting several hours on it now

gardnervickers13:04:12

Can you provide a gist of the error?

lmergen13:04:23

yeah i’ll post everything on GH, one moment!

lmergen13:04:21

mondrian.cqrs..... is my namespace

gardnervickers13:04:28

Make sure your Kafka client and server versions match, I've seen some pretty ambiguous errors when they don't match.

lmergen13:04:36

aha, right!

lmergen13:04:19

i’ll try flushing my dependencies cache

gardnervickers13:04:47

Looks like the latest Gregor is running 0.10.0.1

lmergen13:04:32

and there you’re using 0.10.1.1 🙂

lmergen13:04:09

perhaps my library include order is not the same, causing some different versions to be chosen… hmz

gardnervickers13:04:14

Sorry mobile right now, the above was a typo heh. Gregor is on [org.apache.kafka/kafka_2.11 "0.10.1.1"]

lmergen13:04:34

perhaps i should just be using franzy, since that seems to be working for onyx-kafka

gardnervickers13:04:30

Are you using the docker compose from the Onyx-commander repo?

lmergen13:04:04

let’s see what that does

gardnervickers13:04:36

Yea hmm, it's very odd that the tests pass in Onyx-commander but not for you locally.

lmergen13:04:30

yeah, i’m 80% sure it’s an environment thingie

lmergen13:04:30

dare i say, 90% sure

lmergen13:04:06

what the fudge… franzy is giving the exact same error

gardnervickers13:04:09

I can try and reproduce this error if you can publish a minimal failing case.

lmergen13:04:27

ERROR in (end-to-end-test) (MarkerFactory.java:52)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.NoSuchFieldError: SINGLETON
 at org.slf4j.MarkerFactory.<clinit> (MarkerFactory.java:52)
    org.apache.log4j.Category.<clinit> (Category.java:53)
    org.I0Itec.zkclient.ZkConnection.<clinit> (ZkConnection.java:40)
    franzy.admin.zookeeper.client$make_zk_connection.invokeStatic (client.clj:11)
    franzy.admin.zookeeper.client$make_zk_connection.invoke (client.clj:8)
    franzy.admin.zookeeper.client$make_zk_utils.invokeStatic (client.clj:29)
    franzy.admin.zookeeper.client$make_zk_utils.invoke (client.clj:25)

lmergen13:04:06

is it a logging issue ? can it be ?

lmergen13:04:25

…. mother of god…. it IS a logging issue 😕

lmergen13:04:05

that was literally the only difference, i included timbre in my deps

lmergen13:04:15

i removed that include, and… tada, it worked

lmergen13:04:17

what the hell 😕

lmergen13:04:26

@gardnervickers thanks for your support! 😉

gardnervickers13:04:47

Huh that's incredibly strange. Welp glad to hear you're sorted now

lmergen13:04:36

fwiw, i always include these libraries in my projects when using timbre:

[com.taoensso/timbre "4.8.0"]
   [com.fzakaria/slf4j-timbre "0.3.2"]
   [org.slf4j/log4j-over-slf4j "1.7.21"]
   [org.slf4j/jul-to-slf4j "1.7.21"]
   [org.slf4j/jcl-over-slf4j "1.7.21"]
i believe they do some funky things, like monkey patching log4j functions or something

lmergen13:04:44

removing all those includes fixed this issue

lmergen13:04:18

sorry for the noise 😉

michaeldrogalis13:04:38

@lmergen Only here for a sec, then I gotta run. Only reason I used Gregor here was to make it easier for the direct audience of that example. 99% of the time I use the Kafka bindings directly now to avoid indirect version mismatches. Which, as you’ve found, is super annoying.

lmergen13:04:06

thanks michael, that is a useful tip. i might start doing that as well

gardnervickers13:04:32

@lmergen my guess is it was an issue with SLF4j versioning.

lmergen13:04:56

yes, it’s probably something like that. well, another afternoon ruined, another lesson learned 🙂

lmergen13:04:27

that’s it

lmergen13:04:35

that’s exactly the error i was seeing indeed

gardnervickers13:04:36

This can likely be resolved with the right :exclusions