This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-04-02
Channels
- # arachne (1)
- # bangalore-clj (3)
- # beginners (4)
- # boot (6)
- # cider (3)
- # cljs-dev (14)
- # cljsrn (2)
- # clojure (309)
- # clojure-ireland (1)
- # clojure-russia (7)
- # clojure-spec (10)
- # clojure-uk (5)
- # clojureremote (2)
- # clojurescript (68)
- # cursive (7)
- # emacs (7)
- # luminus (3)
- # lumo (21)
- # off-topic (14)
- # om (1)
- # onyx (53)
- # perun (3)
- # re-frame (4)
- # slack-help (14)
- # unrepl (56)
- # untangled (3)
is there something like a ‘no-op’ output for onyx ? i’ve seen leaf functions, is there an alternative approach ?
yeah but it appears as if onyx always wants to have a workflow end in an :output
type
hmmm, i’m having some trouble getting onyx-kafka to work properly — specifically, getting errors about unable to match a protocol for kafka.Cluster.Endpoint: https://www.refheap.com/708d9229757d311059aca3472
i’m not sure where to look for the cause of the error — misconfiguration of catalog, incompatible library versions, or something else ?
@lmergen How are you running Kafka?
@gardnervickers in a docker container
Are you setting your KAFKA_ADVERTISED_HOSTNAME
?
I'm considering if I could create a wrapper around Pyroclast and make it our analytics engine. This would mean letting our data-scientists submit onyx jobs. I’m trying to understand the possible pros and cons of such an approach. * Would pyroclast be able to handle a large amount of chaotic job requests. e.g lots of changes to their jobs, deletes, etc..? * What is the pricing model? Is it data used, does it depend on number of jobs? * Could we build a wrapper around the simulator to give it our brand.
should i ask this question somewhere else 🙂
@drewverlee 1. Yes, frequently updating jobs is something we expect to happen, and we’re actively working on tooling to help users stop/start jobs at the right offset in their event stream. 2. Mike our Lucas would have to speak towards that, we’ve discussed this internally quite a bit but I don’t think we have anything public to share yet. 3. We’ve had a lot of interest in on-prem Pyroclast, which has some nice advantages but complicates distribution/support a bit.
@gardnervickers so that’s likely to be the root case ? i’m a bit of a kafka noob, and was just using some run-of-the-mill kafka docker container
Possibly, can you share how you’re running Kafka?
And ZK for that matter?
well, zookeeper is even just the standard zookeeper:latest
docker image, and kafka (as i have just discovered) does not have advertised.host.name
or advertised.port
set
Ahh ok, you’ll want to advertise a hostname and port that you can reach from outside of docker.
Depends on where docker is running on your machine
I would just use this TBH https://github.com/wurstmeister/kafka-docker/blob/master/docker-compose.yml
In fact we do use that 😄
@gardnervickers well, that doesn’t appear to be it. anything else i can do to figure this out ?
it really feels as if it is a configuration issue somewhere, with onyx-kafka not being able to figure out what the kafka brokers are
Where is docker running for you? 192.168.99.100
or 127.0.0.1
?
So docker machine then?
https://github.com/onyx-platform/onyx-commander-example/blob/master/docker-compose.yml#L13
So you’re using docker for mac, not docker-machine?
Yea me too hmm
Thats odd, in your container logs for kafka does everything look ok?
I use this sometimes to work with Kafka in dev https://github.com/edenhill/kafkacat
We’ll get it figured out tomorrow, catch ya later!
fwiw, the kafka logs clearly state > Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXT -> EndPoint(127.0.0.1,9092,PLAINTEXT) (kafka.utils.ZkUtils)
could it, hypothetically, be that the latest kafka version is not compatible ? i’m looking at this code https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/cluster/EndPoint.scala and it looks like protocolType has been removed just a few months ago in this commit: https://github.com/apache/kafka/commit/d25671884bbbdf7843ada3e7797573a00ac7cd56#diff-bab4de8ac88a63d2a69ea54e08d286dcL58
(which triggers an exception in this library, which hasn’t been updated in well over a year) : https://github.com/ymilky/franzy-admin/blob/master/src/franzy/admin/codec.clj#L200