@sirsean: Any luck?

Ah, alas no. But I only tried for a little bit after you left, and now I’m about to get some dinner. I’ll be back at it in the morning, with a different “get stuff into Kafka” approach than kafkacat. (It would probably be useful for me to figure out the kafkacat problem too, but honestly that’s lower priority for me right now than the app I’m trying to build.)

Yup gotcha! The way I see it is if you can get it working with onyx-seq/`onyx-core-async` then thats 90% of the struggle. The input/output plugins are just configuration at that point

thanks, @mention me if you want to work through this more!

Cool, thanks a lot.

@lvh I'm keen to turn the meetup part of the template into a template option, but we didn't have the time yet. We wanted to get something out and leaned towards including it

For now we think it's better to include more and strip it down

lucasbradstreet: Yep, absolutely. That way at least I can backport stuff that’s interesting

reaperhulk has also just appeared out of the Rackspace Desdemona clown car

now if only I knew what a lisp was.

Hahaha

@greywolve: I’ve decided that more the merrier for onyx-metrics - we’d love a PR

@lucasbradstreet: will try get to that this Friday 😄

So I’m wondering if anyone can help me debug this problem? Rather than using kafkacat, I made a different Docker image that produces messages to Kafka. I’ve verified this works by making it send to Kafka, and then setting up a consumer that runs on my host and connects to the Kafka topic running inside Docker, and I can see the messages. (Likewise, I can send messages from outside and see them come out. So Kafka seems to be working.)

But when I submit a job, I see an error that looks like this:

peer_1      | 16-Feb-02 21:40:50 3c89bb885d2b INFO [onyx.peer.task-lifecycle] - [566c0298-7791-408a-894b-ecbc591c4146] Stopping Task LifeCycle for :read-lines
peer_1      | 16-Feb-02 21:40:50 3c89bb885d2b WARN [onyx.peer.task-lifecycle] - Caught exception inside task lifecycle. Rebooting the task.
peer_1      |                               java.lang.Thread.run              Thread.java:  745
peer_1      | java.util.concurrent.ThreadPoolExecutor$Worker.run  ThreadPoolExecutor.java:  617
peer_1      |  java.util.concurrent.ThreadPoolExecutor.runWorker  ThreadPoolExecutor.java: 1142
peer_1      |                java.util.concurrent.FutureTask.run          FutureTask.java:  266
peer_1      |                                                ...
peer_1      |                clojure.core/binding-conveyor-fn/fn                 core.clj: 1938
peer_1      |          onyx.plugin.kafka/start-kafka-consumer/fn                kafka.clj:  164
peer_1      |                      onyx.plugin.kafka/reader-loop                kafka.clj:   80
peer_1      |         onyx.plugin.kafka/reader-loop/invokeStatic                kafka.clj:   97
peer_1      |                         onyx.peer.operation/kw->fn            operation.clj:   27
peer_1      |            onyx.peer.operation/kw->fn/invokeStatic            operation.clj:   34
peer_1      |                               clojure.core/ex-info                 core.clj: 4617
peer_1      |                  clojure.core/ex-info/invokeStatic                 core.clj: 4617
peer_1      | clojure.lang.ExceptionInfo: Could not resolve symbol on the classpath, did you require the file that contains the symbol ?
peer_1      |     data: {:kw nil}

And I hardly know where to start digging.

(I’d been wondering how I can get read-lines to log more, but this gives me the impression it’s failing to start read-lines because it can’t set up a Kafka consumer correctly. I am so open to being corrected on that.)

Looks like you haven't supplied the deserializer-fn to the Kafka task

Ah! Since my message is just plaintext, I removed the JSON deserializer … and didn’t replace it. Trying...

Ah. We should have a better error message there too

It looks like that was it. (I’m getting messages from Kafka and they’re flowing through my workflow.) Yes! (Now trying to debug the data format, but I think that’s mostly on me for now.)