We will be releasing 0.12 very soon, if you’re able to upgrade. CI is building a 0.12 snapshot with the fix now.

I can deploy to clojars a 0.11 snapshot to tide you over if you really need it soon.

@kenny give onyx-kafka 0.11.1.2-20171123.001220-3 a go.

We'll be upgrading to 0.12 soon. That release solves my issue. Thanks!

:thumbsup:

BTW OCD thing: anyway you guys could update to a newer version of clj-fuzzy to get rid of this warning:

WARNING: any? already refers to: #'clojure.core/any? in namespace: clj-fuzzy.helpers, being replaced by: #'clj-fuzzy.helpers/any?

Yeah, I really should. I’ll get that in 0.12 too.

There were so many of those previously with 1.9 on the way that I had gotten a bit too used to tuning them out.

Incidentally when we drop Schema we can drop clj-fuzzy too

Anyway.. The issue that led us down this rabbit hole is still present even after updating to onyx-kafka 0.11.1.1.

I'll try and create an isolated reproduction of the problem.

Try validating your task map against the schema in the onyx.kafka.tasks file

That’s about as minimal as you can get here

Ok

(schema/check kafka-tasks/KafkaOutputTaskMap
              {:onyx/type           :output
               :onyx/name           :kafka-out
               :onyx/plugin         :onyx.plugin.kafka/write-messages
               :onyx/medium         :kafka
               :kafka/offset-reset  :earliest
               :kafka/zookeeper     "127.0.0.1:2181"
               :kafka/topic         "test1"
               :kafka/serializer-fn :taoensso.nippy/freeze})
=> {(not (= (name (:kafka)) (namespace :kafka/offset-reset))) invalid-key}

Try retyping the kafka offset line?

Maybe it’s a weird Unicode thing?

Same result.

Oh wait. You did it against the output task schema

Isn't that what it is?

Oh man

Offset reset is only for input tasks

Wow that error was wonderful

I was looking at the first task the whole time.

So what’s happening is that we ban any kafka namespaced keywords that aren’t in the schema for the task. It would have been hard for us to specifically check whether you had mistakenly supplied an input task option.

Makes sense. But still, quite the wild goose chase 😬

Yeah :/

Not sure how to make that one better. I could improve the error message to say disallowed kafka namespaced keyword detected but I still wouldn’t have noticed that it was on the output since I would have looked at your input task immediately

Oh dear T_T

I think a holiday project for me during the slow days is going to be switching this all out for Spec.

kafka.input/offset-reset may have been a better namespace choice

Well, that was certainly a win for Occum’s razor.

Haha indeed.

Hah

I have a question regarding the balanced task scheduler. Why does it skip task allocation if more peers are available as a given job can consume? In my case, I have two peers (with some tag) but I like to use only one (max-peers). In this case the job doesn’t start at all. Why?

@akiel May be a bug in the scheduler. Can you send us your job definition?

The fact that it doesn't start at all sounds wrong.

I‘ll do that in one hour. Thanks.

Np