Has anyone had this zookeeper issue when starting peers?

[2016-08-03 10:02:04,048] INFO Got user-level KeeperException when processing sessionid:0x1564f9d4b9e0008 type:create cxid:0x17 zxid:0x8d txntype:-1 reqpath:n/a Error Path:/onyx/11/messaging/messaging Error:KeeperErrorCode = NodeExists for /onyx/11/messaging/messaging (org.apache.zookeeper.server.PrepRequestProcessor)

Actually the onyx.log file is giving me some good info.

ZooKeeper's logging is a bit noisy. Some valid operations that onyx makes results in these.

Ah yeah, it might just be a red herring with the real error in onyx.log

Yes I agree. The two things aren’t linked but it’s answered some questions I had from last night.

Thanks for the headsup @lucasbradstreet

No problem.

Did anyone submit talk about Onyx to EuroClojure … if not, I’m planning to submit one 😛

That’d be great. I haven’t heard of anyone planning to yet.

ok, I’ll submit one, fingers crossed 🙂

Am I right in the assumption that with 3 partitions on Kafka I’m starting three peers.

$ java -cp target/peer.jar testapp.core start-peers 3 -c resources/config.edn -p :default
Starting peer-group
Starting env
Starting peers
Attempting to connect to Zookeeper @ 127.0.0.1:2181
Started peers. Blocking forever.

INFO [onyx.log.commands.submit-job] - Job ID b78713e1-0db0-474d-bd2d-f07ada0a90cd has been submitted, but received no virtual peers to start its execution.
                     Tasks each require at least one peer to be started, and may require more if :onyx/n-peers or :onyx/min-peers is set.
                     If you were expecting your job to start now, either start more virtual peers or choose a different job scheduler, or wait for existing jobs to finish.

Yes, but you may not have accounted for the other tasks in the job

But you have to set the min-peers and max-peers to the number of partitions, which I’ve done. I don’t see what other tasks would be in that job. It’s merely :in :process-message :out

Right, so you have a kafka :in task with :onyx/max-peers 3 (one for each of three partitions)

yes

then you need at least one peer for :process-message and 📤

:out

Because each task needs at least one peer to be allocated to it

gotcha.

thanks @lucasbradstreet

any time

task-map: {:kafka/zookeeper "127.0.0.1:2181", :onyx/plugin :onyx.plugin.kafka/read-messages, :onyx/medium :kafka, :kafka/offset-reset :smallest, :kafka/force-reset? true, :onyx/batch-timeout 1000, :onyx/type :input, :onyx/name :in, :kafka/topic “my-message", :kafka/group-id "onyx-consumer", ...}
clojure.lang.ExceptionInfo: Caught exception inside task lifecycle. Rebooting the task. -> Exception type: clojure.lang.ExceptionInfo. Exception message: :onyx/min-peers must equal :onyx/max-peers and the number of partitions, or :onyx/n-peers must equal number of kafka partitions

right, so you need to set :onyx/n-peers to the number of partitions, i.e. 3

Started with five peers with one peer set for the :process and :out workflow. three for the kafka input

{:onyx/name :in
                    :onyx/plugin :onyx.plugin.kafka/read-messages
                    :onyx/type :input
                    :onyx/medium :kafka
                    :kafka/topic “my-message"
                    :kafka/group-id "onyx-consumer"
                    :kafka/fetch-size 307200
                    :kafka/chan-capacity 1000
                    :kafka/zookeeper "127.0.0.1:2181"
                    :kafka/offset-reset :smallest
                    :kafka/force-reset? true
                    :kafka/empty-read-back-off 500
                    :kafka/commit-interval 500
                    :kafka/deserializer-fn :testapp.shared/deserialize-message-json
                    :kafka/wrap-with-metadata? false
                    :onyx/min-peers 3
                    :onyx/max-peers 3
                    :onyx/batch-size 100
                    :onyx/doc "Reads messages from a Kafka topic”

Yup, that looks fine

Think I found it, code issue was overriding the settings.

Ah, nothing that could be our fault?

Not yet 😉

Okay, I still get exceptions for some reason, can’t have :onyx/n-peers in the same block with the min/max-peers but then throws an exception as n-peers is nil. It’ll be a setting somewhere I just need to root around for it. Will get there. Thanks for your help.

Will look later on as I have go out this afternoon.

Yeah, it's exclusively either n-peers or min + max peers

clojure.lang.ExceptionInfo: Caught exception inside task lifecycle. Rebooting the task. -> Exception type: clojure.lang.ExceptionInfo. Exception message: :onyx/min-peers must equal :onyx/max-peers and the number of partitions, or :onyx/n-peers must equal number of kafka partitions
          job-id: #uuid "d4f1582f-784c-47fc-bf2e-411aaf6ddc10"
       max-peers: 3
        metadata: {:job-id #uuid "d4f1582f-784c-47fc-bf2e-411aaf6ddc10", :job-hash "205bd6c1978cd6e8ea32c4ad99e3c0bb3f3ad1c31b237fdcd253bea0bf31c6"}
       min-peers: 3
    n-partitions: 1
         n-peers: nil
         peer-id: #uuid "ff158cb1-b169-47c7-bb56-ddf0ead1963b"
        task-map: {:kafka/zookeeper "127.0.0.1:2181", :onyx/plugin :onyx.plugin.kafka/read-messages, :onyx/medium :kafka, :kafka/offset-reset :smallest, :kafka/force-reset? true, :onyx/type :input, :onyx/name :in, :kafka/topic “my-message", :kafka/group-id "onyx-consumer", :onyx/max-peers 3, ...}
       task-name: :in

" n-partitions: 1"

it’s detecting 1 partition for that topic

Where’s n-partitions being set? Is that a default, should I have that in my :in catalog?

It’s inferring it from the topic’s metadata

ah

that explains that then. now I know what else to check 🙂 thanks

no worries

@lucasbradstreet: have you come across this error before?

org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading array of size 1078646, only 36 bytes available
                            clojure.lang.ExceptionInfo: Caught exception inside task lifecycle. Rebooting the task. -> Exception type: org.apache.kafka.common.protocol.types.SchemaException. Exception message: Error reading field 'responses': Error reading array of size 1078646, only 36 bytes available
       job-id: #uuid "cce4f4ce-fb73-458f-8f1a-5f77ec48d328"
     metadata: {:job-id #uuid "cce4f4ce-fb73-458f-8f1a-5f77ec48d328", :job-hash "96a0bed39c8e804ac25b18d2fcd1fd0df7723ca5f674aa46a70bbdfe4e7380"}
      peer-id: #uuid "695d6ded-b41e-4824-b461-d5e5fdb69d53"
    task-name: :in

more a Kafka issue than an Onyx one I know.

Maybe you’re using kafka 0.8?

8.2.2

Just a complete guess. We have a separate plugin release for 0.8

onyx-kakfa-0.8

ah cool

@vijaykiran I'm planning on sitting off the speaker's circuit for the rest of the year. Definitely feel free to submit Onyx talks to conferences, I've said pretty much everything I want to for now 🙂

Nice! - I don’t want to compete with the core-folks 😛

It would be a privilege to have "too many Onyx talks" submitted to a conference.

Yes indeed

Sounds like a plan 🙂

@michaeldrogalis or @lucasbradstreet can you give a quick idea about the order of the docs in the userguide ?

One sec

We used to have every doc linked to Markdown in 0.6: https://github.com/onyx-platform/onyx/tree/0.6.x#user-guide-060

There are a few more sections added now, but that would be a start. Thoughts, Mr. @lucasbradstreet?

I was just looking that up 😄

@lucasbradstreet: onyx-kafka-0.8 worked a treat thanks. All looking good.

thanks for your help earlier today

Glad you got it working 🙂

https://dataissexy.wordpress.com/2016/07/31/using-onyx-template-to-craft-a-kafka-streaming-application-part-1-clojure-onyx-kafka-streaming-data/

https://dataissexy.wordpress.com/2016/08/03/using-onyx-template-to-craft-a-kafka-streaming-application-part-2-clojure-onyx-kafka-streaming-data/

Finally. 🙂

jasonbell: nice one

@otfrom: thank you

I should take these holiday things more often.

Nice! I love these posts. Re: serialiser, I sneak english spelling into our internal code where I can

@jasonbell: Great work ^^

@lucasbradstreet: I always thought that was your way of culturing me.

thank you kindly

jasonbell & michaeldrogalis I suppose the question is whether and/or how this can go into the official docs if it is useful

Maybe we could fill out this page more and link to tutorials like the above? http://www.onyxplatform.org/learn/

I value non-official material, it gives the community a better sense of vibrance than if everything appears to be published from the same source.

Well I’m happy to put more up as I write it and I’m more than happy for @michaeldrogalis and @lucasbradstreet to use it if they wish.

Ill make some links off from the main site tomorrow, just give me a reminder if I forget. 🙂

Is there a way to execute :flow/predicate with additional arguments specified in the catalog? kinda how :onyx/fn is to :onyx/params?

@aengelberg: Yes, but it’s a bit different from how catalog functions take parameters. See “Predicate Parameters” here http://www.onyxplatform.org/docs/user-guide/latest/flow-conditions.html

@gardnervickers: thanks! so the :flow/predicate changes from a keyword to a vector?

Yup! So there’s the example

(defn child? [event old-segment new-segment all-new max-age]
  (<= (:age new-segment) max-age))

It’s like with catalog task parameterization, you pass in the key where you’re value can be found.

Thanks!

:thumbsup: