Fork me on GitHub
#onyx
<
2016-08-03
>
jasonbell09:08:10

Has anyone had this zookeeper issue when starting peers?

[2016-08-03 10:02:04,048] INFO Got user-level KeeperException when processing sessionid:0x1564f9d4b9e0008 type:create cxid:0x17 zxid:0x8d txntype:-1 reqpath:n/a Error Path:/onyx/11/messaging/messaging Error:KeeperErrorCode = NodeExists for /onyx/11/messaging/messaging (org.apache.zookeeper.server.PrepRequestProcessor)
As I’m running locally there’s a good chance it’s my config but just wanted to check. It worked perfectly well before I tried 0.9.9 via onyx-template so wanted to ask all the same.

jasonbell09:08:57

Actually the onyx.log file is giving me some good info.

lucasbradstreet09:08:14

ZooKeeper's logging is a bit noisy. Some valid operations that onyx makes results in these.

lucasbradstreet09:08:28

Ah yeah, it might just be a red herring with the real error in onyx.log

jasonbell09:08:51

Yes I agree. The two things aren’t linked but it’s answered some questions I had from last night.

jasonbell09:08:00

Thanks for the headsup @lucasbradstreet

vijaykiran09:08:06

Did anyone submit talk about Onyx to EuroClojure … if not, I’m planning to submit one 😛

lucasbradstreet10:08:30

That’d be great. I haven’t heard of anyone planning to yet.

vijaykiran10:08:36

ok, I’ll submit one, fingers crossed 🙂

jasonbell10:08:10

Am I right in the assumption that with 3 partitions on Kafka I’m starting three peers.

$ java -cp target/peer.jar testapp.core start-peers 3 -c resources/config.edn -p :default
Starting peer-group
Starting env
Starting peers
Attempting to connect to Zookeeper @ 127.0.0.1:2181
Started peers. Blocking forever.
Aeron starts the three daemons as expected then reports
INFO [onyx.log.commands.submit-job] - Job ID b78713e1-0db0-474d-bd2d-f07ada0a90cd has been submitted, but received no virtual peers to start its execution.
                     Tasks each require at least one peer to be started, and may require more if :onyx/n-peers or :onyx/min-peers is set.
                     If you were expecting your job to start now, either start more virtual peers or choose a different job scheduler, or wait for existing jobs to finish.
Looking on the dashboard there are three virtual peers and only one job. Am I missing something?

lucasbradstreet10:08:47

Yes, but you may not have accounted for the other tasks in the job

jasonbell10:08:09

But you have to set the min-peers and max-peers to the number of partitions, which I’ve done. I don’t see what other tasks would be in that job. It’s merely :in :process-message :out

lucasbradstreet10:08:43

Right, so you have a kafka :in task with :onyx/max-peers 3 (one for each of three partitions)

lucasbradstreet10:08:51

then you need at least one peer for :process-message and 📤

lucasbradstreet10:08:22

Because each task needs at least one peer to be allocated to it

jasonbell10:08:24

task-map: {:kafka/zookeeper "127.0.0.1:2181", :onyx/plugin :onyx.plugin.kafka/read-messages, :onyx/medium :kafka, :kafka/offset-reset :smallest, :kafka/force-reset? true, :onyx/batch-timeout 1000, :onyx/type :input, :onyx/name :in, :kafka/topic “my-message", :kafka/group-id "onyx-consumer", ...}
clojure.lang.ExceptionInfo: Caught exception inside task lifecycle. Rebooting the task. -> Exception type: clojure.lang.ExceptionInfo. Exception message: :onyx/min-peers must equal :onyx/max-peers and the number of partitions, or :onyx/n-peers must equal number of kafka partitions

lucasbradstreet10:08:03

right, so you need to set :onyx/n-peers to the number of partitions, i.e. 3

jasonbell10:08:05

Started with five peers with one peer set for the :process and :out workflow. three for the kafka input

jasonbell10:08:41

{:onyx/name :in
                    :onyx/plugin :onyx.plugin.kafka/read-messages
                    :onyx/type :input
                    :onyx/medium :kafka
                    :kafka/topic “my-message"
                    :kafka/group-id "onyx-consumer"
                    :kafka/fetch-size 307200
                    :kafka/chan-capacity 1000
                    :kafka/zookeeper "127.0.0.1:2181"
                    :kafka/offset-reset :smallest
                    :kafka/force-reset? true
                    :kafka/empty-read-back-off 500
                    :kafka/commit-interval 500
                    :kafka/deserializer-fn :testapp.shared/deserialize-message-json
                    :kafka/wrap-with-metadata? false
                    :onyx/min-peers 3
                    :onyx/max-peers 3
                    :onyx/batch-size 100
                    :onyx/doc "Reads messages from a Kafka topic”
In my :in block here?

lucasbradstreet10:08:55

Yup, that looks fine

jasonbell10:08:30

Think I found it, code issue was overriding the settings.

lucasbradstreet10:08:53

Ah, nothing that could be our fault?

jasonbell10:08:28

Not yet 😉

jasonbell11:08:12

Okay, I still get exceptions for some reason, can’t have :onyx/n-peers in the same block with the min/max-peers but then throws an exception as n-peers is nil. It’ll be a setting somewhere I just need to root around for it. Will get there. Thanks for your help.

jasonbell11:08:26

Will look later on as I have go out this afternoon.

lucasbradstreet11:08:00

Yeah, it's exclusively either n-peers or min + max peers

jasonbell11:08:56

clojure.lang.ExceptionInfo: Caught exception inside task lifecycle. Rebooting the task. -> Exception type: clojure.lang.ExceptionInfo. Exception message: :onyx/min-peers must equal :onyx/max-peers and the number of partitions, or :onyx/n-peers must equal number of kafka partitions
          job-id: #uuid "d4f1582f-784c-47fc-bf2e-411aaf6ddc10"
       max-peers: 3
        metadata: {:job-id #uuid "d4f1582f-784c-47fc-bf2e-411aaf6ddc10", :job-hash "205bd6c1978cd6e8ea32c4ad99e3c0bb3f3ad1c31b237fdcd253bea0bf31c6"}
       min-peers: 3
    n-partitions: 1
         n-peers: nil
         peer-id: #uuid "ff158cb1-b169-47c7-bb56-ddf0ead1963b"
        task-map: {:kafka/zookeeper "127.0.0.1:2181", :onyx/plugin :onyx.plugin.kafka/read-messages, :onyx/medium :kafka, :kafka/offset-reset :smallest, :kafka/force-reset? true, :onyx/type :input, :onyx/name :in, :kafka/topic “my-message", :kafka/group-id "onyx-consumer", :onyx/max-peers 3, ...}
       task-name: :in
That’s what I’m getting at the moment. It’ll be something silly.

lucasbradstreet11:08:17

" n-partitions: 1"

lucasbradstreet11:08:22

it’s detecting 1 partition for that topic

jasonbell11:08:14

Where’s n-partitions being set? Is that a default, should I have that in my :in catalog?

lucasbradstreet11:08:33

It’s inferring it from the topic’s metadata

jasonbell11:08:20

that explains that then. now I know what else to check 🙂 thanks

jasonbell11:08:26

@lucasbradstreet: have you come across this error before?

org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading array of size 1078646, only 36 bytes available
                            clojure.lang.ExceptionInfo: Caught exception inside task lifecycle. Rebooting the task. -> Exception type: org.apache.kafka.common.protocol.types.SchemaException. Exception message: Error reading field 'responses': Error reading array of size 1078646, only 36 bytes available
       job-id: #uuid "cce4f4ce-fb73-458f-8f1a-5f77ec48d328"
     metadata: {:job-id #uuid "cce4f4ce-fb73-458f-8f1a-5f77ec48d328", :job-hash "96a0bed39c8e804ac25b18d2fcd1fd0df7723ca5f674aa46a70bbdfe4e7380"}
      peer-id: #uuid "695d6ded-b41e-4824-b461-d5e5fdb69d53"
    task-name: :in

jasonbell11:08:39

more a Kafka issue than an Onyx one I know.

lucasbradstreet11:08:28

Maybe you’re using kafka 0.8?

lucasbradstreet11:08:59

Just a complete guess. We have a separate plugin release for 0.8

michaeldrogalis15:08:35

@vijaykiran I'm planning on sitting off the speaker's circuit for the rest of the year. Definitely feel free to submit Onyx talks to conferences, I've said pretty much everything I want to for now 🙂

vijaykiran15:08:36

Nice! - I don’t want to compete with the core-folks 😛

michaeldrogalis15:08:04

It would be a privilege to have "too many Onyx talks" submitted to a conference. onyx

vijaykiran15:08:12

Sounds like a plan 🙂

vijaykiran15:08:32

@michaeldrogalis or @lucasbradstreet can you give a quick idea about the order of the docs in the userguide ?

michaeldrogalis15:08:04

There are a few more sections added now, but that would be a start. Thoughts, Mr. @lucasbradstreet?

lucasbradstreet15:08:02

I was just looking that up 😄

jasonbell16:08:37

@lucasbradstreet: onyx-kafka-0.8 worked a treat thanks. All looking good.

jasonbell16:08:49

thanks for your help earlier today

lucasbradstreet16:08:03

Glad you got it working 🙂

jasonbell17:08:32

Finally. 🙂

otfrom17:08:44

jasonbell: nice one

jasonbell17:08:55

I should take these holiday things more often.

lucasbradstreet18:08:10

Nice! I love these posts. Re: serialiser, I sneak english spelling into our internal code where I can

michaeldrogalis18:08:19

@lucasbradstreet: I always thought that was your way of culturing me.

jasonbell18:08:23

thank you kindly

otfrom18:08:42

jasonbell & michaeldrogalis I suppose the question is whether and/or how this can go into the official docs if it is useful

michaeldrogalis18:08:37

Maybe we could fill out this page more and link to tutorials like the above? http://www.onyxplatform.org/learn/

michaeldrogalis18:08:20

I value non-official material, it gives the community a better sense of vibrance than if everything appears to be published from the same source.

jasonbell18:08:16

Well I’m happy to put more up as I write it and I’m more than happy for @michaeldrogalis and @lucasbradstreet to use it if they wish.

michaeldrogalis19:08:22

Ill make some links off from the main site tomorrow, just give me a reminder if I forget. 🙂

aengelberg20:08:59

Is there a way to execute :flow/predicate with additional arguments specified in the catalog? kinda how :onyx/fn is to :onyx/params?

gardnervickers20:08:39

@aengelberg: Yes, but it’s a bit different from how catalog functions take parameters. See “Predicate Parameters” here http://www.onyxplatform.org/docs/user-guide/latest/flow-conditions.html

aengelberg20:08:43

@gardnervickers: thanks! so the :flow/predicate changes from a keyword to a vector?

gardnervickers20:08:24

Yup! So there’s the example

(defn child? [event old-segment new-segment all-new max-age]
  (<= (:age new-segment) max-age))
You’d call that with [:my.ns/child? :my/max-child-age] where also specified in the flow condition map is a key :my/max-child-age with whatever value you want passed in.

gardnervickers20:08:26

It’s like with catalog task parameterization, you pass in the key where you’re value can be found.