Fork me on GitHub
#onyx
<
2017-02-16
>
jasonbell17:02:37

@michaeldrogalis @lucasbradstreet I’m working on a onyx project (using onyx-kafka-8 plugin) that has 3 Kafka partitions but it only reads from the last one. When I look in zookeeper the consumers in Kafka only report one consumer being read from.

jasonbell17:02:53

Has anyone else had the same issue? Or is there something I should be looking at in the config.

michaeldrogalis17:02:11

@jasonbell Are you assigning a partition in the catalog entry?

jasonbell17:02:15

It works once through then hangs on the next run. Deployed on mesos.

jasonbell17:02:29

@michaeldrogalis using min/max peers set to the number of partitions.

jasonbell17:02:51

so they should auto-assign

michaeldrogalis17:02:11

Mmkay.. We do auto-discovery to make sure there are enough peers to cover all the partitions, so it’d throw if there weren’t enough.

michaeldrogalis17:02:56

I’m not sure how easy this would be to test — but can you set up a consumer outside of Onyx to test that connectivity to all partitions is working?

jasonbell17:02:52

I’ll see what I can do

jasonbell17:02:11

What’s the preferred method, assign one task per partition

jasonbell17:02:19

or use min/max to auto assign

michaeldrogalis17:02:24

Nah, what you’re doing is fine.