onyx 0.8.0-SNAPSHOT , is there a matching version for onyx-kafka?

The alpha should work fine with it unless there's anything in particular that you need in the latest couple of commits

Generally the plugins are backwards compatible though we do release them all together so that we can CI the versions together

ok. I am trying out kafka plugin . I have not seen any errors nor results either .

Have you previously used it? What does it show in the log?

yes. I tested kafka with some other code. which works as presumed.

I mean the plugin

nope. today is the first day.

Ok. What's showing up in onyx.log?

at first ,there is kafka partition related error. i resolved this issue.

then there is namespace related error.

all resolved then there is no error.

I cant get values form kafka.

Hmm

no error in log

Reading?

What's the last few linea in the log?

please wait a sec. I am getting the log.

15-Nov-10 13:06:26 whm.c.tunlld-01.internal INFO [onyx.peer.virtual-peer] - Starting Virtual Peer e4654903-ac58-419c-9b8c-f9be23d5096b 15-Nov-10 13:06:26 whm.c.tunlld-01.internal INFO [onyx.log.zookeeper] - Starting ZooKeeper client connection 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.acking-daemon] - Starting Acking Daemon 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.messenger-buffer] - Starting Messenger Buffer 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.aeron] - Starting Aeron 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.peer.virtual-peer] - Starting Virtual Peer 237792b1-e002-4eb2-b20f-4b4aafd16963 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.log.zookeeper] - Starting ZooKeeper client connection 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.acking-daemon] - Starting Acking Daemon 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.messenger-buffer] - Starting Messenger Buffer 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.aeron] - Starting Aeron 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.peer.virtual-peer] - Starting Virtual Peer 22d300e8-123c-448c-b807-659f6e006250

I have two nodes

Have you submitted a job?

yes

Hmm. It just never seems to have started a job

I think so too.

let me check my code again.

@michaeldrogalis @lucasbradstreet Everything is working with onyx-kafka if I have a single node, but once I have two processes running (with the same :onyx/id) then my job seems to go forever, repeating segments over and over and never reaching the :done sentinel

Any ideas on that?

very strange

@spangler: My guess is that the two machines still can't talk properly to one another, so messages are timing out and being replayed.

It is when I am testing with two processes on the same machine

How can I test whether the processes are able to talk to each other?

in other words: How are they trying to talk to each other?

Via the port specified in the peer config

Use netcat or whatever to make sure data can flow over that channel.

Hmm.... so if both processes were using the same port on localhost, would that be an issue?

Yes

I will try that, forgot about the port

:done is not supported for auto assigned kafka partitions. (:kafka/partition must be supplied)^[[m

@yusup Right

Hi @yusup

If I set :kafka/partition to specific partition

It's supported if you set a specific partition

The problem is that if you have multiple partitions but you wrote the done to one partition, we can't know that you've read all your data.

ok. got it

0 then.

One way around this is to make you write a done to each partition, however we will be changing the sentinel mechanism later, so we thought that it's best not to support it

What do you mean by 0?

I have to set it to 0 for now

@lucasbradstreet Ah, how are you changing the sentinel mechanism?

Currently using that for batch processing

I am curious too

This is the proposal: https://github.com/onyx-platform/onyx/blob/0.8.x/doc/design/proposals/batch_primitives.md. Originally it was the focus for 0.8.0, however streaming primitives, i.e. windowing, aggregation, and state management were deemed more important

It may change given our latest streaming work

We’ll provide a good way to transition from the current sentinel method though

Okay, so using onyx-kafka with two nodes on a single machine with different messaging ports: The tasks cycle over and over and never ends, ie: The job never receives the :done sentinel, and certain tasks repeat over and over

I can tell they are communicating, because the tasks are distributing between the two processes

Why would the tasks repeat?

Also, I have a couple other fixes to the onyx-kafka lib that deal with some exceptions that were coming up during usage

@michaeldrogalis Why does a message replay? If the task takes too long to complete?