Fork me on GitHub
#onyx
<
2015-11-10
>
yusup12:11:56

onyx 0.8.0-SNAPSHOT , is there a matching version for onyx-kafka?

lucasbradstreet12:11:54

The alpha should work fine with it unless there's anything in particular that you need in the latest couple of commits

lucasbradstreet12:11:07

Generally the plugins are backwards compatible though we do release them all together so that we can CI the versions together

yusup12:11:25

ok. I am trying out kafka plugin . I have not seen any errors nor results either .

lucasbradstreet12:11:19

Have you previously used it? What does it show in the log?

yusup12:11:06

yes. I tested kafka with some other code. which works as presumed.

lucasbradstreet13:11:11

I mean the plugin

yusup13:11:21

nope. today is the first day.

lucasbradstreet13:11:09

Ok. What's showing up in onyx.log?

yusup13:11:11

at first ,there is kafka partition related error. i resolved this issue.

yusup13:11:42

then there is namespace related error.

yusup13:11:54

all resolved then there is no error.

yusup13:11:02

I cant get values form kafka.

yusup13:11:22

no error in log

lucasbradstreet13:11:03

What's the last few linea in the log?

yusup13:11:26

please wait a sec. I am getting the log.

yusup13:11:53

15-Nov-10 13:06:26 whm.c.tunlld-01.internal INFO [onyx.peer.virtual-peer] - Starting Virtual Peer e4654903-ac58-419c-9b8c-f9be23d5096b 15-Nov-10 13:06:26 whm.c.tunlld-01.internal INFO [onyx.log.zookeeper] - Starting ZooKeeper client connection 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.acking-daemon] - Starting Acking Daemon 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.messenger-buffer] - Starting Messenger Buffer 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.aeron] - Starting Aeron 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.peer.virtual-peer] - Starting Virtual Peer 237792b1-e002-4eb2-b20f-4b4aafd16963 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.log.zookeeper] - Starting ZooKeeper client connection 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.acking-daemon] - Starting Acking Daemon 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.messenger-buffer] - Starting Messenger Buffer 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.messaging.aeron] - Starting Aeron 15-Nov-10 13:06:27 whm.c.tunlld-01.internal INFO [onyx.peer.virtual-peer] - Starting Virtual Peer 22d300e8-123c-448c-b807-659f6e006250

yusup13:11:28

I have two nodes

lucasbradstreet13:11:30

Have you submitted a job?

lucasbradstreet13:11:46

Hmm. It just never seems to have started a job

yusup13:11:45

I think so too.

yusup13:11:54

let me check my code again.

spangler17:11:28

@michaeldrogalis @lucasbradstreet Everything is working with onyx-kafka if I have a single node, but once I have two processes running (with the same :onyx/id) then my job seems to go forever, repeating segments over and over and never reaching the :done sentinel

spangler17:11:31

Any ideas on that?

spangler17:11:33

very strange

michaeldrogalis17:11:55

@spangler: My guess is that the two machines still can't talk properly to one another, so messages are timing out and being replayed.

spangler17:11:26

It is when I am testing with two processes on the same machine

spangler17:11:42

How can I test whether the processes are able to talk to each other?

spangler18:11:17

in other words: How are they trying to talk to each other?

michaeldrogalis18:11:35

Via the port specified in the peer config

michaeldrogalis18:11:44

Use netcat or whatever to make sure data can flow over that channel.

spangler18:11:18

Hmm.... so if both processes were using the same port on localhost, would that be an issue?

spangler18:11:49

I will try that, forgot about the port

yusup19:11:49

:done is not supported for auto assigned kafka partitions. (:kafka/partition must be supplied)^[[m

yusup19:11:47

If I set :kafka/partition to specific partition

lucasbradstreet19:11:02

It's supported if you set a specific partition

lucasbradstreet19:11:36

The problem is that if you have multiple partitions but you wrote the done to one partition, we can't know that you've read all your data.

yusup19:11:53

ok. got it

lucasbradstreet19:11:25

One way around this is to make you write a done to each partition, however we will be changing the sentinel mechanism later, so we thought that it's best not to support it

lucasbradstreet19:11:36

What do you mean by 0?

yusup19:11:52

I have to set it to 0 for now simple_smile

spangler19:11:18

@lucasbradstreet Ah, how are you changing the sentinel mechanism?

spangler19:11:24

Currently using that for batch processing

yusup19:11:56

I am curious too

lucasbradstreet19:11:47

This is the proposal: https://github.com/onyx-platform/onyx/blob/0.8.x/doc/design/proposals/batch_primitives.md. Originally it was the focus for 0.8.0, however streaming primitives, i.e. windowing, aggregation, and state management were deemed more important

lucasbradstreet19:11:46

It may change given our latest streaming work

lucasbradstreet19:11:04

We’ll provide a good way to transition from the current sentinel method though

spangler22:11:16

Okay, so using onyx-kafka with two nodes on a single machine with different messaging ports: The tasks cycle over and over and never ends, ie: The job never receives the :done sentinel, and certain tasks repeat over and over

spangler22:11:43

I can tell they are communicating, because the tasks are distributing between the two processes

spangler22:11:54

Why would the tasks repeat?

spangler22:11:10

Also, I have a couple other fixes to the onyx-kafka lib that deal with some exceptions that were coming up during usage

spangler23:11:48

@michaeldrogalis Why does a message replay? If the task takes too long to complete?