Fork me on GitHub
#onyx
<
2018-01-10
>
sreekanth12:01:11

i’m trying upgrade to onyx-0.12.x, getting this exception when trying to read from kafka using kafka plugin. scala.collection.immutable.Map$Map1 cannot be cast to scala.collection.Seq Caught exception inside task lifecycle :lifecycle/initializing. Rebooting the task. -> Exception type: java.lang.ClassCastException. Exception message: scala.collection.immutable.Map$Map1 cannot be cast to scala.collection.Seq Any help?

gklijs12:01:29

Coming from someone who has experience with kafka, but none with onyx yet, is it possible the kafka client has a higher version then the broker/server?

lmergen12:01:51

kafka breaking changes is notorious inded

jasonbell14:01:24

@sreekanth What Kafka version are you using and what’s the Kafka plugin version?

jasonbell14:01:56

I’ve not done any updating since 0.10 so I’d need to spend some time having a look. Need to get up to speed on the newer versions.

sreekanth14:01:52

[org.apache.kafka/kafka_2.11 “0.10.1.1”]

jasonbell14:01:40

And the Onyx Kafka plugin version?

sreekanth14:01:18

[org.onyxplatform/onyx-kafka "0.12.2.0"]

sreekanth14:01:42

before was using 0.10.beta-x

jasonbell14:01:42

I’ll try and find some time and have a play later on. Nothing springs to mind apart from agreeing with @gklijs and @lmergen

gklijs14:01:21

I’ll be starting to prepare some hands-on tonight (CET), using latest server, and the kafka endpoint. Hoping it ‘just works’ but if not I’ll let it know here.

michaeldrogalis16:01:48

@sreekanth Onyx-kafka is using [org.apache.kafka/kafka_2.11 "0.11.0.0"]

danielcompton20:01:09

Is there a way with metamorphic to set a match to a prior event? I'm parsing an event stream or re-frame actions, and I know that a match has finished when an event happens, but I actually want the end of the match to be the event one-prior

danielcompton20:01:27

Kind of a look-back and say "Oh, know I know what I know here, the match actually finished back there"

danielcompton20:01:43

I'm guessing it's not possible to do this efficiently

michaeldrogalis20:01:58

@danielcompton Can you simply axe off the last event from the returned match?

danielcompton20:01:32

That was my plan, but I need to then grab the n-1 and add that to the match

danielcompton20:01:08

I think I know what the event preceding the finish would be, so I could optionally match on that, but I don't know if I've covered all of the edge cases

danielcompton20:01:54

That's probably the best option, and having a fixup path if I don't have id=n-1 and id=n as the last two events in a match

michaeldrogalis20:01:35

It's a tricky problem domain. I don't think the algorithm can express what you're asking it to directly because it needs to keep all of the events in its tracking path linked together.

danielcompton20:01:24

Yeah, I think the real answer is to figure out what the preceding events could be. My use for metamorphic is more for slicing an event stream, rather than parsing it

djjolicoeur20:01:58

does anyone know if there is an easy way to see consumer offsets via 0.9.x onyx?

djjolicoeur20:01:08

^ sorry, kafka consumer offsets. we're having a bit of trouble gauging whether we are keeping pace or not

danielcompton20:01:24

Also on metamorphic, what is the cost of getting the history in an event match influenced by? Is it number of events that were matched, or in-flight matches, or number of total events?

danielcompton20:01:52

I need to look back at history to detect the start of a match, and am just trying to figure out how big a deal derefing history is

danielcompton20:01:25

Not that it hugely matters cause it's a dev tool operating on 100's to 1,000's of traces, but good to know what influences that cost

michaeldrogalis20:01:14

@djjolicoeur It's stored in ZooKeeper for <= 0.9

michaeldrogalis20:01:22

@danielcompton Number of actual matched events.

djjolicoeur20:01:32

do you happen to know where in ZK. we've been poking around trying to find it with no luck

michaeldrogalis20:01:38

It discards non-matches immediately

danielcompton20:01:10

Great, thanks!

sreekanth21:01:02

it did work after changing to 0.11.0.0. thanks @michaeldrogalis

djjolicoeur21:01:39

damnnit. I looked right past it. thanks appreciate it.