I went looking for the best channel on Clojurians to find Kafka users before realizing that's probably here, hah. Is anyone here using Kafka with or without Onyx that wouldn't mind 3-4 questions about how you use it?

fire away

My answers would be around the same issues as @camechis but I'm happy to answer any as well.

Thanks! Ill PM @camechis since you're in cahoots 😛

np 🙂

I'm getting "tons"(not sure if tons, everything in zookeeper is verbose) of PrepRequestProcessor

- Got user-level KeeperException when processing sessionid:0x25f77fe7d5d0005 
  type:create cxid:0x386d zxid:0x100006868 txntype:-1 
  reqpath:n/a Error Path:/onyx/1a6000c9-13f0-4b96-bb89-91e7f1f406bf/checkpoint/9d2c5ffc-a3c1-7c67-d09d-50b2a119d08d/44-259 
  Error:KeeperErrorCode = NoNode for /onyx/1a6000c9-13f0-4b96-bb89-91e7f1f406bf/
  checkpoint/9d2c5ffc-a3c1-7c67-d09d-50b2a119d08d/44-259

Those are actually fine. ZooKeeper is way too chatty

Hmm, actually, I see that it’s using ZK for checkpointing which is not something we recommend in prod as it doesn’t get garbage collected

@michaeldrogalis using Kafka w/out Onyx.

@the2bears Great! Ill PM you to keep the side chatter out of the channel.

but would love to figure out a way to use Onyx instead of Storm... 🙂

We can help with that 😉

It's probably more of a political minefield but yes, I realize you're the best source for help with this.

Well, politics and process

happy to answer kafka qs @michaeldrogalis

Thanks! Actually need to catch a bus somewhere, Ill PM you when Im back. 🙂

having a dumb moment here. I am setting up a couple peers on compute instances using my docker images and for some reason I cannot get the virtual peers to start all I see on the peer log is

Attempting to connect to Zookeeper @ 10.142.0.3:2181
Started peers. Blocking forever.

I think your logging is configured to suppress INFO messages

no i am seeing those

17-11-01 20:33:09 304f8674517b INFO [onyx.http-query:468] - Starting http query server on 0.0.0.0:8095
17-11-01 20:33:09 304f8674517b INFO [onyx.monitoring.metrics-monitoring:114] - Started Metrics Reporting to JMX.

also not seeing any peers in zookeeper either just the new Tenant Id

It’s probably having trouble connecting to Aeron

lol, my friend

I should probably make it emit a message periodically to say that Aeron is not up

ok will double check, probably just some configs since I am constructing this by hand

We do it like that so that you don’t have peers that say they’re up but don’t have an aeron connection

makes since, figured it was something like that

"It’s probably having trouble connecting to Aeron" I laughed out loud at this one, sorry

lol

it knows I am beat down and its trying to finish me off

Hahaha

Is there a low-friction way to get the latest state for a window on a running system that has a server repl installed? Using zkCli to pull out the info manually from the last checkpoint gives the right data based on some ascii-strings matching, but it also has a lot of binary characters and doesn't seem straightforward to get edn. Is there an api call or something to get the state, or should I just update my job code to store the state into an atom that could be accessed via server repl?

@eriktjacobsen Yep there is! Via peer-http-query

😞 we're still on 0.10. Thanks, another reason to upgrade

Ah 😕 Sure thing.