Hi @michaeldrogalis lucasbradstreet ... I am trying to switch over from core.async to kafka for reading messages. I've gone through the docs here: https://github.com/onyx-platform/onyx-kafka But onyx does not seem to be reading from the topic I established

Any ideas how to triage this?

I have checked and the messages have been put on the kafka topic referred to by my :read-messages task in the catalog

But nothing is taking them off that topic on the onyx side

@spangler: I'd need to see a bit of code and logs to help.

Hi , how am I gonna confine one task to be executed by each physical peer ,once only ?

is this possible?

@yusup: do you mean that you want one task running on a peer on each physical node? There's no way to do this with our current scheduler I'm afraid

yes. I want to ensure my network operations are evenly distributed.

Ah. We've been discussing improving the scheduler to allow that, but it's actually quite complex to get right, and we want to do it right.

The best thing I have for you is to suggest you use bigger instances so that things are normally more evenly spread out

currently I can do submit a job to a cluster , each physical node only contains only single virtual peer .which basically achieves what I want.

each physical node has just enough resource to do the networking stuff.

but I have to submit another job to other stuff.

You could use a second set of peers running on another onyx/id, and submit the other job to that cluster. It's not ideal though.

two clusters .

Especially since you only have one peer per node, because your job will stop if any of them dies

Yeah. Two "clusters" running on the same machine since onyx/id will logically separate them

let me try this out first .

actually I mean two physical clusters.

It's not ideal and hopefully we can have a better solution for you in the future

Ha ok. That works too