Hello, has anyone ever met the exception indicating ${path}/cnc.dat is already in use when calling with-test-env with peer-config: {:onyc.messaging/impl :aeron ... ....}?

@clyce can you please post the exception? It could mean more than one media driver is running

Oh ... the original exception was an Error refreshing environment: java.nio.file.FileSystemException and the detailed message was not shown in English (from my system locale settings), but the meaning was: "another process is using this file"

Ok hmm. Was just wondering whether it was thrown by the media driver or the peers

Seems like some stale state from another test anyway.

let me see...

`

clojure.lang.ExceptionInfo: Error in component :messaging-group in system onyx.system.OnyxPeerGroup calling #'com.stuartsierra.component/start
     component: #<Aeron Peer Group>
      function: #'com.stuartsierra.component/start
        reason: :com.stuartsierra.component/component-function-threw-exception
        system: #<Onyx Peer Group>
    system-key: :messaging-group

So from peer group

I killed all my jvm processes but it seems no use....let me restart the system to try 🙂

It's safe to delete the file if it's giving you the full path

OK let me try

The exception is gone, thank you! 🙂

Wait...it was not....I thought it was gone just cuz I miscommented the with-test-env ..

let me see

Downgrading [org.onyxplatform/onyx "0.9.15"] to [org.onyxplatform/onyx "0.9.10"] eliminates the error

Sounds like a comparability problem between Aeron versions then. If you delete the cnc file all should be well

OK thank you....

deleting didn't work on 0.9.15

it worth mentioning that I was using windows currently for some reason. I'd try this again when I was back on Linux

and 0.9.10 works, I mean, no need to delete cnc.dat manually 🙂

Weird. Hmm.

Windows could be part of it, but it’s hard to know

@lucasbradstreet I’ve got a question on the Kafka plugin and deployment. I’ve the option of using min/max-peers to set the total number of partitions or defining a single partition to consume messages from.

With the total number of partitions then I’m essentially running one peer with n number of virtual peers to do the work. Can I deploy this across three tenancies for example all consuming the same partitions t1 = p0, p1, p2, t2 = p0, p1, p2, t3 = p0, p1, p2 so there’s some safety if a peer dies.

What I can’t afford to happen is that a single peer die with all the partitions and Kafka continue to log messages as I run the risk of running out of volume space and killing the machine.

Just wondering what your thoughts were?

It would be nice to have one peer consuming partitions 1,2 and 3 with another peer consuming 4, 5 and 6 and a third peer consuming 7, 8 and 9 for example.

Looking at the plugin there’s a function check-num-peers-equals-partitions which throws an exception if the total number of partitions doesn’t match what is set in Kafka plugin options.

So that says to me that it’s either all or one partition per peer.

@jasonbell Your last statement is correct. We’d take a PR for parameterizing a vpeer to read from a collection of partitions.

michaeldrogalis if you don't assign any partitions would it just distribute them across the actual peers itself or would they all end up on one peer?

(and re-balance if a peer dies and you have fewer peers than partitions)

p1 = t1, p2 = t2, p3 = t3

p2 dies

p1 = t1, t2

p3 = t3

or similar

I think that behaviour would be the one I was after unless I was trying to be clever with data locality

@otfrom It self-distributes.

michaeldrogalis OK, that's what I thought and hoped it would do

thx!