Fork me on GitHub
#onyx
<
2017-03-07
>
clyce03:03:47

Hello, has anyone ever met the exception indicating ${path}/cnc.dat is already in use when calling with-test-env with peer-config: {:onyc.messaging/impl :aeron ... ....}?

lucasbradstreet03:03:59

@clyce can you please post the exception? It could mean more than one media driver is running

clyce03:03:21

Oh ... the original exception was an Error refreshing environment: java.nio.file.FileSystemException and the detailed message was not shown in English (from my system locale settings), but the meaning was: "another process is using this file"

lucasbradstreet03:03:10

Ok hmm. Was just wondering whether it was thrown by the media driver or the peers

lucasbradstreet03:03:27

Seems like some stale state from another test anyway.

clyce03:03:47

let me see...

clyce03:03:11

clojure.lang.ExceptionInfo: Error in component :messaging-group in system onyx.system.OnyxPeerGroup calling #'com.stuartsierra.component/start
     component: #<Aeron Peer Group>
      function: #'com.stuartsierra.component/start
        reason: :com.stuartsierra.component/component-function-threw-exception
        system: #<Onyx Peer Group>
    system-key: :messaging-group

clyce03:03:17

So from peer group

clyce03:03:43

I killed all my jvm processes but it seems no use....let me restart the system to try 🙂

lucasbradstreet03:03:42

It's safe to delete the file if it's giving you the full path

clyce03:03:08

OK let me try

clyce03:03:55

The exception is gone, thank you! 🙂

clyce03:03:22

Wait...it was not....I thought it was gone just cuz I miscommented the with-test-env ..

clyce03:03:24

let me see

clyce04:03:22

Downgrading [org.onyxplatform/onyx "0.9.15"] to [org.onyxplatform/onyx "0.9.10"] eliminates the error

lucasbradstreet04:03:30

Sounds like a comparability problem between Aeron versions then. If you delete the cnc file all should be well

clyce04:03:07

OK thank you....

clyce04:03:18

deleting didn't work on 0.9.15

clyce04:03:12

it worth mentioning that I was using windows currently for some reason. I'd try this again when I was back on Linux

clyce04:03:59

and 0.9.10 works, I mean, no need to delete cnc.dat manually 🙂

lucasbradstreet04:03:53

Windows could be part of it, but it’s hard to know

jasonbell07:03:21

@lucasbradstreet I’ve got a question on the Kafka plugin and deployment. I’ve the option of using min/max-peers to set the total number of partitions or defining a single partition to consume messages from.

jasonbell07:03:50

With the total number of partitions then I’m essentially running one peer with n number of virtual peers to do the work. Can I deploy this across three tenancies for example all consuming the same partitions t1 = p0, p1, p2, t2 = p0, p1, p2, t3 = p0, p1, p2 so there’s some safety if a peer dies.

jasonbell07:03:37

What I can’t afford to happen is that a single peer die with all the partitions and Kafka continue to log messages as I run the risk of running out of volume space and killing the machine.

jasonbell07:03:45

Just wondering what your thoughts were?

jasonbell07:03:35

It would be nice to have one peer consuming partitions 1,2 and 3 with another peer consuming 4, 5 and 6 and a third peer consuming 7, 8 and 9 for example.

jasonbell07:03:10

Looking at the plugin there’s a function check-num-peers-equals-partitions which throws an exception if the total number of partitions doesn’t match what is set in Kafka plugin options.

jasonbell07:03:33

So that says to me that it’s either all or one partition per peer.

michaeldrogalis15:03:36

@jasonbell Your last statement is correct. We’d take a PR for parameterizing a vpeer to read from a collection of partitions.

otfrom15:03:19

michaeldrogalis if you don't assign any partitions would it just distribute them across the actual peers itself or would they all end up on one peer?

otfrom15:03:48

(and re-balance if a peer dies and you have fewer peers than partitions)

otfrom15:03:08

p1 = t1, p2 = t2, p3 = t3

otfrom15:03:26

p1 = t1, t2

otfrom15:03:46

or similar

otfrom15:03:06

I think that behaviour would be the one I was after unless I was trying to be clever with data locality

michaeldrogalis15:03:25

@otfrom It self-distributes.

otfrom15:03:57

michaeldrogalis OK, that's what I thought and hoped it would do