Reading on the environment notes in the user guide I should be setting :onyx.messaging/bind-addr to an actual interface IP. How do you approach that on the likes of Mesos when you don't know what the ip is going to be? Will localhost suffice?

@jasonbell solved this with a little shell scripting and I believed I used host networking

ok, cheers

@jasonbell that will be the address published to ZooKeeper and discovered by other peers. It should be routeable from every other peer in your Onyx cluster. For Kubernetes, we use the downward API to discover the pod/container IP and set it there. I'm not familiar with Mesos/Marathon but I believe they offer something similar.

thanks @gardnervickers @camechis

I can give you the snippet we used when I get on my computer.

It may not be the best way but it seems to work

@camechis that would be very help, even just to give me some pointers.

thank you

ok, so basically in the run_peer.sh i set the

BIND_ADDR=$HOST

and

BIND_PORT=$PORT0

cool, thanks

no prob!

Okay two more quick questions. Firstly, steps to control the docker container getting totally exhausted

17-02-27 14:51:43 8c8779428f59 INFO [onyx.messaging.aeron.status-publisher:33] - Closing status pub. {:completed? false, :src-peer-id #uuid "3027650e-7c58-a6e3-d10c-2ad8c6e5ae57", :site {:address "localhost", :port 40200, :aeron/peer-task-id nil}, :blocked? nil, :pos 0, :type :status-publisher, :stream-id 0, :dst-channel "aeron:udp?endpoint=localhost:40200", :dst-peer-id #uuid "adefe2ab-9ce1-3939-fca9-97aad3ba15e9", :dst-session-id 1453547509, :short-id 0, :status-session-id nil}
Warning: space is running low in /dev/shm (shm) threshold=167,772,160 usable=10,391,552

@jasonbell 0.10 forwards all metrics into JMX.

ta

17-02-27 16:02:46 f05509f2c393 INFO [onyx.http-query:282] - Starting http query server on 0.0.0.0:18083
17-02-27 16:02:46 f05509f2c393 INFO [onyx.monitoring.metrics-monitoring:82] - Started Metrics Reporting to JMX.

isn’t 1803 the http endpoint in this case ?

yes but I don't see any mention of an exposed JMX port in the Docker setup, do I setup one myself?

Lots of questions mainly because I don't see any obvious instruction to.

ah gotcha, I am not sure. My guess is there will need to be a port exposed

but not sure which one

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false

https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html

Unless those Onyx folk are already baking something in that I don't see.

There'll be a blog post in it, too much useful stuff not to talk about.

@jasonbell Did a connection to 9010 work?

Not tried yet, going to put that in the run_peer.sh and see where we get.

Yeah, you need to open the standard JMX ports for Flight Record/JConsole or whatever else you’re using to talk to it.

That warning above is troubling, if you’re running in Mesos you’ll want to allocate some space at /dev/shm for the Aeron media driver.

I think Mesos/Marathon has as —shm-size option?

yes

@jasonbell There are some caveats with running JMX monitoring inside a container too

It does, I'm passing in 1024mb already

Memory_ObjectPendingFinalizationCount 0
Memory_HeapMemoryUsage_committed 966787072
Memory_HeapMemoryUsage_init 264241152
Memory_HeapMemoryUsage_max 966787072
Memory_HeapMemoryUsage_used 820930136
Memory_NonHeapMemoryUsage_committed 152645632
Memory_NonHeapMemoryUsage_init 2555904
Memory_NonHeapMemoryUsage_max -1
Memory_NonHeapMemoryUsage_used 151136728

I can ping the metrics from within the container though, so I can see things

You’ll need to set -Djava.rmi.server.hostname= to something that JConsole/FlightRecorder can resolve to the container.

These are my settings for remote monitoring inside a Kubernetes cluster

-Dcom.sun.management.jmxremote
          -Dcom.sun.management.jmxremote.authenticate=false
          -Dcom.sun.management.jmxremote.ssl=false
          -Dcom.sun.management.jmxremote.local.only=false
          -Dcom.sun.management.jmxremote.port=1099
          -Dcom.sun.management.jmxremote.rmi.port=1099
          -Djava.rmi.server.hostname=127.0.0.1

Kubernetes has a port-forward mechanism that will setup a VPN and allow container ports to be connected to at 127.0.0.1.

ah cool, wonder if Mesos has something similar.

That's enough to keep me amused, appreciate it.

Also looking at FlightRecorder... not looked at that before.

Far too old school for my own good.