Fork me on GitHub
#onyx
<
2016-11-20
>
michaeldrogalis03:11:24

The added benefit of writing out results as you go is that reporting intermediate progress to outside systems becomes a breeze. It’s not something bolted on.

akiel11:11:17

@michaeldrogalis Thanks I‘ll try it. With :trigger/on should I use? Is the :segment trigger also called on the end of the window?

michaeldrogalis17:11:30

@akiel All the default triggers are called at the stopping of a task, yes.

michaeldrogalis17:11:49

You could trigger on a timer every N seconds perhaps, or every 10,000 segments maybe.

akiel19:11:18

@michaeldrogalis: I ran into the shared memory size problem under Docker and I see no way to increase the size in Kubernetes. For what is shared memory used if I'm inside a Docker container with one process only? Can we disable something in case it's not needed?

gardnervickers19:11:42

There’s your peer JVM and the out-of-process media driver

gardnervickers19:11:22

You’ll need to set your heap size on the both JVMs, since the JVM won’t correctly detect available memory in a docker container.

gardnervickers19:11:38

Kubernetes lets you supply the memory allocation amount as an env var through the downward api. Then you can set your heap sizes to a ratio off that

gardnervickers19:11:19

You can also do something like this to correctly detect memory limits using the containers cgroup

CGROUPS_MEM=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)
MEMINFO_MEM=$(($(awk '/MemTotal/ {print $2}' /proc/meminfo)*1024))
MEM=$(($MEMINFO_MEM>$CGROUPS_MEM?$CGROUPS_MEM:$MEMINFO_MEM))
JVM_MEDIA_DRIVER_HEAP_RATIO=${JVM_MEDIA_DRIVER_HEAP_RATIO:-0.2}
XMX=$(awk '{printf("%d",$1*$2/1024^2)}' <<< " ${MEM} ${JVM_MEDIA_DRIVER_HEAP_RATIO} “)

gardnervickers19:11:50

Then launch your JVM with "-Xmx${XMX}m"

akiel20:11:31

@gardnervickers Thanks. I currently run the embedded media driver. My JVM is set to 4 GB Heap with matching Kubernetes mem limits. Do I really need the separate media driver? What is the advantage?

gardnervickers20:11:30

Better messaging performance

gardnervickers20:11:45

You don’t absolutely need it though.

akiel20:11:31

Can you please explain, why the messaging performance is better if the media driver runs in a separate JVM?

akiel20:11:22

Thanks I’ll read this.

gardnervickers20:11:10

Also the JVM will allocate for other things besides heap, like a stack for each thread, class metadata, etc.. -Xmx shouldn’t be 100% of available memory.

akiel20:11:02

I was able to set the aeron.dir option to a docker volume as first work around.

akiel20:11:20

Aeron puts 197 MB of data into the aeron.dir at startup.

akiel21:11:56

Ahh nice - thanks. I used a normal emptyDir. Lets try...

gardnervickers21:11:08

Be aware that it’s part of your container memory limit

gardnervickers21:11:20

I’ve hit that before

akiel21:11:49

Yes I read this.

akiel21:11:40

@gardnervickers Thanks a lot. It works now. 🙂

yonatanel21:11:53

@gardnervickers Is this something that's generally good to apply? I'm keeping a file of gotchas and things to remember and wonder if I should keep this for later (the separate JVM for media driver).

gardnervickers21:11:27

Yes, if you’re running Onyx in production you should be running an out-of-process media driver