Fork me on GitHub
#clojure-uk
<
2018-09-21
>
mccraigmccraig07:09:19

thanks for the OOM-related pointers yday @conor.p.farrell @reborg

alexlynham07:09:51

Weird JVM based fun with dbs in docker was why only the rails stuff ended up staying in containers long term at the old place iirc

alexlynham07:09:21

Forever debugging memory issues on larger data tasks

jasonbell07:09:04

@mccraigmccraig Is there any value opening up the JMX port and exposing it through Docker so you can run JConsole on it?

jasonbell07:09:51

Also the run script within Onyx container was pretty good, so I use that quite a lot.

jasonbell07:09:22

SANDBOX=${MESOS_SANDBOX:-"."}
CGROUPS_MEM=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)
MEMINFO_MEM=$(($(awk '/MemTotal/ {print $2}' /proc/meminfo)*1024))
MEM=$(($MEMINFO_MEM>$CGROUPS_MEM?$CGROUPS_MEM:$MEMINFO_MEM))
JVM_PEER_HEAP_RATIO=${JVM_PEER_HEAP_RATIO:-0.30}
XMX=$(awk '{printf("%d",$1*$2/1024^2)}' <<< " ${MEM} ${JVM_PEER_HEAP_RATIO} ")
# Use the container memory limit to set max heap size so that the GC
# knows to collect before it's hard-stopped by the container environment,
# causing OOM exception.

## Default BIND_ADDR to the machines hostname
: ${BIND_ADDR:=$(hostname)}
## For use when running a peer with access to the hosts eth0 interface
#: ${BIND_ADDR:=$(ifconfig eth0 | grep "inet addr:" | cut -d : -f 2 | cut -d " " -f 1)}
## For use when running a peer on AWS, where the routeable address is different
### from the address bound to eth0
#: ${BIND_ADDR:=$(curl )}

: ${PEER_JAVA_OPTS:="-Xmx${XMX}m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$SANDBOX -XX:ErrorFile=$SANDBOX/hs_err_pid_%p.log -XX:+UseG1GC -Xloggc:$SANDBOX/gc_%p.log -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3 -XX:GCLogFileSize=2M -XX:+PrintGCDateStamps -server"}

echo "Peer Java Options: $PEER_JAVA_OPTS"


/usr/bin/java $PEER_JAVA_OPTS -jar /opt/myjar.jar

mccraigmccraig08:09:34

@jasonbell i'm already using yourkit against those instances - dyu know if i would get anything extra from jconsole ?

practicalli-john12:09:27

@mccraigmccraig I use VisualVM for monitoring my Clojure apps - mostly resources usage but there is also some nice diagnosis tools if things go badly wrong https://visualvm.github.io/features.html If you need really deep analysis then AppDynamics is really powerful (commercial software with a free trial) https://www.appdynamics.com/java/

jasonbell08:09:56

Probably not, it's just my go to.

maleghast08:09:25

Morning All 🙂

Conor08:09:17

Won an axe throwing tournament last night through pure luck skill

🎲 12
👍 12
danm08:09:34

Nice. Where was that?

danm08:09:56

Got to play a level of Phoenix Point, which was awesome

danm08:09:23

Also CD Projekt RED weren't there officially, but one of their team was just attending the conference as a regular person, so I got to chat to him, which was also ace

3Jane09:09:59

oooOOOoooOOO

3Jane10:09:02

any insider insights? 😄

danm08:09:55

Haha, sadly nothing concrete that wasn't already known. I did say to him that my wife was also well hyped for the game, but slightly worried that she wouldn't be able to complete it because her twitch/FPS skills are poor, so some kind of 'story difficulty' (or, the holy grail, cross platform co-op) would be ace, and he said "I think you're going to be very happy with some of the things we're working on". Could mean they intend co-op if they have time, could just mean there's going to be a mega easy mode of the game

danm08:09:38

He did also semi jokingly ask if I'd consider coming to work in Poland 😉

3Jane09:09:52

that sounds extremely hopeful. Also, interesting, it sounds like Polish studios/branches have been recruiting recently, I’ve gotten some pings recently. I wonder if it’s an opportunistic Brexit related grab, or if there’s a sudden growth there

danm10:09:51

What's your experience of Brexit been? Any feeling like you're less welcome, or thoughts of moving somewhere else in Europe before it all comes crashing down?

3Jane10:09:08

TLDR I knew it was going to happen, I just didn’t expect it to be such a mahoosive clusterfuck

3Jane10:09:37

Being a Polish (or other new-to-EU-central-european) always made you a bit of a “special” cousin, because we had to fill out extra paperwork that other nationalities didn’t

3Jane10:09:57

Registering for some special card, proving that you were gainfully employed for a year after moving in, etc. It felt like being suspect by virtue of where you were born in

danm10:09:54

I know a number of European folks who've said they've got more abuse and felt unwelcome post Brexit, and have headed off to other EU countries

3Jane10:09:44

They later removed it apparently; but anyway once I ticked off the obligatory year, I never experienced anyone being nasty to my face about it; on the contrary, I met people who had Polish relatives in their families, Polish friends, or their friends had spouses from Poland

3Jane10:09:45

The number of UKIP flags on the houses in the neighbourhood grew steadily over the years though, and the relevant politicians started appearing in the news, and on people’s Facebook walls, etc, so you could see which way the wind was blowing, you just had to mentally accept that this was possible. I got somewhat involved during the elections (I’m naturalised, been here for > 10 years) then frustrated, then stopped reading news to cope (it’s amazing how good no news is good for your mental health.)

3Jane10:09:43

What happened to your friends? Were they also CS people? I wonder how much class, or location, makes a difference to how you’re treated. I have a feeling that as an engineer I’m shielded from a lot of ugliness.

danm14:09:40

Sorry, meetings. Some were CS people, most were not (or were actually foreign spouses of CS people). Admittedly 'friends' is pushing it in a lot of cases. 'Online acquaintances'. I suspect it has a lot to do with where you are living, which is probably quite heavily tied to your job etc

danm14:09:21

Weirdly when it comes to voting for Brexit it was mainly the rural areas who didn't have much in the way of 'them foreigners' who seemed to vote for it, but when it came to people saying they got more abuse/problems it was people in the cities etc. That may simply be self-selection that there aren't many other Europeans in the more rural areas though

danm14:09:51

One of my friends (who is English) lives in Eccles, which simultaneously has a sizeable Polish population (and a number of Polish corner shops with amazing not-Jaffa-cakes in different flavours and such) and also the only UKIP 'shop' I am aware of in Manchester (which presumably some of the 'native' population of the area went to. I don't know)

danm14:09:29

My area is mostly white UK. Non-UK folk in the area are fewer and tend to be from former commonwealth countries rather than elsewhere in Europe

danm14:09:44

Apart from round the corner where there is a house turned into 2 flats which has a number of Eastern Europeans (I don't actually know which country) who smoke copious amounts of weed and are bloody awesome (I am biased though because they think our dog is amazing and always say hi and give him a load of fuss when we go past)

Conor09:09:51

Whistlepunks in the Great Northern @carr0t

danm09:09:45

Aah yes, that's it. Laura and I were talking about going there, but haven't got around to it yet

danm09:09:59

Was that a work thing, or out with mates?

Conor09:09:45

Work celebration thing followed by curry

alexlynham11:09:24

axe throwing paid for by coop dollars 😉

alexlynham11:09:39

(or more likely, taxpayer dollars I guess)

Conor11:09:26

I effectively paid for it myself by not efficiently extracting the maximum rate for my skills

trollface 4
practicalli-john12:09:27

@mccraigmccraig I use VisualVM for monitoring my Clojure apps - mostly resources usage but there is also some nice diagnosis tools if things go badly wrong https://visualvm.github.io/features.html If you need really deep analysis then AppDynamics is really powerful (commercial software with a free trial) https://www.appdynamics.com/java/

practicalli-john12:09:41

there is also https://prometheus.io/ which is an open source app for monitoring and paging which Signal Media uses (and was presented in their talk this week).

firthh12:09:11

There is a risk with Prometheus along with a lot of other monitoring software, that if memory usage spikes too quickly prometheus won’t scrape soon enough and the metrics look fine right up until your container gets killed

firthh12:09:34

But I think that’s entirely dependant on the workload you’re running

practicalli-john13:09:03

Yes, I would be interested to know more about the different tooling. I guess I'll be asking my friends that work at Signal Media about that 🙂

practicalli-john13:09:22

I also notice https://sysdig.com/opensource/ which talks about instrumenting containers and using Prometheus, seems to be part open source and part commercial.

👍 4
firthh13:09:06

I think it’s one of the downsides of the pull model for metrics collection in general, that if things change quickly you might lose metrics right before the container dies

firthh13:09:28

I think the risk still exists to some extent for the push model depending on the implementation. Some of the StatsD collectors for metrics I’ve seen run in another thread, so even there is still a risk that all of your metrics haven’t been pushed before the container dies

practicalli-john21:09:16

i've used ELK stack where we pushed logs and metrics from the app to the ELK stack via its api. We had a lot of metrics (30 million transactions a day), so it was tricky to see if anything was missing and we never had problems with our applications crashing.

rickmoynihan21:09:12

I’d highly recommend datadog. We’ve enabled dozens of timeboards etc in it, and have detailed metrics on our apps. They support the JMX stuff too, so you can push all those events through the datadog (statsd++) daemon. We don’t run our clj/jvm apps in containers, so no problems for us there.

rickmoynihan21:09:48

agree also on jvisualvm but I only really use it for local dev/tuning… on that front yourkit is even better though (but paid for). Also eclipse mat is good for memory leaks, and finding where a lazy seq is holding head etc…

alexlynham13:09:05

I've had some success with monit and docker

alexlynham13:09:47

b/c monit is quite lightweight and you can set the interval you can probably catch things... though the difficulty is capturing the error as & when