Morning!

Madainn mhath

had a fun trial ride on a brompton (h3l) yesterady. Gonna try a brompton m3l today (slightly lower handlebars, but otherwise the same). cc @mccraigmccraig and @agile_geek;-)

and I did my first riding in London and didn't die (a quick round the houses in Covent Garden)

cycling is by far my fave way to get around central london

mogge 😼

Bore da pawb

mornings!

Morning Everyone!

😄

How's everyone doing?

Quick question... How do those of you using Docker manage temporary filesystem usage (in this case for an HTTP upload that will then be pushed to S3)?

also, side-issue, is everyone going uber-jar -> Docker Container -> Deploy, or are some people running in a more interactive manner..?

morning

@maleghast Personally I'd mount a volume outside of the container on machine. It depends on a few factors, size of the docker container and the average size of the files being uploaded. In the early days of messing around with it all I was filling docker containers up with pulled data from kafka, obviously it blows up the container.

nods

That sounds like a good plan @jasonbell

I am aware that I risk blowing the container if I am not careful...

I am going to be deploying into a K8s cluster - is there a received way of accessing host-machine disk to provide for a mounted Volume in a container?

@maleghast Even more fun when it's handled by Mesos/Marathon. Yay! Boom! Yay! Boom! Yay! Boom! Repeat.... 🙂

Hehe

Never tried on K8s sadly so I don't the know the answer to that.

@jasonbell - No worries, thanks for the input / answer regardless - it confirmed something that I suspected 🙂

np

(I have asked in #docker)

@maleghast another option is to give your container filesystem a few gigs to play with ... but if you mount something from the host, make sure that it's not from any partition where filling that partition up would cause badness (e.g. if say you have ZK coordinating your dc/os cluster you really don't want the partition it's writing to to fill up)

@mccraigmccraig - Yeah, I was wondering if I could do this safely as well...

here are the block-storage mounts i ended up using on my dc/os hosts: https://gist.github.com/mccraigmccraig/52b0b11dfef35487b457d9622a443802

I mean, as long as I clear out the temporary fs location to S3 regularly then even a few hundred MBs would be sufficient - it's an image uploader for jpegs only, and we are talking about logos, so not HUGE files.

@mccraigmccraig - Cool, thanks 🙂

in particular, /var/log and /home/centos (or wherever you put home dirs in your distro) are on their own partitions - that's just generally a good idea... the /var/lib/* stuff depends on what you are running

*nods*

oh, and the /var/lib/docker is where docker puts its filesystems, so you probably want that on it's own partition

I was even considering provisioning some EBS to run next to the K8s cluster and using that, but I think that it might be massive overkill

some early dc/os learning (and i presume it will be similar for k8s), is that you don't want things like uncontrolled logging, or accidentally filling up a filesystem when you gunzip an archive, to bring down your cluster

@maleghast what dyu mean "EBS to run next to the K8s cluster" ?

You can provision storage-as-a-service on AWS, I would have my containers mount space on that.

(I may have misunderstood how EBS works)

https://aws.amazon.com/ebs/

oh, yeah, i generally run all my EC2 instances with an EBS root volume, then mount a bunch of other EBS volumes (at the mount points in the gist above in our dc/os case)

we use terraform to generate the clusters, 'cos doing it manually would be awful

*nods*

I am not going to use DC-OS - first port of call is a small K8s cluster for pre-prod / semi-live environment, but I think that I will be using Terraform and / or KOPS to build this first environment.

i'd see how it goes with k8s @maleghast - i'm thinking of ditching dc/os for k8s anyway, largely because EKS and google-cloud seem to turn the "building and managing the cluster" piece into a service

Yeah, I am really hoping that EKS takes the headache away from me, but I don't want to have my workloads deployed in the US, so I am waiting for EKS to be available in EU-West-1 (Ireland) before I adopt it.

@maleghast I've done a fair bit of file shuffling inside containers

granted in the early days (2015 or so)

but basically an uploader with a temporary file

as I knew it was sub 20MB in general and the files would be definitively cleaned up on failure & the container had space it wasn't too bad

o/

Thanks @alex.lynham - I think that I will go this route; seems the simplest. I want to re-factor the code to stream to S3 without the tempfile, but I am not going to have time to do that right now__

that said, the only larger files I've had to deal with from a process in docker I pulled them into a mounted volume

*nods*

so they had access to the host filesystem for storage

those were like potentially in the GB range tho

Yeah, that's what I would do under those circumstances too.

just make sure you clean up files otherwise strange stuff happens

I had a thing that was a bit too lenient on failing and retrying and it kept doing weird stuff locally

While we are on this subject, what are everyone's favourite Docker Container Registries for when you need private registry...

in the end made it ruthlessly nuke the file whatever happened and that was easier to reason about

just use docker hub private imo

although AWS registry is nice n' easy if you're using EKS/ECS

I was wondering if the AWS one would work well, but the online docs only talk about ECS integration, so I was wondering about alternatives

I think as far as the docker command line tool is concerned they all work. You should be able to pull/push from private dockerhub, aws, google cloud. They have slightly different UIs but they implement the same API for docker to talk to

Or at least that’s my understanding

We’re on Google Cloud and I need to run gcloud auth before hand but I can pull our private images then by just running docker pull

You can even deploy your own docker registry if you want

@firthh - Thanks, that's all pretty reassuring. I don't want to run my own registry, because I don't want the fixed cost of an ASG, 2 EC2s and ... etc...

But I am pretty sure that I can use AWS ECR, and that keeps everything in the same bill for our company (except Github)

Morning 🙂

https://github.com/clojure/clojure/compare/71511b7800...db94999b54

@bronsa - which update has you most excited?

it's the new error messages changes

I thought as much 🙂

user=> (ns)
CompilerException clojure.lang.ExceptionInfo: Call to clojure.core/ns did not conform to spec:
val: () fails spec: :clojure.core.specs.alpha/ns-form at: [:args :name] predicate: simple-symbol?,  Insufficient input
 #:clojure.spec.alpha{:problems [{:path [:args :name], :reason "Insufficient input", :pred clojure.core/simple-symbol?, :val (), :via [:clojure.core.specs.alpha/ns-form :clojure.core.specs.alpha/ns-form], :in []}], :spec #object[clojure.spec.alpha$regex_spec_impl$reify__2436 0x44e3a2b2 "[email protected]"], :value nil, :args nil}, compiling:(NO_SOURCE_PATH:1:1)

vs

user=> (ns)
Syntax error macroexpanding clojure.core/ns at (1:1). Cause: Call to clojure.core/ns did not conform to spec:
val: () fails spec: :clojure.core.specs.alpha/ns-form at: [:args :name] predicate: simple-symbol?,  Insufficient input

pfft

i for one prefer the first one

🙃

😂

nah im just kidding

that looks good

They are greatly improved, I agree 🙂

wow

yeah that's a big improvement

ace

less is more...

or something like that.