Fork me on GitHub
#clojure-uk
<
2018-09-05
>
guy06:09:26

Morning!

otfrom07:09:21

Madainn mhath

otfrom07:09:03

had a fun trial ride on a brompton (h3l) yesterady. Gonna try a brompton m3l today (slightly lower handlebars, but otherwise the same). cc @mccraigmccraig and @agile_geek;-)

otfrom07:09:28

and I did my first riding in London and didn't die (a quick round the houses in Covent Garden)

πŸ‘ 24
agile_geek07:09:47

I find London safer than some of my commute to Newcastle at home as 5-6 miles of that is on open A roads with 60mph limits. Nothing in London goes faster than about 20mph... except the bikes!

maleghast08:09:25

Not dying == ++Good

maleghast08:09:00

Also, London Cycling is a LOT safer than it was 10-12 years ago... Drivers are much more bike-aware than they were, there is less density of traffic, there are better provisions on-road for cyclists... I could go on, but the number of people cycling is still going up and the number of deaths is going down, so shrug Still, do be careful and conservative / defensive in your thinking - better to be late than squashed etc...

otfrom09:09:43

@agile_geek I had a similar bit when cycling in Pennsylvania. There was a bit on a state route where the speed limit was 45 and the road was bendy but the locals thought 60 was a good speed to go

mccraigmccraig07:09:56

cycling is by far my fave way to get around central london

guy07:09:55

if u cycle lots are you ever worried about the pollution produced by cars you are near by. If that makes sense? Like would a mask help?

agile_geek07:09:35

I can't ride with a mask...feels like I'm suffocating!

guy08:09:26

Yeah i would just be worried of the car exhaust fumes etc 😞

mccraigmccraig08:09:03

a mask might help with particulates, although probably not with gases... but what @agile_geek said - it's horrible wearing a mask

βœ”οΈ 4
mccraigmccraig09:09:02

the cycle-courier was better off than everyone apart from the office worker

guy14:09:02

thanks for this! really interesting πŸ˜„

practicalli-johnny15:09:17

I was a cycle courier for a year in london back in 2005-6 and I am still alive. I've cycled to work in London almost every day since 2006. I did try masks initially, they certainly cut down the particles, but were horrible to wear. Almost every car and bus is a lot cleaner now than 10 years ago so i no longer come home with a face covered in soot. If you cycle fast enough, I am sure you dont get any pollution πŸ™‚

❀️ 4
thomas07:09:50

mogge 😼

agile_geek07:09:49

Bore da pawb welsh_flag

maleghast08:09:53

Morning Everyone!

maleghast08:09:11

How's everyone doing?

maleghast08:09:58

Quick question... How do those of you using Docker manage temporary filesystem usage (in this case for an HTTP upload that will then be pushed to S3)?

maleghast08:09:59

also, side-issue, is everyone going uber-jar -> Docker Container -> Deploy, or are some people running in a more interactive manner..?

jasonbell09:09:16

@maleghast Personally I'd mount a volume outside of the container on machine. It depends on a few factors, size of the docker container and the average size of the files being uploaded. In the early days of messing around with it all I was filling docker containers up with pulled data from kafka, obviously it blows up the container.

maleghast09:09:47

That sounds like a good plan @jasonbell

maleghast09:09:05

I am aware that I risk blowing the container if I am not careful...

maleghast09:09:45

I am going to be deploying into a K8s cluster - is there a received way of accessing host-machine disk to provide for a mounted Volume in a container?

jasonbell09:09:52

@maleghast Even more fun when it's handled by Mesos/Marathon. Yay! Boom! Yay! Boom! Yay! Boom! Repeat.... πŸ™‚

jasonbell09:09:31

Never tried on K8s sadly so I don't the know the answer to that.

maleghast09:09:33

@jasonbell - No worries, thanks for the input / answer regardless - it confirmed something that I suspected πŸ™‚

maleghast09:09:30

(I have asked in #docker)

mccraigmccraig09:09:35

@maleghast another option is to give your container filesystem a few gigs to play with ... but if you mount something from the host, make sure that it's not from any partition where filling that partition up would cause badness (e.g. if say you have ZK coordinating your dc/os cluster you really don't want the partition it's writing to to fill up)

maleghast09:09:17

@mccraigmccraig - Yeah, I was wondering if I could do this safely as well...

mccraigmccraig09:09:29

here are the block-storage mounts i ended up using on my dc/os hosts: https://gist.github.com/mccraigmccraig/52b0b11dfef35487b457d9622a443802

maleghast09:09:39

I mean, as long as I clear out the temporary fs location to S3 regularly then even a few hundred MBs would be sufficient - it's an image uploader for jpegs only, and we are talking about logos, so not HUGE files.

practicalli-johnny16:09:02

@maleghast treat your docker file space as completely transitory and you will save yourself a lot of pain. Writing to your files space makes your docker statefull, leading to the usual issues of managing state across multiple instances. What happens if an instance dies between uploading the file and persisting that file elsewhere. Expect your docker to be wiped out at any moment and be replaced by a new container from your docker image and your architecture will be much more robust and scalable. Or if loosing data is not that important, then please ignore πŸ™‚

maleghast16:09:40

Yeah, I absolutely feel this way about the actual container in almost all circumstances, but a Volume mounted in to the container / containers that is on a native filesystem should__ be ok even if one or more of the containers were to die and / or be replaced.

maleghast16:09:46

What I want__ to be doing is streaming the upload to S3 without the interim step to the / a FS, but mccraigmccraig suggested that there are some performance concerns with doing this...

maleghast16:09:58

(in that it locks up a thread)

maleghast16:09:15

But this particular application is small, internal only and will never need to be all that performant or resilient as it will only ever have a handful of users.

maleghast09:09:55

@mccraigmccraig - Cool, thanks πŸ™‚

mccraigmccraig09:09:45

in particular, /var/log and /home/centos (or wherever you put home dirs in your distro) are on their own partitions - that's just generally a good idea... the /var/lib/* stuff depends on what you are running

mccraigmccraig09:09:17

oh, and the /var/lib/docker is where docker puts its filesystems, so you probably want that on it's own partition

maleghast09:09:36

I was even considering provisioning some EBS to run next to the K8s cluster and using that, but I think that it might be massive overkill

mccraigmccraig09:09:24

some early dc/os learning (and i presume it will be similar for k8s), is that you don't want things like uncontrolled logging, or accidentally filling up a filesystem when you gunzip an archive, to bring down your cluster

mccraigmccraig09:09:21

@maleghast what dyu mean "EBS to run next to the K8s cluster" ?

maleghast09:09:12

You can provision storage-as-a-service on AWS, I would have my containers mount space on that.

maleghast09:09:24

(I may have misunderstood how EBS works)

mccraigmccraig09:09:00

oh, yeah, i generally run all my EC2 instances with an EBS root volume, then mount a bunch of other EBS volumes (at the mount points in the gist above in our dc/os case)

stuarthalloway13:09:09

@U0510KXTU that seems not-right πŸ™‚. We will try to repro.

steveb8n13:09:24

thanks. let me know if I can provide more info

stuarthalloway13:09:29

that was a good problem description already!

jaret12:09:55

@U0510KXTU I am working on reproducing this error for the team to review. Would you be able to put your rule and query in a gist? I was able to invoke a rule without running into an error and I want to look at the specifics of your case and incorporate them into my testing.

mccraigmccraig09:09:50

we use terraform to generate the clusters, 'cos doing it manually would be awful

maleghast09:09:31

I am not going to use DC-OS - first port of call is a small K8s cluster for pre-prod / semi-live environment, but I think that I will be using Terraform and / or KOPS to build this first environment.

mccraigmccraig10:09:06

i'd see how it goes with k8s @maleghast - i'm thinking of ditching dc/os for k8s anyway, largely because EKS and google-cloud seem to turn the "building and managing the cluster" piece into a service

maleghast10:09:01

Yeah, I am really hoping that EKS takes the headache away from me, but I don't want to have my workloads deployed in the US, so I am waiting for EKS to be available in EU-West-1 (Ireland) before I adopt it.

alexlynham10:09:46

@maleghast I've done a fair bit of file shuffling inside containers

alexlynham10:09:52

granted in the early days (2015 or so)

alexlynham10:09:01

but basically an uploader with a temporary file

alexlynham10:09:45

as I knew it was sub 20MB in general and the files would be definitively cleaned up on failure & the container had space it wasn't too bad

maleghast10:09:54

Thanks @alex.lynham - I think that I will go this route; seems the simplest. I want to re-factor the code to stream to S3 without the tempfile, but I am not going to have time to do that right now__

alexlynham10:09:59

that said, the only larger files I've had to deal with from a process in docker I pulled them into a mounted volume

alexlynham10:09:09

so they had access to the host filesystem for storage

alexlynham10:09:17

those were like potentially in the GB range tho

maleghast10:09:29

Yeah, that's what I would do under those circumstances too.

alexlynham10:09:38

just make sure you clean up files otherwise strange stuff happens

alexlynham10:09:57

I had a thing that was a bit too lenient on failing and retrying and it kept doing weird stuff locally

maleghast10:09:18

While we are on this subject, what are everyone's favourite Docker Container Registries for when you need private registry...

alexlynham10:09:25

in the end made it ruthlessly nuke the file whatever happened and that was easier to reason about

alexlynham10:09:31

just use docker hub private imo

alexlynham10:09:53

although AWS registry is nice n' easy if you're using EKS/ECS

maleghast10:09:41

I was wondering if the AWS one would work well, but the online docs only talk about ECS integration, so I was wondering about alternatives

firthh10:09:08

I think as far as the docker command line tool is concerned they all work. You should be able to pull/push from private dockerhub, aws, google cloud. They have slightly different UIs but they implement the same API for docker to talk to

firthh10:09:16

Or at least that’s my understanding

firthh10:09:58

We’re on Google Cloud and I need to run gcloud auth before hand but I can pull our private images then by just running docker pull

firthh10:09:06

You can even deploy your own docker registry if you want

maleghast11:09:57

@firthh - Thanks, that's all pretty reassuring. I don't want to run my own registry, because I don't want the fixed cost of an ASG, 2 EC2s and ... etc...

maleghast11:09:37

But I am pretty sure that I can use AWS ECR, and that keeps everything in the same bill for our company (except Github)

yogidevbear12:09:03

Morning πŸ™‚

πŸ‘‹ 12
maleghast13:09:08

@bronsa - which update has you most excited?

bronsa13:09:26

it's the new error messages changes

maleghast13:09:33

I thought as much πŸ™‚

bronsa13:09:16

user=> (ns)
CompilerException clojure.lang.ExceptionInfo: Call to clojure.core/ns did not conform to spec:
val: () fails spec: :clojure.core.specs.alpha/ns-form at: [:args :name] predicate: simple-symbol?,  Insufficient input
 #:clojure.spec.alpha{:problems [{:path [:args :name], :reason "Insufficient input", :pred clojure.core/simple-symbol?, :val (), :via [:clojure.core.specs.alpha/ns-form :clojure.core.specs.alpha/ns-form], :in []}], :spec #object[clojure.spec.alpha$regex_spec_impl$reify__2436 0x44e3a2b2 "clojure.spec.alpha$regex_spec_impl$reify__2436@44e3a2b2"], :value nil, :args nil}, compiling:(NO_SOURCE_PATH:1:1)

bronsa13:09:18

user=> (ns)
Syntax error macroexpanding clojure.core/ns at (1:1). Cause: Call to clojure.core/ns did not conform to spec:
val: () fails spec: :clojure.core.specs.alpha/ns-form at: [:args :name] predicate: simple-symbol?,  Insufficient input

guy13:09:46

i for one prefer the first one

guy13:09:53

πŸ˜‚

guy13:09:15

nah im just kidding

guy13:09:16

that looks good

maleghast13:09:01

They are greatly improved, I agree πŸ™‚

alexlynham13:09:38

yeah that's a big improvement

thomas14:09:35

less is more...

thomas14:09:41

or something like that.