This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-11-22
Channels
- # aws (1)
- # beginners (102)
- # boot (5)
- # cljs-dev (59)
- # cljsjs (1)
- # clojure (154)
- # clojure-australia (1)
- # clojure-brasil (1)
- # clojure-dusseldorf (4)
- # clojure-greece (36)
- # clojure-italy (10)
- # clojure-poland (5)
- # clojure-romania (1)
- # clojure-russia (7)
- # clojure-spec (32)
- # clojure-uk (113)
- # clojure-ukraine (3)
- # clojurescript (107)
- # cursive (13)
- # data-science (25)
- # datomic (23)
- # emacs (3)
- # events (1)
- # fulcro (72)
- # funcool (10)
- # graphql (1)
- # leiningen (1)
- # luminus (2)
- # lumo (38)
- # off-topic (14)
- # onyx (78)
- # planck (4)
- # re-frame (55)
- # reagent (1)
- # ring (3)
- # ring-swagger (2)
- # rum (19)
- # shadow-cljs (89)
- # spacemacs (101)
- # sql (2)
- # unrepl (88)
Slow start in #clojure-uk land
Morning @jasonbell 👋
mornings
mweurggh
mwaaggghgh
That bad, huh?
Takes a massive swig from my coffee mug to calm my heart rate
Thank you all for the feedback on AWS, etc. yesterday. It's good to see what people are using and their reasoning behind it. @mccraigmccraig I'm a little confused as to where you're using EC2/ELB/Auto-scaling and DC/OS. Is the AWS stuff for the app code (Clojure/ClojureScript/etc) and DC/OS for your data layer, etc?
@yogidevbear the DC/OS agents are in auto-scaling groups (one for private agents, one for public agents) - when i want more (or less) capacity for my cluster i just fiddle with the auto-scaling group params
i use ELB for TLS termination and to balance across the (yada) API instances running on the public agents
all the data layer runs on private agents
So the DC/OS stuff is hosted on AWS?
Okay cool
Thank you 👍
DC/OS effectively turns your instances into a big bag of compute/ram/disk and tasks get run somewhere in that bag, and moved around when necessary
Yeah, that makes complete sense now
there's an IP overlay network, and a service DNS, so containers for tasks get their own IP address and a well-known DNS name - which makes it easy to deploy tasks and easy to discover running tasks
I've watched the demo videos on their website and was in awe of the awesomeness of DC/OS. It looks like a fun tool to play with
it's not all roses - i've just run in to a problem with the datastax-enterprise package which is quite opaque and doesn't provide the configuration to work around the problem
I want to be able to add autoscaling to secondary indexes on a dynamodb table. On the web UI this is 2 clicks. I can then use the CLI to see what change this made to the scaling policies for dynamodb. When I try and add those same scaling policies via CloudFormation I get the error "Unsupported service namespace, resource type or scalable dimension"
(Because the scalable dimension of my new targets are dynamodb:index:[Read|Write]CapacityUnits instead of dynamodb:table:[Read|Write]CapacityUnits)
We've hit a number of problems like this. It's not possible to set the Path of an IAM role via CloudFormation, or it didn't used to be, so we had to have CLI scripts to create our roles where we needed a path, instead of using CF
It's not new stuff, it's just stuff that the majority of people don't notice/care about/use
But when you do get to the stage of wanting to use it, you've probably got hundreds of things you want to use it on, so not being able to do that via your standard CF deployment pipeline then becomes a massive PITA
@mccraigmccraig - Do you have the Enterprise version of DC/OS?
no @maleghast just community
enterprise buys you fine-grained authentication, which would be nice, but i can get away without it for now
Oh, shame, I was hoping to ask you if you’d ballpark the prices… Also the Enterprise page / site does say that they will offer you support on getting Cassandra working, so I assume that they have some insight into working with Datastax Enterprise…
i had a call with mesosphere about enterprise a while back @maleghast - here's my note on pricing: enterprise dc/os annual pricing per node - $3950/node/year - non-production nodes $3250/node/year
those were approximately my thoughts
The Enterprise site does seem to intimate that they might sell you one-off support if you get REALLY stuck with the issue you mentioned ^^, but they are pricey buggers, huh?
there is a dc/os slack, and the mesosphere people on there are helpful and never ask if you have enterprise
I mean they are going to get paid by any Public Company as and where it needs the indemnity, so I suppose they take a more long term view with emerging companies, which is wise on their part - builds strong goodwill if nothing else.
datastax have similarly eye-watering prices, but will also enrol you in their startup programme, which gives you everything for free until your revenues hit $5m or your funding $100m
Note to self - if ever buying this kind of thing have the money to also have big, crunchy nodes…
@mccraigmccraig That’s pretty cool too - I like the model. Cognitect appear to have a similar approach for Datomic too, which allayed some of my sticker-shock on that one.
@maleghast I'd say that was a good idea anyway as jvm things use up RAM fast
@otfrom - Oh, don’t I know it… I am currently running the largest AWS instance I have ever been responsible for because of that…
I must admit that the RAM issue is one of the things that’s putting me off the idea of using Docker with the app(s) I am currently prototyping.
biggest I run at the moment are a few r3.8xlarge machines for a spark job that gets a bit chunky in the middle (like me)
I'm a bit embarrassed tbh. There should be a way to do it in much less RAM, but I've not found it yet.
I had basically never spun up anything larger than a T2 before I spun up my / our m4.large
an 8xlarge is pretty darn chunky @otfrom
what do you have that requires 0.25TB of RAM @otfrom?
i guess a bunch of massive hash-joins would do it
it at least works and only takes a few hrs to run on a 3 node cluster w/an extra master (that is smaller)
they can be pretty cheap on spot (the larger instances often are and the prices can be quite stable, esp if you choose a slightly "unfashionable" one)
no reasonable possibility of sort-merge joins then @otfrom?
@mccraigmccraig I'd have to do the sorting (as it is just a big pile of badly formatted csv files)
sure - probably wouldn't take many passes of some smaller machines though, and would give you a fixed-memory solution
though doubtless slower
but possibly cheaper if slower is still fast enough (I've got about 18hrs to turn it around in)
"The sort-merge join can be activated through spark.sql.join.preferSortMergeJoin property that, when enabled, will prefer this type of join over shuffle one."
if you are using spark-sql that would presumably make it happen automatically
using sparkling, but thinking about the way I'm doing it I could probably do it in plain old clojure (need to think a bit about it)
Ha this is brilliant https://beta.companieshouse.gov.uk/company/10542519
Maybe it did and they had a backup 😉
Looks like this is an expensive hobby: https://beta.companieshouse.gov.uk/company/08768324
I usually use aleph, because it is close at hand.
@otfrom aleph so i don't have to convert to manifold/deferred
Did something happen @mccraigmccraig
i think i've just found the source of my frustration with my apparent inability to effect a change in a docker image for the last 2 hours
and it is because whoever created the image installed 2 copies of the relevant packages at different paths. and i've been changing the unused one
@otfrom - I used to be all about the http-kit and then I met Aleph… Never looking back now…
I’ve been thinking about this in the background of my mind for a while, so I am just going to come out and ask y’all… Who’s using Docker in Production (with Clojure, obvs), why’d you decide to Containerise, do you run Docker in dev as well as your deployed envs and finally are you finding it’s possible to run on meagre resources?
I have had really satisfying experiences with Docker in Python and Ruby, but I have heard some really negative things about it vis à vis the JVM in particular…
we're using it for production and staging... not for local dev though
yeah, a compiled uberjar
That seems like a solid idea - do you use a lightweight basebuild like Alpine etc., or do you base your containers on Ubuntu / CentOS type of thing?
Also, I’ve heard people talking about Docker having issues with running Oracle Java…?
it varies - our api is on ubuntu and stream processing on alpine/oracle ... not for any particularly considered reasons
nope, no issues with oracle java - i've always used oracle java for production deploys of everything
haha only for people with limited google-fu - https://gist.github.com/c801baa7e2378b19ea02686165d68c6f
@mccraigmccraig - thanks for the Gist... why do you add OpenJDK and then install Oracle Java though?
the danger of comments @maleghast - i don't add OpenJDK, but i probably did at some point in the past
Any PS4 owners here? I'm getting one for Christmas as a joint gift for my son and I. PlayStation are running their official Black Friday deals from tomorrow. Huge discounts on loads of titles including latest releases. https://store.playstation.com/en-gb/grid/STORE-MSF75508-BLACKFRIDAY17PS4/1?smcid=pdc%3Aen-gb%3Apdc-buy-playstation-store