Fork me on GitHub
#clojure-uk
<
2018-02-27
>
yogidevbear08:02:52

I find that a comical thought for some reason (re Malcolm cursing)

yogidevbear08:02:09

Morning Craig simple_smile

yogidevbear08:02:09

How are you doing this fine snowy morning?

mccraigmccraig08:02:43

my kids are all running around in the snow this morning, but i've been ill and am confined indoors 😭

yogidevbear08:02:15

Oh dear. Then I won't mention that white stuff again

mccraigmccraig08:02:22

ha, it's fine - i'm off to norway on wednesday for some snowboarding with my eldest, so i'll get a good dose of snow... just a bit sad to be missing out on the snowman building and snowball fights though

yogidevbear08:02:35

Lovely. I'm very envious

dominicm08:02:48

We have such a sprinkle of snow, it's pitiful.

otfrom08:02:08

got a bit of snow in Dundee. Not much yet. We're supposed to get our big bit tomorrow

otfrom08:02:16

oh, and "morning"

Rachel Westmacott08:02:29

my southern train ran this morning:exclamation:

mccraigmccraig08:02:34

@peterwestmacott you should perhaps save your surprise for after your home-train 😬

Rachel Westmacott08:02:43

I’m half way through building a lego rocket with my kids, they’ll be disappointed if I’m late home.

dominicm08:02:23

I bet you can't wait until it's complete. You'll never need depend on Southern again.

Rachel Westmacott08:02:01

It isn’t modelled on Stephenson’s Rocket!

otfrom09:02:45

I think ballistic commutes are the future of transport in the UK

otfrom09:02:40

another in a line of stupid clojure questions here...

mccraigmccraig09:02:18

a quick look at bullet acceleration gives figures in the ~10,000g range... i'm not sure can cope with that, even eyes-in @otfrom

otfrom09:02:25

if you are maintaining a databases as a hashmap. Is it worth having it as a atom of a hashmap with <k, atom> pairs?

otfrom09:02:56

@mccraigmccraig ballistic doesn't need to be that fast

mccraigmccraig09:02:16

@otfrom it's quite a long way from brighton to london

Rachel Westmacott09:02:23

@mccraigmccraig to beat Southern I would estimate that 0.1g would be more than sufficient.

mccraigmccraig09:02:49

@otfrom why are you wanting the inner atoms ?

otfrom09:02:14

@mccraigmccraig partly so that I can work on them independently of the outer atom

otfrom09:02:23

partly to reduce contention on the data structure

mccraigmccraig09:02:27

@otfrom and have you considered agents too ?

otfrom09:02:48

IIUC if I swap! the inner atom it won't cause an issue w/the outer atom (which I'm also mutating)

otfrom09:02:03

I'll have a think about agents

otfrom09:02:25

I'm a bit worried with this design that I'm going full erlang. And you should never go full erlang. 😉

mccraigmccraig09:02:54

correct - swap!ing the inner atom won't cause any contention on the outer atom (since the value of the outer atom remains unchanged)

otfrom09:02:57

I'm gonna see the design out w/core.async and then see where it chafes

otfrom09:02:16

(so basically a go loop per inner atom)

otfrom09:02:30

(getting every event from the channel)

mccraigmccraig09:02:03

a while back i had an atom which was suffering from contention and swapped it for an agent and everything was happy again... i think it's the only time i've ever used an agent in anger

mccraigmccraig09:02:34

what's the thing that you are doing?

otfrom09:02:17

creating audit trails for various resources (so consuming events across different domains and a single event can update a lot of things)

otfrom09:02:33

more embarrassingly parallel than concurrent

otfrom09:02:53

bloody idea was keeping me up last night

otfrom09:02:12

agents look good. It is more of the channel metaphor I'm after though

otfrom09:02:39

as finding the right things to mutate is actually the issue

otfrom09:02:58

so I'm just letting the go loops look after a single resource each and made the decision for its one thing

otfrom09:02:06

(which is the bit of design that I like the most about it)

otfrom09:02:15

(and the bit the feels most erlang like)

mccraigmccraig09:02:37

ah, i see - so you will send events to all the channels and each go-block will decide whether to apply to its atom or not ?

otfrom09:02:16

being able to combine it with filtering transducers feels good too

mccraigmccraig09:02:29

well it sounds very simple to implement - but you are multiplying your event flow so you might have throughput problems if you have too many keys to update

otfrom09:02:35

ok, so filtering early will help with that. I was wondering where the bottlenecks might be

mccraigmccraig09:02:26

how many keys and what event rate are you talking about ?

otfrom09:02:51

the event rate is a batch so I'm limited by whatever gets most contended

otfrom09:02:28

some will be noddy (only a few 100s max). Some might be up to 10k

otfrom09:02:57

but I can filter quite a bit of the event stream as some things aren't interested in some of the messages

mccraigmccraig09:02:35

i'd go with the simplest solution first and iterate if you hit any throughput problems

otfrom09:02:54

that was my plan

otfrom10:02:27

I do like how simple it makes it

mccraigmccraig10:02:04

simple and obviously correct is a valuable attribute of an event processing system 🙂

otfrom10:02:05

I was going to do something really wacky with specter otherwise

mccraigmccraig10:02:53

i've never played with specter - is it good ?

otfrom10:02:03

some on the team like it

otfrom10:02:07

I've not had a go yet

otfrom10:02:35

looks like an interesting alternative to zippers for some things and datascript for others

maleghast10:02:57

Morning everyone...

maleghast10:02:32

@otfrom - I am fascinated by ^^ about atoms withing atoms... Also WTF is Specter, and how does the maintainer justify the misspelling?

maleghast10:02:43

@otfrom - I Googled... Initial reaction is "wow", even with the dodgy spelling.

mccraigmccraig10:02:44

hmm @otfrom i'm not sure the atom within atom thing works - creating go blocks is a side-effect and you don't want your swap! fns to have any side-effects, since they can (and do) get repeatedly applied

otfrom17:02:30

the go block isn't in the atom. It just has a reference to it and can change it as it receives things on its channel

mccraigmccraig17:02:12

right, but if the go-block is created in the outer-atom's swap! then it's a side-effect of that, and you may get multiple go-blocks created for a single key

otfrom17:02:52

it isn't in the out atom's swap either

otfrom17:02:04

(or at least I don't think it will be)

otfrom17:02:40

probably create the inner atom, swap! it into the outer atom and then hand the inner atom off to the go block

dominicm10:02:44

specter's also fast

mccraigmccraig10:02:48

although an agent with inner atoms should be ok @otfrom

mccraigmccraig10:02:11

i think i should look at specter for our re-frame app... it looks like it might make a bunch of our app-db transforms a lot simpler

mccraigmccraig10:02:21

have you used it lots @dominicm?

dominicm10:02:40

@mccraigmccraig low-medium. I understood it for about a day or two.

dominicm10:02:03

Novice is probably the word I'm looking for. I know where it would help, and roughly what it provides.

yogidevbear10:02:55

I used it for the first time the other day. Made traversing nested vectors quite easy

dominicm10:02:48

I hear a lot "the problem might be your data structure" with regards to Specter.

yogidevbear10:02:04

So would that be interpreted as "bend your data to fit the tooling"?

dominicm10:02:46

Often it's self-inflicted 😛

dominicm10:02:52

Like the shape of your app-db 😉

mccraigmccraig10:02:12

that's pretty much the lisp philosophy @yogidevbear - bend your language to be easily transformable by the tooling

maleghast11:02:41

Is is just me that is offended that he library is not called Spectre?

maleghast11:02:46

You know, with the word spelled not only correctly, but also in such a way as to preserve the meaning of the shadowy organisation that dogs the heels of James Bond..? (the Special Executive for Counter-intelligence, Terrorism, Revenge, and Extortion)

chrjs11:02:46

Morning ❄️

bronsa11:02:27

I subscribe to the school of thought that specter is solving the wrong problem

bronsa11:02:46

the performance optimizations it uses are undeniably good, but also completely unnecessary to me

bronsa11:02:07

in my experience when - data gets too nested, I don’t reach for a better zipper or a different lens abstraction, but I flatten it out instead and use some querying language over the flat db - the default updating operators get too slow, micropotiminzing a few microseconds away is not enough, it means I have to switch data representation/data structure

mccraigmccraig11:02:49

(specter noob warning) the other thing specter seems to offer is type-preserving transformations - that seems quite useful ?

dominicm11:02:46

I have found myself writing a few type-preserving transformation protocols for places where I expect the algorithmic complexity to not be problematic.

bronsa11:02:53

you don’t need a massive dsl for that

bronsa11:02:11

and the time it actually matters IME are really small

bronsa11:02:31

but sure, everybody’s experience is different

bronsa11:02:41

@dominicm I don’t know how that library works but if it does translate the nested DS in a flat EAV-like db, then yes

dominicm11:02:32

Pretty much, using datascript as the storage engine.

dominicm11:02:41

Thinking about it, I did find odin better for this stuff.

yogidevbear11:02:53

I have a Clojure question. I have the following function:

(defn F
  [X Y C]
  (let [_C (get-point-value X Y)]
    ;...
    (map
      #(if (= _C (get-point-value (inc (first %)) (inc (last %))))
          (recur [(inc (first %)) (inc (last %)) (str "\"" C "\"")]))
      (get-adjacent-points [X Y]))))
I get the following error when running (F 1 1 "X"):
ClassCastException java.base/java.lang.String cannot be cast to java.base/java.lang.Number  clojure.lang.Numbers.inc (Numbers.java:112)
The offending line is: (recur [(inc (first %)) (inc (last %)) (str "\"" C "\"")])) If I adjust the function to the following...
(defn F
  [X Y C]
  (let [_C (get-point-value X Y)]
    ;...
    (map
      #(if (= _C (get-point-value (inc (first %)) (inc (last %))))
        (println [(inc (first %)) (inc (last %)) (str "\"" C "\"")]))
        ;(recur [(inc (first %)) (inc (last %)) (str "\"" C "\"")]))
      (get-adjacent-points [X Y]))))
... and run (F 1 1 "X"), then I don't see the error, but rather:
[2 1 "X"]
[1 2 "X"]
which is correct for the scenario. My question: Does anyone know what might be causing the issue in the offending line?
(recur [(inc (first %)) (inc (last %)) (str "\"" C "\"")]))

bronsa11:02:44

user=> (println ["foo" "\"bar\""])
[foo "bar"]

bronsa11:02:03

2 and 1 in your output could very well be strings

bronsa11:02:08

as that exception suggests

dominicm11:02:23

prn is good for this 🙂

yogidevbear11:02:08

If I change it to prn I get:

[2 1 "\"X\""]
[1 2 "\"X\""]

yogidevbear11:02:36

Or better yet, if I adjust it to:

(prn [(inc (first %)) (inc (last %)) C]))
I get
[2 1 "X"]
[1 2 "X"]

bronsa11:02:22

(inc (last %))

yogidevbear11:02:23

Alright, if I change my original function to:

(defn F
  [X Y C]
  (let [_C (get-point-value X Y)]
    ;...
    (map
      #(if (= _C (get-point-value (inc (first %)) (inc (last %))))
        ;(prn [(inc (first %)) (inc (last %)) C]))
        (recur (prn [(inc (first %)) (inc (last %)) C])))
      (get-adjacent-points [X Y]))))
I get the following error: NullPointerException clojure.lang.Numbers.ops (Numbers.java:1018)

bronsa11:02:24

in your recur

bronsa11:02:27

last % is your string

bronsa11:02:38

after first recursion

bronsa11:02:02

you’re recurring to the target point of #(..)

bronsa11:02:06

not of your F function

dominicm11:02:35

@bronsa I'm curious what querying language(s) you've used on the flat db, and what your experiences are with that?

yogidevbear11:02:45

What would be the idiomatic lazy way of recuring to F?

bronsa11:02:59

I’ve used odin, datalog, and a lib I’ve written myself (still private tho)

dominicm15:02:05

do you mean https://github.com/martintrojer/datalog ? or something else for datalog?

bronsa15:02:13

nah, datascript

dominicm11:02:06

(F …)? recur isn't lazy though.

bronsa11:02:53

@dominicm I massively prefer declarative querying over “structural” navigation, for expressiveness and power

yogidevbear11:02:54

I thought recur did work in a lazy fashion as it doesn't hold onto the tail (not sure I'm using the right terminology here)

bronsa11:02:06

and updating a flat db is trivial so you don’t need to reach for crazy DSLs

dominicm15:02:19

ah, I presumed you flattened for querying (as a parallel read-only copy)

bronsa15:02:01

I mean that could work too but if you work on flattened data from the start then you get easy updates for free

dominicm16:02:51

Seems difficult to model, is it?

bronsa16:02:58

not if you do the flattening on insertion

dominicm16:02:03

I suppose if you have datalog, then it's much easier

yogidevbear11:02:19

Or only holds onto the tail or something

bronsa11:02:34

recur has nothing to do with laziness or head holding

bronsa11:02:43

recur has to do with tail call optimization

bronsa11:02:48

and doing recursion w/o growing the stack

bronsa11:02:55

but in your scenario it’s just not possible to use it

yogidevbear11:02:38

Ok cool, thanks for clarifying that 🙂

yogidevbear11:02:34

Yay! It works! 🎉 aw_yeah

mccraigmccraig13:02:29

anyone tried out https://github.com/dcos/terraform-dcos ( @otfrom ? ) - i have to rebuild my dc/os cluster (to increase a sysctl dammit) and terraform-dcos looks like it might be a better approach than the mesosphere AWS CF templates (which work fine, but don't support upgrades)

mccraigmccraig13:02:16

ah, yeah - we talked about it a while ago 🙂 but slack is terrible at history, and my memory is poor

mccraigmccraig13:02:29

how has terraform-dcos been for you ?

dotemacs13:02:02

it’s not a module, so if you don’t care about that, it’s fine

mccraigmccraig13:02:15

what does that mean ?

dotemacs13:02:25

the annoying thing is that it creates the cluster with node’s port 22 exposed to the world

dotemacs13:02:39

so people will scan you constantly, but that should not really matter

dotemacs13:02:48

or better scan your port 22

mccraigmccraig13:02:06

it creates it with PK only ssh auth though ?

mccraigmccraig13:02:44

ok, so a bit poo but probably not actually insecure

mccraigmccraig13:02:07

can you specify custom AMIs for the different roles (master, public-agent, private-agent) ?

dotemacs13:02:28

about the module thing: when you do terraform, you can include some libraries as a module and then have your main setup just use that module. But the way this setup is created, you’ll be using this as the “the main thing” and adding your own tweaks to it

mccraigmccraig13:02:51

i see... i'm ok with that i think

dotemacs13:02:03

I think that it’s fairly rigid in the sense that you specify one AMI for it all

dotemacs13:02:16

but you can tweak the size of the bootstrap node and agents

mccraigmccraig13:02:27

the CF templates are the same - one AMI to rule them all

dotemacs13:02:28

but you’re sort of stuck with the single AMI across the cluster

mccraigmccraig13:02:03

it's a bit pants, when you want your private agents to have a load of storage and your public agents and masters not so much

dotemacs13:02:28

I think that you can tweak that

dotemacs13:02:42

i think that i provided the tweaks for that to them

dotemacs13:02:46

let me check

dotemacs13:02:31

yea, you can set the disk size per functionality for sure (master, bootstrap, agent…)

dotemacs13:02:40

So you’re not constrained there at all

mccraigmccraig13:02:42

currently i'm specifying a bunch of EBS mounts, which i can then use as dc/os MOUNT volumes

mccraigmccraig13:02:10

it makes for easy isolation of the storage for my persistence services

dotemacs13:02:11

The thing that fsck-ed me up is that they specify a particular AWS DNS server on setup, but only for Google cloud. And if you don’t declare it for AWS, when you try to boot up Kafka, it just won’t work. Kafka broker won’t be able to look up any of the agents it has. It just won’t resolve them. So you must have this declared before hand. It’s sort of tribal knowledge… I really should do a PR for this…

mccraigmccraig13:02:15

is that just a tf variable ?

mccraigmccraig13:02:42

or something else ?

dotemacs13:02:55

👀 one sec

dotemacs13:02:10

yes, that’s it. So the default is fine there

dotemacs13:02:16

it seems they fixed it

dotemacs13:02:26

that bit us once…

mccraigmccraig13:02:18

cool... well i guess i'll give it a go and see how far i get - thanks for the help 🙂

dotemacs13:02:59

no problem. one thing that you might want to do is declare a s3 bucket, which you should create before hand, to store the terraform state in

dotemacs13:02:41

Do let me know what you do for monitoring

mccraigmccraig13:02:54

is that the dcos_aws_template_storage_bucket tf var ?

mccraigmccraig13:02:26

or the dcos_s3_bucket var ?

dotemacs13:02:36

the first one

dotemacs13:02:53

the second one is related to the exibitor’s setup/backup

dotemacs13:02:30

but the thing is, I think that you need to set up your s3 bucket before you start running this setup… from what I remember

dominicm16:02:24

http://www.bbc.co.uk/news/uk-england-coventry-warwickshire-41108152 I just had a letter through the door about disbanding the council, this was one of their reasons. It Shocked me.

jasonbell17:02:31

Councils are happy to attempt to fill empty space and get the retailer on the business rates instead. Been on the rise for a while.

dominicm17:02:36

What's business rates?

jasonbell17:02:46

A business would pay the rent on the lease of the property and also rates to the council (policing and so on). They are two different payments. A lot of landlords/councils will sweeten a deal by writing off the rent but still expect the business rates to be paid.

dominicm19:02:51

> Councils are happy to attempt to fill empty space I think this part particularly surprises me. Why not use the reduced prices to make it more appealing for a local business to open? They always bring something unique to the area.