Fork me on GitHub
#clojure-uk
<
2018-04-18
>
jasonbell07:04:48

morning morning

thomas08:04:48

mogge from a warm and sunny Maastricht... working at a customer site for two days.

alexlynham08:04:45

so I've got a question

alexlynham08:04:42

a dev who's mainly .net/java called me up on connascence of execution order in some code... and I was a bit like "uhm, imperative/functional language, it's a pipeline of transforms"

alexlynham08:04:06

am I an idiot because I'm wrong or am I an idiot because I didn't understand the question?

alexlynham08:04:18

he tends to be right about things so I've been mulling.

mccraigmccraig09:04:02

can you give an example of the dependency he was concerned about @alex.lynham?

alexlynham09:04:31

it was a chain of lazy seq processes

alexlynham09:04:49

so in pseudocode something like:

(->> (io/read-from-file)
     (map transform-fn)
     (map send-to-kafka))

alexlynham09:04:24

kind of a hackneyed example, but that's roughly the shape of what I was doing

mccraigmccraig09:04:01

looks pretty sane to me. what was he suggesting would be a less complex/lower-connascence solution ?

alexlynham09:04:37

to close over the transforms in a black box function and move it to a namespace closer to the interface with the external service

yogidevbear09:04:07

Morning ☀️

yogidevbear09:04:14

How is everyone doing today?

mccraigmccraig09:04:32

that would lower comprehensibility, to my fp adapted eye anyways, because i'd have to go away and look up that blackbox fn... and would doing so not be just moving connascence around, rather than lowering overall system connascence ? the blackbox-fn would still have all the pipeline's type and order dependencies

mccraigmccraig09:04:18

although if the pipeline got to be much longer, or was re-used then i'd put some or all of it in a function

danm09:04:19

I mean, that pattern you've got there pretty much matches what we have in all our stuff. In shell there's an initial 'get data' function, and a final 'write data' function, and the middle function is a transform that is in our core namespace. And then that transform might be pretty complex but is entirely pure

danm09:04:39

Why/how would you want to put that shell logic into a black box somewhere?

alexlynham09:04:04

where the code is isn't really as important as how you compose it... I think(?) I mean that's what higher-order functions, transducers etc are for right

alexlynham09:04:26

which to me, I'm sort of like, in the case where you wanted a black box - wouldn't you write a wrapper at the other end - at the io end - and then pass in a comp of several side-effecting transforms that includes the send action

danm09:04:27

What we try to avoid is something like

(->> (io/read-from-file)
     (map transform-fn-1)
     (map transform-fn-2)
     (map transform-fn-3)
     (map send-to-kafka))

danm09:04:01

In that case all the transforms can be put into a single fn

alexlynham09:04:59

@carr0t but right you could just (let [xfms (comp xfm1 xfm2 xfm3)] ... in their order (altho agree c'est not good) & then (map xfms)) inside the let

alexlynham09:04:09

which I think is what you're saying

alexlynham09:04:26

but again, I can't think of when ordering really matters

alexlynham09:04:33

maybe a filter at the end of a long set of xfms?

alexlynham09:04:58

at my old work we had a lot of pipelines end in (filter identity coll)

alexlynham09:04:10

(for finite streams)

danm09:04:43

Yeah, but I wouldn't have that comp within a let binding in the same fn. I'd have something else generate it and this fn just use that

alexlynham09:04:57

oh yeah, you'd probably pass it in

alexlynham09:04:53

(defn file->xfrmed-coll [file-path xfrm] ...) is prob how I'd do it in the wild... maybe?

alexlynham09:04:24

I've been reading up on connascence and basically -> and ->> seem to break the rules too, so far as I understand, bc we have scalar in first pos and coll in last pos

danm10:04:40

I tend to avoid comp and do threads most of the time, just because then I can map/pmap different bits as necessary/allowed, instead of having to do it all in either a map or a pmap

alexlynham10:04:15

have you done a side-by-side on perf? Also, isn't pmap eager?

alexlynham10:04:29

sorry if that's a dumb q or you mentioned it before

danm10:04:41

Side by side on perf of what? a thread vs a comp? We haven't, I have to admit. And pmap is 'semi-lazy' 😉 (See https://clojuredocs.org/clojure.core/pmap)

alexlynham10:04:13

so my benchmarking of JSON parsing & transformation was a single run through from IO, as lazy as poss, and comping the xfrms. Multiple map runs were slower but I've not comprehensively benchmarked yet bc I need to focus on implementation

alexlynham10:04:26

but memory overhead and input file size are the biggest concerns for me atm

danm10:04:30

But you can go map > pmap > map and know that order will be maintained in the second map even though the pmap parallelises stuff, so that's a pattern we quite often use. map for CPU intensive stuff, but pmap where 90% of the time you're waiting for external I/O

alexlynham10:04:46

need the program to have enough mem to run & to load the input files so it has to be lazy

otfrom10:04:29

using comp you can put transducers into pipeline or pipeline blocking to get parallelism and the speed ups from transducers

alexlynham11:04:30

^ that's kind of my thinking

Aleksander09:04:35

good morning everyone!

maleghast09:04:33

Morning everyone...

maleghast09:04:50

BTW, does anyone know when Clojure Conj is going to be, or is likely to be..?

maleghast10:04:17

Moreover, has StrangeLoop been announced yet..?

3Jane10:04:47

GraalVM is making rounds, which I haven’t seen before, and the description of Truffle I found somewhere is “Truffle is a framework for implementing languages using nothing more than a simple abstract syntax tree interpreter”

3Jane10:04:13

strong lispy flashbacks

3Jane10:04:19

I haven’t seen the internals though

bronsa10:04:32

the internals are not lispy at all :)

3Jane10:04:41

but they could be -.-

maleghast10:04:29

@lady3janepl - Graal is an odd choice of name for a VM... Well it's odd from my perspective, I am sure the occult symbolism was intended...

3Jane10:04:04

better than “one ring”, Tolkien estate could have sued them 😄

maleghast10:04:03

Does Clojure "play nice" in there..?

maleghast10:04:28

(I assume it would, but note with interest that Clojure is not mentioned on the homepage of the website, where Kotlin and Scala are)

3Jane10:04:34

(I’m aware of what graal is, but it’s long since leaked to normal culture - people call stuff “the holy grail of…” if it represents the pinnacle/ultimate realisation of whatever category. and Grails already existed, so they couldn’t do Grail :))

3Jane10:04:01

there’s a thread on Reddit with someone running a compiled Clojure image, apparently it works, but no further research

3Jane10:04:06

and yeah, I felt the lack of mention of Clojure as well, guess they just don’t consider it a major target

3Jane10:04:50

“ponies all the way down” sounds promising 😄

alexlynham10:04:48

if that's chris seaton's thing he was aiming (originally, dno about now) to try and support a few JVM langs out of the box, but I think at the time he was keen on jruby

3Jane10:04:00

It’s an Oracle research project

3Jane10:04:32

I dug around and found a presentation on integrating Ruby - seems like they did use jruby heavily 🙂

dominicm10:04:09

@maleghast I believe it's a reference to something to do with compilation techniques.

maleghast10:04:36

@dominicm - Yeah, from the looks of it, you can already use it to run Ruby, Python, R, C++ and Java side by side in the same environment, though some of the examples seem a bit contrived and it doesn't have the feel of a technology that's ready for primetime... The webpage name-checks Scala and Kotlin as well as Java, but omits Clojure, so I was just curious

dominicm10:04:06

Oh, no I lied

dominicm10:04:11

It's named after the "Holy Grail"

dominicm10:04:27

GraalVM is the Holy Grail of compilers

maleghast10:04:34

Yeah, exactly (hence my comment about the occult significance of the name)

dominicm10:04:46

Oh, I completely missed that

dominicm10:04:50

brain non-functional today

3Jane10:04:28

hump day hits hard 🙂

alexlynham10:04:17

yeah he is or was based in Manchester so he popped up at the user groups here

alexlynham10:04:37

I think he came to the fp group and was asked the clj question... can't for the life of me recall what the answer was lol

otfrom10:04:29

using comp you can put transducers into pipeline or pipeline blocking to get parallelism and the speed ups from transducers

mccraigmccraig11:04:40

so my cassandra data has gotten beyond any reasonable ability to dump/restore using the small-scale-but-easy routes (dump to CQL statements and load via cqlsh, or dump to transit and load via CQL) i have been using, and i'm wanting to start using sstableloader - it seems to be at least an order of magnitude faster than loading via CQL

mccraigmccraig11:04:22

does anyone know of any tools to help with moving (to s3 or to another cluster) sstables from snapshots, and of restoring sstable snapshots maybe to a different keyspace ?

mccraigmccraig11:04:39

opscenter is ok for lifesaving backup/restores to s3 for a keyspace, but you can't restore to a different keyspace and it seems to be version fragile so not much good for migration to a new cluster

otfrom11:04:34

Not sure there I'm afraid. Spark maybe?

mccraigmccraig12:04:31

it's mostly about moving snapshot sstables and renaming directories, and then invoking sstableloader... i found https://github.com/tbarbugli/cassandra_snapshotter which does the snapshot side, and can be used for catastrophic restores, but is missing an sstableloader based restore option

Sean Michael Murphy13:04:17

Good Afternoon, Sorry to bother you on this professional forum however I had a ClojureScript 6 Month Contract role paying up to £700 a day or £100,000P/A permanent role and wanted to see if this might be of interest to yourself? - Please get intouch if so and lets see what we can do!

thomas13:04:15

@sean854 welcome and jobs should go to #jobs and #remote-jobs if they also allow remote.

thomas13:04:39

and good luck with the search

thomas13:04:32

aahhh I see you already posted there 😇

Sean Michael Murphy14:04:14

Thanks thomas much appreciated