This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-04-18
Channels
- # architecture (14)
- # beginners (89)
- # cider (336)
- # cljsrn (2)
- # clojure (181)
- # clojure-berlin (1)
- # clojure-dusseldorf (3)
- # clojure-finland (4)
- # clojure-germany (5)
- # clojure-italy (18)
- # clojure-norway (10)
- # clojure-spec (9)
- # clojure-uk (94)
- # clojurescript (84)
- # cursive (3)
- # data-science (4)
- # datomic (82)
- # emacs (2)
- # events (4)
- # figwheel (1)
- # fulcro (6)
- # graphql (2)
- # hoplon (46)
- # instaparse (24)
- # jobs (9)
- # lein-figwheel (2)
- # luminus (18)
- # lumo (3)
- # mount (1)
- # off-topic (14)
- # onyx (17)
- # parinfer (22)
- # planck (1)
- # protorepl (1)
- # re-frame (50)
- # reagent (7)
- # ring-swagger (6)
- # rum (4)
- # shadow-cljs (94)
- # spacemacs (9)
- # specter (7)
- # tools-deps (2)
- # uncomplicate (4)
- # vim (33)
månmån
morning
ahoy hoy
so I've got a question
a dev who's mainly .net/java called me up on connascence of execution order in some code... and I was a bit like "uhm, imperative/functional language, it's a pipeline of transforms"
am I an idiot because I'm wrong or am I an idiot because I didn't understand the question?
he tends to be right about things so I've been mulling.
can you give an example of the dependency he was concerned about @alex.lynham?
it was a chain of lazy seq processes
so in pseudocode something like:
(->> (io/read-from-file)
(map transform-fn)
(map send-to-kafka))
kind of a hackneyed example, but that's roughly the shape of what I was doing
looks pretty sane to me. what was he suggesting would be a less complex/lower-connascence solution ?
to close over the transforms in a black box function and move it to a namespace closer to the interface with the external service
(I think)
Morning ☀️
How is everyone doing today?
that would lower comprehensibility, to my fp adapted eye anyways, because i'd have to go away and look up that blackbox fn... and would doing so not be just moving connascence around, rather than lowering overall system connascence ? the blackbox-fn would still have all the pipeline's type and order dependencies
although if the pipeline got to be much longer, or was re-used then i'd put some or all of it in a function
I mean, that pattern you've got there pretty much matches what we have in all our stuff. In shell there's an initial 'get data' function, and a final 'write data' function, and the middle function is a transform that is in our core namespace. And then that transform might be pretty complex but is entirely pure
where the code is isn't really as important as how you compose it... I think(?) I mean that's what higher-order functions, transducers etc are for right
which to me, I'm sort of like, in the case where you wanted a black box - wouldn't you write a wrapper at the other end - at the io end - and then pass in a comp
of several side-effecting transforms that includes the send
action
What we try to avoid is something like
(->> (io/read-from-file)
(map transform-fn-1)
(map transform-fn-2)
(map transform-fn-3)
(map send-to-kafka))
@carr0t but right you could just (let [xfms (comp xfm1 xfm2 xfm3)] ...
in their order (altho agree c'est not good) & then (map xfms))
inside the let
which I think is what you're saying
but again, I can't think of when ordering really matters
maybe a filter at the end of a long set of xfms?
at my old work we had a lot of pipelines end in (filter identity coll)
(for finite streams)
Yeah, but I wouldn't have that comp
within a let binding in the same fn. I'd have something else generate it and this fn just use that
oh yeah, you'd probably pass it in
(defn file->xfrmed-coll [file-path xfrm] ...)
is prob how I'd do it in the wild... maybe?
I've been reading up on connascence and basically ->
and ->>
seem to break the rules too, so far as I understand, bc we have scalar in first pos and coll in last pos
I tend to avoid comp
and do threads most of the time, just because then I can map/pmap different bits as necessary/allowed, instead of having to do it all in either a map or a pmap
have you done a side-by-side on perf? Also, isn't pmap
eager?
sorry if that's a dumb q or you mentioned it before
Side by side on perf of what? a thread vs a comp? We haven't, I have to admit. And pmap
is 'semi-lazy' 😉 (See https://clojuredocs.org/clojure.core/pmap)
so my benchmarking of JSON parsing & transformation was a single run through from IO, as lazy as poss, and comping the xfrms. Multiple map runs were slower but I've not comprehensively benchmarked yet bc I need to focus on implementation
but memory overhead and input file size are the biggest concerns for me atm
But you can go map > pmap > map and know that order will be maintained in the second map even though the pmap parallelises stuff, so that's a pattern we quite often use. map
for CPU intensive stuff, but pmap
where 90% of the time you're waiting for external I/O
need the program to have enough mem to run & to load the input files so it has to be lazy
interesting
using comp you can put transducers into pipeline or pipeline blocking to get parallelism and the speed ups from transducers
^ that's kind of my thinking
good morning everyone!
GraalVM is making rounds, which I haven’t seen before, and the description of Truffle I found somewhere is “Truffle is a framework for implementing languages using nothing more than a simple abstract syntax tree interpreter”
@lady3janepl - Graal is an odd choice of name for a VM... Well it's odd from my perspective, I am sure the occult symbolism was intended...
(I assume it would, but note with interest that Clojure is not mentioned on the homepage of the website, where Kotlin and Scala are)
(I’m aware of what graal is, but it’s long since leaked to normal culture - people call stuff “the holy grail of…” if it represents the pinnacle/ultimate realisation of whatever category. and Grails already existed, so they couldn’t do Grail :))
there’s a thread on Reddit with someone running a compiled Clojure image, apparently it works, but no further research
and yeah, I felt the lack of mention of Clojure as well, guess they just don’t consider it a major target
if that's chris seaton's thing he was aiming (originally, dno about now) to try and support a few JVM langs out of the box, but I think at the time he was keen on jruby
I dug around and found a presentation on integrating Ruby - seems like they did use jruby heavily 🙂
@maleghast I believe it's a reference to something to do with compilation techniques.
@dominicm - Yeah, from the looks of it, you can already use it to run Ruby, Python, R, C++ and Java side by side in the same environment, though some of the examples seem a bit contrived and it doesn't have the feel of a technology that's ready for primetime... The webpage name-checks Scala and Kotlin as well as Java, but omits Clojure, so I was just curious
yeah he is or was based in Manchester so he popped up at the user groups here
I think he came to the fp group and was asked the clj question... can't for the life of me recall what the answer was lol
using comp you can put transducers into pipeline or pipeline blocking to get parallelism and the speed ups from transducers
so my cassandra data has gotten beyond any reasonable ability to dump/restore using the small-scale-but-easy routes (dump to CQL statements and load via cqlsh, or dump to transit and load via CQL) i have been using, and i'm wanting to start using sstableloader
- it seems to be at least an order of magnitude faster than loading via CQL
does anyone know of any tools to help with moving (to s3 or to another cluster) sstables from snapshots, and of restoring sstable snapshots maybe to a different keyspace ?
opscenter is ok for lifesaving backup/restores to s3 for a keyspace, but you can't restore to a different keyspace and it seems to be version fragile so not much good for migration to a new cluster
it's mostly about moving snapshot sstables and renaming directories, and then invoking sstableloader... i found https://github.com/tbarbugli/cassandra_snapshotter which does the snapshot side, and can be used for catastrophic restores, but is missing an sstableloader based restore option
Good Afternoon, Sorry to bother you on this professional forum however I had a ClojureScript 6 Month Contract role paying up to £700 a day or £100,000P/A permanent role and wanted to see if this might be of interest to yourself? - Please get intouch if so and lets see what we can do!
@sean854 welcome and jobs should go to #jobs and #remote-jobs if they also allow remote.
Thanks thomas much appreciated