Fork me on GitHub
#clojure-uk
<
2020-01-14
>
dharrigan05:01:12

Sometimes I wish I wasn't such an early bird

seancorfield05:01:49

Hey, it means you get to chat with me before I go to bed in California @dharrigan! 🙂

dharrigan05:01:30

Well, only thing to report is that I'm looking to deploy clojure code into our live environment for the first time EVAR today

sheepy 12
bananadance 8
8
dharrigan05:01:07

I'm hopeful that in a few years time, I'll look back at that codebase and think - ehhh? what was I thinking? Since I hope my skills/experience will have improved by then 🙂

seancorfield05:01:19

Very cool milestone!

seancorfield05:01:39

I no longer remember our very first production deploy that included Clojure... 2011... Spring... Clojure 1.3 Alpha 7 or 8, embedded into our legacy apps. But we had a lot of other stuff going on that meant Clojure was sort of the least of our worries back then.

seancorfield05:01:42

These days, our production deploys are mostly "meh" which is a good thing 🙂

dharrigan05:01:51

The MD is interested to see how this goes. I'm having to maintain a functionally equivalent Kotlin version, you know, just in case....

dharrigan05:01:18

I'm hopeful I won't need it - as my tests show that my Clojure version produces the same results as the Kotlin version

dharrigan05:01:34

Plus, there's a bus factor of 1 here

dharrigan05:01:44

So until that changes (which it will!) then I have to do that.

dharrigan05:01:13

I see that Rich is coming to Lawndawn this year.

seancorfield05:01:38

Oh cool! Is that for the PL conf?

dharrigan05:01:51

I guess it would be far too much (imposing) for him to give a small 30 minute talk to London Clojurists whilst here?

dharrigan05:01:02

You know, to inspire the community

seancorfield05:01:28

It never hurts to ask 🙂

seancorfield05:01:55

I'm looking forward to the transcript/video of his History of Clojure talk from that. I know he's been researching it heavily for a while -- Alex has been reaching out to folks to confirm historical details etc.

dharrigan06:01:28

I've reached out to Cognitect to see if there might even be a possibility of a small meetup chat from Rich

🤞 4
👍 4
dharrigan06:01:44

If there was one thing you could say to your younger self, as you embarked upon writing production Clojure code, what would that be? Any particular style, methodology, functions to really learn?

seancorfield06:01:38

Hell yeah: avoid global data! Pass all that shit through the call chain instead!

seancorfield06:01:03

We figured, back in 2011, we only have a fixed set of data sources so we'll just stick them in atoms in and initialize them at startup. That was a terrible idea.

seancorfield06:01:29

We built huge call chains on top of that where the data sources are implicit, and only hardwired into the bottom layer.

seancorfield06:01:16

When we started to want to switch code from the master to the slave, or to a different DB account so we could leverage different access permissions, we got into a world of hurt.

seancorfield06:01:53

And we did it with our caching layer too -- but that wasn't so painful: we have only ever needed one set of caching logic and we built in multiple named caches from the get-go so... yay, us.

dharrigan06:01:06

So, avoid global state

seancorfield06:01:40

I've spent a lot of the last several weeks refactoring code to remove those globals. I'm still only partially done.

dharrigan06:01:01

In my codebase, I have one atom, that's when I define the datasource. I keep it in it's own namespace txi.higgins.db and all SQL queries etc., all go through that namespace.

seancorfield06:01:40

Fine for MVP launch. Won't scale. So plan to refactor away from that pattern before your code gets any more complex.

dharrigan06:01:44

so, the callees never see the datasource.

dharrigan06:01:11

Sure, I'll look to refactor definitely - it's as much a learning experience for me as getting a MVP out to prove Clojure can be used at work 🙂

dharrigan06:01:49

(and I'm still very much a newbie...)

seancorfield06:01:40

Funnily enough, Clojure has given us a "magic power" that no one expected: the ability to apply critical patches to live, running, production processes with zero downtime 🙂 Via a Socket REPL into production. Not something we'd really planned on -- and not something we do very often.

seancorfield06:01:01

Most of our processes are clustered so a rolling, automated redeploy is our normal procedure.

dharrigan06:01:11

Right, that was in mind this morning (strange things when one wakes up at 3:30am)

dharrigan06:01:48

Does the socket repl need to be running when the application is launched? It's not like you can "attach" to already running Clojure application magically?

seancorfield06:01:55

But we have one app that is used internally and downtime is a major issue during work hours -- so being able to patch that via a REPL is amazingly convenient.

seancorfield06:01:49

Well, you actually can attach a REPL to a process that was started without one -- but that's a security problem and you should have it disabled (Bing for "jvm attach security").

dharrigan06:01:54

so, having a switch (say environmental, or runtime configuration) that would create a listening socket repl on the server application (if appropriate) is the answer - i.e., only enable listening if the computer says yes.

seancorfield06:01:58

So we have a JVM options file that is picked up by our daemon script that starts/stops services and we add a Socket Server option to that as needed. Some processes have that enabled all the time, others only when we need debugging (and we're happy to restart a process).

dharrigan06:01:16

right right, same approach 🙂

seancorfield06:01:23

It's literally a -D option on the JVM.

dharrigan06:01:42

Why choose a socket repl over a nrepl?

dharrigan06:01:49

any advantages?

seancorfield06:01:31

Socket REPL is built into Clojure -- no dependencies. You can start a Socket REPL in any Clojure process (or Java app that includes Clojure as a dependency!) just by adding a JVM option.

seancorfield06:01:47

And I sort of despise nREPL as a protocol 🙂

seancorfield06:01:13

We used to start nREPL with middleware in some of our processes but I always hated having that as a dependency.

seancorfield06:01:53

I was super happy when we dumped all of that and switched to a basic Socket REPL -- for both dev and production.

dharrigan06:01:49

I'm going to experiment with that today

dharrigan06:01:14

i.e., run my app in a docker container, expose the port, enable the socket repl (via a feature toggle on the app startup) and see what I can do 🙂

seancorfield06:01:24

I use Atom/Chlorine so a Socket REPL is all I ever need. My team mate uses Emacs and, I think, inf-clojure mode or something similar so he also avoids nREPL.

dharrigan06:01:38

How about pREPL?

dharrigan06:01:43

that came with 1.10

seancorfield06:01:51

That can be started with a JVM option too.

dharrigan06:01:00

I use that with my editor (nvim + Conjure)

seancorfield06:01:06

Basically it's a Socket REPL with a different listener.

dharrigan06:01:36

cool - eager to try this out

dharrigan06:01:19

so, say you want to redefine a function on running code, you in-ns '.... to the namespace and redefine the function (or def...)?

seancorfield06:01:51

Yeah... I mean, Chlorine does that behind the scenes, so I can just put my cursor into an updated version of the code in that file and ctrl-; B and it evals the top-level block (form) into the Socket REPL which, yeah, does the in-ns and the defn

seancorfield06:01:21

And I can use RCFs containing code to explore/debug things in a production context.

dharrigan06:01:26

Conjure does the same - you can eval a buffer etc..

seancorfield06:01:28

For me, production is identical to (local) development in that respect -- and so is our QA/showcase server since it also runs Socket REPLs.

dharrigan06:01:40

Yes, that makes sense, it's all network based anyway

dharrigan06:01:06

so no difference between your editor and the clojure code on your machine, and your editor and the clojure code 1,000 miles away in some data centre.

seancorfield06:01:10

The only difference is that locally I run my REPL in a context that has the add-lib branch of t.d.a. so I can load new dependencies on-the-fly.

dharrigan06:01:14

(maybe a bit of latency...) 🙂

dharrigan06:01:31

I hope Alex considers adding in add-lib

dharrigan06:01:36

it's pretty handy

seancorfield06:01:02

But once add-lib lands in a non-alpha context, I'll probably run that in production too... just in case I ever need to make a code change that requires a new dependency be added.

seancorfield06:01:32

(but I'd prefer a rolling deploy via our automated system with an updated uberjar!)

seancorfield06:01:17

Given that we can have a change committed to git be deployed to production in just a few minutes already...

dharrigan06:01:39

so, if you make a change to production code, and it works, you then commit that code into your repo and shortly afterwards your application in production is updated with the fixed code (officially)

seancorfield06:01:00

For all but one app in production, yes.

dharrigan06:01:14

I'm going to demonstrate that on my Friday Sessions

dharrigan06:01:18

where I show some Clojure stuff

seancorfield06:01:30

We have one legacy app that is not all Clojure and that still requires a manual deploy process 😐

dharrigan06:01:43

Ooops, the kiddo is stirring and calling out for me

dharrigan06:01:51

Time to attend to fatherly duties!

seancorfield06:01:59

I'm about to head to bed anyway...

dharrigan06:01:06

thanks sean, a doppo!

seancorfield06:01:12

(once I've finished my beer)

Eamonn Sullivan09:01:52

So, I have a very small chance of introducing Clojure/ClojureScript on our team, for whitelisting. We whitelist people and services (e.g., allowing one service to call another) using properties of SSL certs (emailAddress, OU). Right now we're manually writing this data structure in JSON, but it's too easy to make a mistake. Was thinking of suggesting a small bit of Clojure, using Spec to validate the relationship between all of the moving parts (access groups, consumers, etc.). We'll probably end up doing this in Node or Scala, but spec seems like a big advantage. Another possible one: can you use something like cljc to generate both a JVM library and a Node library from the same code?

mccraigmccraig09:01:20

you can certainly generate both a JVM lib and a node lib from the same code

mccraigmccraig09:01:36

shadow-cljs makes it very easy to target different javascript targets - https://shadow-cljs.github.io/docs/UsersGuide.html#_build_target

mccraigmccraig09:01:19

there are of course environment differences you may need to be conscious of when targeting both jvm and js - e.g. single thread on js

Eamonn Sullivan09:01:16

I don't think it needs to be complicated -- it's just a big, nested map, where you have to validate that the whitelisted entries or the service you are whitelisting have been defined in another part of the big, nested map. Or that you have no orphan nodes (access groups that aren't used anywhere, or services defined that are then not whitelisted for anything). Stuff like that.

mccraigmccraig09:01:18

sounds like something which could straightforwardly be written to target both jvm and js... i've no idea how good spec is at validating the sort of distant constraints you are wanting though

Rachel Westmacott09:01:36

might it be cheaper to just leave it in JSON and write a script for your CI pipeline (possibly in Clojure) to validate it?

Eamonn Sullivan09:01:39

Yes, we still want JSON output. At a minimum, it's human readable and in an emergency can be edited manually. The whitelist is just a JSON file that's read by many (eventually all) our services from an S3 bucket and cached for a short period. It's a simple system, but the trouble is it's easy to make a mistake. Validating it as JSON is the simplest problem. Validating that the JSON makes sense is a bit harder.

dharrigan11:01:36

Following on from the conversation I had with Sean this morning (vvvvvery early this morning). I tried launching my Clojure process with the JVM arg running a socket repl listening on port 55555. Anyone had experience of doing this within a docker container? Even although I'm mapping the ports, when I telnet (to kick the tyres) to the local port, I don't get a connection

yogidevbear11:01:06

Are you able to test it locally without it being inside a docker container?

Sam H11:01:38

Also what JVM options and Docker file settings do you currently have?

mccraigmccraig11:01:59

@dharrigan have you done a kubectl execto login to your container and check if the port is open directly ?

dharrigan11:01:40

Hi - yes works locally (it's how I connect for my editor), JVM options shows the -D.... setting being applied (same as local setup) and not using kubectl 🙂

dharrigan11:01:01

we are old sckool docker swarm

yogidevbear11:01:24

I'm not familar with docker swarm, but can you use something like docker ps and then docker exec ... (Need to double check the syntax for that)

dharrigan11:01:57

yup, port all showing as mapped correctly

👍 4
dharrigan11:01:02

I'l dig deeper

dharrigan11:01:35

address - host or address, defaults to loopback

dharrigan11:01:45

I think I have to tell it to listen to all interfaces

mccraigmccraig11:01:11

that would do it

dharrigan11:01:05

Well that sucks

dharrigan11:01:07

txi-higgins_1  | Exception in thread "main" java.lang.NumberFormatException: Invalid number: 0.0.0.0
txi-higgins_1  | 	at clojure.lang.EdnReader.readNumber(EdnReader.java:224)

dharrigan12:01:32

Sorted - has to be wrapped in quotes

dharrigan12:01:38

now it works, I can connect

👏 4
yogidevbear12:01:44

Now you can write up a blog post about it 😉

dharrigan12:01:17

it works

😂 4
dharrigan12:01:25

I can modify running "production" dockerised code

dharrigan12:01:40

The power! the power!

dharrigan12:01:55

I'm terrible at writing blogs

yogidevbear12:01:17

Practice makes perfect 😉

dharrigan12:01:25

You're making me feel like I should do something 🙂

yogidevbear12:01:49

Good, it's means I'm doing something right 😇

dharrigan12:01:29

Would you know of a clojure blogging sw?

yogidevbear12:01:09

Yeah, I looked at Cryogen a while back. It looked good

Samuel11:01:55

I've officially reached the end of my tether with windows / c# / powershell

🐧 8
clj 8
❤️ 8
8
8
3Jane13:01:40

o/ Do any of you guys know of companies with open positions for a QA engineer? An excellent QA I've previously worked with is looking for immediate employment in London.

dharrigan13:01:42

I will ask my boss

dharrigan13:01:00

Would the person be willing to travel to Epsom?

dharrigan13:01:10

it's about 40 mins from Waterloo, going out

3Jane13:01:02

Thanks! I'll ask about the location. It's less of a question of willingness (which there is) and more of school/childcare hours.

Eamonn Sullivan14:01:34

The potential opening for Clojure in my team (described above) came to naught. We'll do something with Python instead. I'd love to try to interest some people in our team to try Clojure, because I think they would like it, but there's an immediate, knee-jerk reaction to lisp syntax that I can't seem to overcome. Does anyone have any success stories about getting Clojure adopted by a team? What kind of project did you try first? We're a Scala (back)/React (front) team and I think that might have something to do with it -- adopting both was a big deal and took quite a long time. Clojure, on the other hand, you could learn 90% of it in a day or two and the Scala developers (we write it purely functionally) would understand most of the other 10%.

dharrigan14:01:10

We are mostly a Kotlin/Python/Javascript shop

dharrigan14:01:58

In terms of project, the one I'm getting into our production is a very isolated, read from kafka, write to database application

dharrigan14:01:23

I also maintained a functionally equivalant kotlin version - just in case

dharrigan14:01:34

but our boss has given me the go-ahead to see how it goes 🙂

dharrigan14:01:50

and he's also open to hiring a programmer 50/50 split between kotlin/clojure

dharrigan14:01:06

also, I did a few (still do) friday sessions where I show clojure stuff

dharrigan14:01:17

so, I just kept on-and-on-and-on-and-on about it 🙂

dharrigan14:01:48

Like Mrs Doyle 🙂

dharrigan14:01:15

I also wrote up use-cases where it fits into what we do (we process large amounts of data per day)

dharrigan14:01:02

and hyping on about immutability, easy concurrency/parallelism etc.. basically the selling points of clojure

Eamonn Sullivan14:01:46

One of the issues for us is that the gap between Scala and Clojure is a bit smaller -- we already do functional, immutable data and have quite high-volume concurrent services. I'm working on a 10%-time project (we can work on whatever we want one day every two weeks), but it's quite a big project and it'll take me months. So, I'm happy (I get to work on Clojure), but I do think Clojure could solve some of the problems we're having -- like on-boarding new developers into Scala. It's a real bear of a language.

Ben Hammond14:01:40

isn't the clojure/python Bridge very much alive and kicking? Karin Meier keeps tweeting about it

dharrigan14:01:49

My own opinion of course, but I think Scala is sorta spining atm until Dotty comes out

dharrigan14:01:13

I also find that scala code can be very hard to read

dharrigan14:01:48

I heard that the python bridge is awesome

Ben Hammond14:01:17

does it create a bit of wiggle room on those python projects?

dharrigan14:01:32

It's a good point,

Ben Hammond14:01:56

Those Scala/Clojure races seem to come down to culture, you can write impenetable spaghetti code in both languages but it seems more acceptable to the Scala community to do so

Ben Hammond14:01:44

dunno, is that unfair?

Eamonn Sullivan14:01:30

You can write spaghetti in any language, but I don't think that's the case here. We even go out of our way to write Javascript using a functional style, which makes this fertile ground for a much simpler functional language. For us, I think it would fit in quite well for projects just above what we use Python for (mostly scripts) and around where we do Node.JS servlets or Scala microservices.

mccraigmccraig14:01:30

if you've got people getting hung up on syntax that's a hard problem - i've encountered that before and it's not generally a rational objection, which makes it very hard to counter with a rational argument

Rachel Westmacott16:01:09

I liken it to judging a book by its cover.

dharrigan14:01:30

You know what, I've seen kotlin code wrote very badly

Eamonn Sullivan14:01:36

yeah, it's possible I've actually poisoned the waters a bit too -- I'm the only Emacs user amid a sea of Intellij/VS Code users (so strike one), have a greying beard (strike two) and have showed people various features I've added to Emacs over the decades (strike three). Lisp? Yeah, that's something Santa Claus uses...

😄 4
Rachel Westmacott15:01:28

they might find cursive an easier sell

Eamonn Sullivan16:01:25

I'm not suggesting for a minute that they use emacs -- it's just that's the only time they've ever seen lisp code. In a 35-year-old editor.

Rachel Westmacott16:01:08

maybe if you show them clojure in intellij it would look less alien

yogidevbear16:01:13

There's also Calva for VSCode 👍

seancorfield17:01:55

@eamonn.sullivan Yeah, I think the historical bias in favor of Emacs within the Clojure community is a double-edged sword. It has great Clojure support and it's where a lot of effort is spent by Clojurians on improving that support, but it does act as a barrier to entry for non-Emacs users wanting to learn Clojure, especially given most beginners are recommended to read Brave and True which starts off by walking folks through Emacs installation and setup...

dharrigan14:01:54

if it's just syntax, you're on a loosing battle anyway

dharrigan14:01:41

Santa Claus is efficient (he sees all the good little boys and girls - billions of them each year), he's parallel and concurrent - he's always predictable and reliable and he's good fun!

dharrigan14:01:55

I would be proud to equate clojure to santa claus 🙂

dharrigan14:01:32

oh, and all the world (mostly!!!) loves him 🙂

Conor14:01:09

Apart from in NL where he's a bit racist

Ben Hammond15:01:49

(but they still love him)

thomas15:01:20

that is not Santa Claus, but Sint Nicholas. @conor.p.farrell

thomas15:01:24

ooh and a 19 day difference.

rickmoynihan15:01:39

@eamonn.sullivan: > Does anyone have any success stories about getting Clojure adopted by a team? Sure. About 6 years ago I joined a startup with an existing ruby code base; and convinced the CTO to do the heavy lifting on the JVM (mainly because the best libraries for our eco-system existed in java), and because ruby was slow for handling large quantities of data, and that farming out jobs to remote external services was creating ops problems we didn’t need at our scale. i.e. we were using job queues because we wanted concurrency which ruby is poor at, not because we needed distributed computing & parallelism. Also he knew I had a lot of clojure experience anyway and had used it in many previous projects, and he was a geek and was convinced by rich hickey’s arguments etc.. fp, lisp, immutability etc. So we wrote some backend stuff in clojure; which worked really well and is still operational and under development today. And have written gradually more and more apps, libraries, tools and services in it. We’re now using it for almost everything, and have about 10 people coding clojure full time. Legacy ruby app is now being rewritten and replaced. We use clojurescript too for most but not all front end work. Regarding the syntax objections; syntax I’ve usually successfully argued with people that syntax is much less important than semantics and simplicity. Once they agree to that; then you explain the semantics of clojure syntax (takes about 10 minutes for most of it (clojure 1.0 syntax at least - the syntax has grown a bit more in recent years; but still fundamentally the same). I’ve never encountered anyone who hated the syntax after having such a discussion; and also being shown the capabilities it enables, e.g. paredit/smartparens/macros etc.

Samuel16:01:22

Thanks for sharing, I'm evangelising all I can, exploring use-cases and developing PoCs - no bites yet, but my fingers will remain crossed

yogidevbear16:01:53

So much this 💯 : > syntax is much less important than semantics and simplicity

mccraigmccraig21:01:32

completely true, but i do know apparently smart people who irrationally reject the parens

Wes Hall16:01:48

@eamonn.sullivan Macros are, in my experience, the best way to get people to realise that clojure (or more widely, lisp), syntax is not just perfectly fine, but the absolute best way to do things. Once people grok the whole, "code is just data, and data is malleable", they fall in love with the syntax quick-sharp. Depending on the organisation culture, you might want to consider putting a little internal, "fans of clojure" group together, or propose some kind of "show and tell" thing. People generally respond this this very well, particularly when you end it with a github link that they can peruse and play with in their own time.

Eamonn Sullivan16:01:37

I mentioned macros to someone recently and they said they know a Clojure developer who has only ever written two macros. And he's regretted one of them. In Elisp, I've created several macros and liked each one (well, most of them). I guess it depends on your view of "dangerous" features in a language (like pointers in C/C++). You could avoid using them entirely, but you wouldn't really be using the language as it was designed to be used.

Wes Hall19:01:22

@eamonn.sullivan Absolutely right, you should write macros rarely. I too have not written that many. It's not entirely the point though. Macros make the beauty of the syntax clear even if you never actually use them. It's a bit subtle, but I think it is true.

rickmoynihan11:01:52

I’ve probably written more macros than most. The standard advice is true, you should certainly avoid writing macros if you can; though very few of the non-toy macros I’ve written I’ve regretted. I’d say they’ve all been very useful. Whilst I 💯 agree with the advantages of using data over macros; the problem with that approach is that you end up writing an interpreter rather than leveraging the existing compiled language you already have. Again there are some huge advantages in doing this; but it’s often a lot less work to just write the macro with the limitations it has. I’m currently looking at porting a datalog/datascript like query language I wrote from macros to a data representation, and it’s a lot more work than the version 1 I wrote that used macros. If I didn’t have macros I genuinely don’t think I would have been able to write the bulk of it in a day; it would’ve taken a week or more.

Wes Hall16:01:54

I do find, much like with any other "product", it is the peripherals that "sell" it. Working in the REPL for example. I don't know any other language that does this even half as well as Clojure does, and a big reason for that is the syntax, or rather the relative lack of it.

rhinocratic18:01:21

Morning. @eamonn.sullivan I can't claim to have had much success in getting Clojure adopted, though not for want of trying - which probably says more about my persuasiveness as a salesman than about any putative shortcoming of the language. I went so far as to found a functional programming "guild" (not Clojure-specific) at one employer, in the hope of winning converts (which I had imagined might be an easier sell in a largely Java/JavaScript house). The reasons for failure seemed to be a mix of those frequently mentioned: • Parentheses ("Man, that just makes my eyes bleed!") • Unfamiliarity (people saw value in the features, but wanted equivalents in their comfort zone) • Fear of dynamic typing • Unwillingness to invest in what is perceived as a "niche" skill Even macros didn't seem to persuade anyone ("I can't think why I would need to do that"). The time that I felt closest to persuading someone was when a colleague saw me using Figwheel with a browser-connected REPL - so I'm inclined to concur with @wesley.hall that tooling might be the best route. I currently use Atom with Chlorine for Clojure development (having followed @seancorfield’s excellent set-up guide), and I can imagine that winning people over, too. Other than that, I'd guess it's an easier job with a greenfield project or a rewrite from an ecosystem whose shortcomings have become apparent. :face_with_rolling_eyes:

seancorfield19:01:09

Aye, I think a good REPL-driven workflow, showing that really tight feedback loop, and showing how you actually develop "in" the REPL, is about the most persuasive thing I've seen so far in terms of making folks go "Wow, that's cool! And that's Clojure, right?" -- but even then your point #2 is a common comeback "How do I do that in <language X>?" and they often don't believe that you can't...

seancorfield19:01:09

When I brought Clojure into World Singles Networks, I cross-trained two CFML developers (yes, really) and they mostly loved Clojure but felt the learning curve was very steep on the switch from imperative/OOP to FP. After a few years one left to go back to CFML at another company (and still does CFML as far as I know) and the other left to take some family time and then went to work for another Clojure shop. So I'd call that sort of 50% success 🙂

seancorfield19:01:35

Most everywhere I've worked I've been able to bring in some sort of new tech, even if it hasn't always been a new language, but I've been in the industry about 35 years at this point and I've been some variant of "Senior Software Architect" for the last 20 of those so companies sort of expect me to move the needle in terms of tech...

mccraigmccraig21:01:26

programming also advances one funeral at a time

⚰️ 8