Clojurians Log v2

Some thoughts from my experience in 2012 as CTO when I switched the company from Ruby (on Rails) to Clojure for a backend project. • We had a backend API that would have to scale up dramatically due to new partnerships. This meant reconsideration of the architectural choices • I knew this backend domain very well and I knew what Ruby could and couldn't do well • The choice for a different Ruby framework seemed to justify also looking at other programming languages • I listened to developers in other candidate languages (e.g. Haskell, Erlang, Go and Clojure) • I read a book discussing and comparing these other languages https://pragprog.com/titles/btlang/seven-languages-in-seven-weeks/ and I was going through language specific books • After studying the theory, I started with independent isolated projects in Erlang and Clojure the languages that I felt most positive about. E.g. one of my first clojure projects was a one time scraper project. I think in these days that would have been a Babashka script • The code I wrote for the Clojure projects felt most natural to me and I decided to port our main API from Rails to Clojure. With the same db setup etc the performance was already much better. The code I wrote was definitely not idiomatic by today standards, but it worked! • I got many questions from investors about my choice for Clojure and during a due diligence I had a heated argument with one of their technical advisors, a .NET guy. It was a very frustrating experience, but the due dilligence had a positive outcome and we got the investment. • At the time, with our company we were heading towards in the "big data" direction. So I was looking for other companies in this space. At the time there were a few example pioneer companies in this space that gave me confidence it could be done (https://en.wikipedia.org/wiki/BackType and https://en.wikipedia.org/wiki/Prismatic_(app)). This was one of the arguments I used for my choice. • Another question was hiring of course. When we started I had no proof that it would be easy. But it became clear quickly that Clojure was attracting amazing talent and at the time there were more people wanting to work with Clojure then there were jobs. So hiring was actually not hard (at that time). We also learnt that people without Clojure background were able to pick it up quickly This was the summary of my experience choosing Clojure in the past. So in the light of a scientific project some of my questions would be: • Do you have experience in the field? If not, what do most people use for this? Maybe start here to gain experience first. • You have experience, but there a problems with the current language/tools. Why do you think Clojure would solve this? • Are there example companies solving similar problems using Clojure? • Are there other candidate languages besides Clojure? How do they compare? • Can you find isolated problems that can be solved with one of the languages? How does it feel? What works well? What doesn't? The answers to the questions will help you to make a convinced choice. If answered honestly, it will also be what other developers will see and be attracted to the language. Developers want to work with nice tools. If Clojure is a nice tool for a scientific project you will have less issues attracting talent. Hope this was useful.

Thank you @jeroenvandijk Very insightful thoughts.

If you haven't already, I would strongly recommend watching Thomas Clark's excellent talk from last year's Clojure/conj: https://www.youtube.com/watch?v=_D5d6Ls6pBw, which specifically covers Clojure's stack for scientific computing from the point of view of a research physicist.

I would say, as a solo practitioner who uses Clojure for data science work in my current job, my off-the-cuff answers to your questions might be the following: 1. Yes (but see below) 2. This is almost impossible to answer in the abstract, but I would say that you won't do yourself any favors by trying to make sure every single line of code is in Clojure. Trying to write your own solvers for PDEs or something similar in Clojure is not time or money well spent. Don't hesitate to use other languages for specific parts of the workflow - just remember that Clojure excels at the system level where it can tie data from multiple subcomponents together, even if that data is processed by a script written in a different language. 3. This will also be tricky, but I'd say you're going to have to invest in training and development for a scientific project anyway. A skilled developer may not have that much trouble adapting to Clojure if the project is interesting - just be prepared for that inevitable learning curve and be prepared to hire people who don't (yet) know Clojure as long as they're interested in learning. 4. This is so vague and big that I can't really give an answer to it. It depends! 5. See the talk I posted for a good overview. 6. The "data-first" nature of Clojure makes it a great fit for these types of things. Databases like Datomic are flexible enough to accommodate lots of heterogeneous types of data that might be challenging to normalize into a SQL model and it offers a powerful query syntax.

Thank you for introducing the video

Did any of your manager or CTO resist over using clojure?

@afoltzm 🙂

No. They cared more about my results than the language I used to produce them.