Clojurians Log v2

I recently met someone online that left development for Devops and claims that Devops is easier "for him".

In my experience the average DevOps problems are easier than the average Dev problem but the hard DevOps problems are far harder than the hardest Dev problems I’ve encountered; with ordinary software you can keep iterating, and there are fewer factors outside the control of your team. The number of times a DevOps issue was blocked by another company (cloud provider or other vendor) being slow or unskilled was staggering.

There are some very difficult DevOps problems like long feedback loops and resources discovery and impact. In the first case, you sometimes only face your infra issues in specific settings and with long deployment cycles and no local iteration, that makes debugging very hard and the impact radius of a mistake is much larger. In the second case, when you're working with multiple systems like on-prem and the cloud, having a clear inventory of your components and context. Because knowing where and when a component can be used is even more difficult than having a comprehensive catalog. Then, knowing the impact of a change, as well as the dependencies, with heterogeneous systems is very hard. You can face similar issues when developing software but the cost of an error is larger in DevOps imo.

You both seem to have more experience and potential war stories about DevOps than I've had to collect in the past 4-5 years. I can relate to your experiences, with the feeling that I've only scratched the surface of the possible nightmares. 🙂

Are we using the same definition of DevOps? When you're DevOps it means you do all the dev work and most of the ops work too. At least that's both what I'm familiar with the term and what my work has been.

Impressive projects, Aditya!! (Including the other one before with what you did for the product with 300M+ devices.) CloudFormation is simple to start small projects, but daunty to operate and to represent larger systems. Terraform is puzzling. I would have loved to have read your post (before it existed) but I wasn't given time to refactor much of our weird IaC codebase. Not functional, stateful... so true. Not imperative, but stateful. Weird! I would have loved to be able to try Pulumi (or some CDKs.)

Pulumi is interesting, had not seen it before. It seems similar to CDK, but supports multi-cloud. I really feel that's the easiest model. I think it's a bit like tools.build, once you reach some level of complexity, having the ability to have reuse and abstractions that a programming language offers is really nice. But for simple cases, it might be overkill. I will say though, having used CDK a lot, I've often wondered if I wouldn't prefer just using Clojure and the raw AWS APIs. The idea of comparing state in my experience is "meh". You always get in weird situations where it fails. Having code that would hit the APIs directly and operate more like SQL migrations, with up/down counterparts. I've always wondered if I'd enjoy it more.

Thanks for the kind appreciation, Daniel. When it comes to IaC, this is where I stand right now, apropos managing whole systems (services, DBs, queues, managed services, what have you): • 1 machine system architecture, vertically scaled to the limit... ◦ Put everything on one box, use jails / containers, and CPU pinning to isolate resources and reserve capacity for hungry software. ▪︎ Yes, postgres and zeromq and redis on the same box, talking over sockets to your Clojure monolith ◦ A second clone machine on hot-standby for high availability. ◦ Like, 64 cores and terabytes of memory will be more than enough for most hosted applications. ▪︎ A good OS can do a lot for us; we just need to be willing to learn its quirks and read endless manuals. • 1 to 10-ish machines ◦ Tiny shell scripts can manage everything (Bash, Babashka, Python, Ruby... whatever floats your boat) ◦ But, something has to force me to split the "vertical system" into a multi-machine one (traffic saturating network card, extreme variations in resource consumption, legal/organisational mandate etc.) ◦ Distributed systems are less reliable and more error prone over time, than shared-memory-space systems. • 10s to 1,000-ish machines ◦ A small competent team + Ansible (maybe Pulumi... it's been on my radar for a while, but i haven't evaluated it yet) ◦ Conway's law has bested us and we are mirroring organisational architecture in our system architecture. • 1,000+ machines ◦ There is a truly complex problem, whether organisational or software workloads (whether demanding, or heterogenous) ◦ This rarely is a "day zero" problem ... instead, life is not bad running everything with duct tape and spit and meetings, until one day we wake up to a nightmare and it is real ◦ "Unified control plane" gets tossed around, and kube is inevitable. ◦ If I have the budget and say-so, I'd probably evaluate and use #rama ... anything that helps collapse complexity will be a huge win at this point

• If it can be said that Dev positions handle the complexity of integrating the software with the domain, • It can surely be said that DevOps handle the complexity of integrating the software with the infra and other software and teams. Just as much as some (like Charity Majors) encourage people to alternate between team lead positions and individual contributor positions, I think there might be something to be said for alternating between Dev and DevOps positions. After some years in DevOps positions, I think it's much more understood in general by colleagues how gruesome Dev projects can be than it's understood how gruesome DevOps projects can be. I'm preparing to go back to Dev positions. A CTO once told me, something like 15 years ago, that there's nothing harder than integrating software with software. (One could argue that this can happen as much, or more, in Dev than in DevOps. In fact, he claimed this while he was hard at work integrating two different frameworks together.) So, in other words, I developed the feeling that people are more likely: • to understand and appreciate how long Dev projects can be, and • to not understand and not show support for how long DevOps projects can be. It's probably easier to imagine oneself • developing the domain of the software (visualizing what will need to be done, what kinds of issues will have to be solved), and how long it can be, • than to anticipate the many dozens of OS/packages/scripting/sdk/infra/process/auth/networking challenges that will need to be solved, and that will not necessarily be visible in the final set of parts of the solution that an external observer will look at when forming their opinion about the project's complexity.

You're right it's the definition in the purest sense, and oftentimes some will display more Ops expertise and take the most daunting DevOps tasks for the profit of the team, and the most Dev-oriented experts will do the equivalent. (Everyone playing to their strengths, and hopefully, everyone else learning from them.) In my case, in each job where I was officially doing DevOps, I was the single DevOps person in a team of regular Devs, so we couldn't live by the definition of DevOps as a team. I wasn't ever even expected to do regular Dev work in those teams, as if I didn't even know what this kind of work is. You could say it's not the healthiest of environments, DevOps-wise. In some way, it's a token DevOps culture, a bit like most places used to say they were doing Agile or Scrum. But still, it's far better than the traditional Dev vs IT divide. So in this discussion, I think we all focused on the extremes of Dev vs the Ops part of DevOps.

I see, it does seem a bit performative, but at least you're right, having you as part of the same team, sitting and engaging with the devs on a regular basis is still better than a separate dedicated Ops team. I do full fledged DevOps, where the whole team fully owns and is accountable for the development work and the operations. There are dedicated infra teams for more fundamental infrastructure, but otherwise we're fully responsible for both and go on-call for all operational issues.

It's the nature of complexity: • dev: primarily domain complexity (of the thing they are writing software to solve for; the right data model, abstraction, SQL query plan, language use, runtime performance etc.) • devops: primarily organisational complexity (of the specific company they are at - teams, software systems, compute, traffic demands, SLAs, costs etc.) dev (narrow, but deep slice of one delicious food item) < > ops (wide, but so many pieces - lunch buffet at fancy hotel) For example, I wrote about stuff I had to consider and account for and build for, in devops / "production engineering" jobs I've had: https://www.evalapply.org/posts/systems-approach-to-infrastructure-as-code/index.html#main

I don't have terraform experience, but I used AWS CloudFormation, which is terrible. But when they released CDK, that is a lot better, as it offers higher level constructs and you can use it in your own programming language. I heard terraform has a CDK of sort now too.