qui custodiet ipsos custodes

https://erlang.org/documentation/doc-4.9.1/doc/design_principles/sup_princ.html

(try
    (re-implement-erlangs-supervision-tree orchestrator-n)
  (catch e ItWasTooHardException
     (resume/logf "Cloud orchestrator %s experience! 
                   Familiar with distributed systems, especially with
                   complexity regarding %s" 
                   orchestrator-n e ))

That’s part of my CV algorithm, feel free to use it. 😄

SRE (Site Reliability Engineering) case study

cascading failure and unobservable bug something something

K8S itself: a case study in complexity

@U04V5V0V4 i do like the small set of verbs operating on an open collection of nouns thing though

I missed that poem :man-shrugging::skin-tone-2:

k8s operators let you use the same small set of verbs to manage new types of thing... we use them to uniformly manage kafka clusters, kafka topics, gnatsd clusters, elasticsearch clusters &c

Tbh I have been put off by the fact that helm charts exist ie a cottage industry in complexity management. But maybe it’s got simpler?

@U0524B4UW oh I thought you mean RESTful architecture (uniform verb is good for heterogenous integration across organisation boundary?)

though HATEOAS (and the larger theme of semantics web) seems to me to remain a pipedream even in 2023…

@U0524B4UW heard of operators (even played with some), but is there some place that clearly explain how it’s different from just a helm chart deployment? my impression after reading various doc is that the point you mentioned isn’t emphasised nearly enough

@U0525KG62 lol the IT tech books I read on monitoring with prometheus/grafana specifically has a section on that… forgotten what’s in there though :man-shrugging:

@U04V5V0V4 helm charts put me off too - i chose not to use any for our deploy because they added what felt like too much complexity in an already unfamiliar (to me) domain - but they do address an abstraction problem with K8S (also targeted by kustomize)

K8S is undoubtedly complex - but a lot of that complexity is a reflection of the underlying domain - AWS, GCP &c, and i suspect the abstraction capabilities will improve over time e.g. operators were introduced a few years after K8S became a thing, kustomize is only since 1.14

@U6L5FFL95 an operator can give you new k8s nouns, e.g. here is yaml for a kafka topic from our system (it's the strimzi operator that defines the new noun in this case): https://gist.github.com/mccraigmccraig/10ec06f56fe6e4a010b523dbaef1aab9 against which you can apply, get, describe, delete as with any other k8s resource... e.g. https://gist.githubusercontent.com/mccraigmccraig/4adbd5da9d59c756a859ea5bfe049a83/raw/27444eb2abe3d5e8c808a6c53928aefa0daa2fec/gistfile1.txt

want more partitions - change your topic yaml and apply - the strimzi operator converges your deployment with kafka specific operations

i chose kafka topics as an example because the yaml for them is embarrassingly simple - but the same idea applies to more complex things - here's our elasticsearch logging cluster definition : https://gist.github.com/mccraigmccraig/92168f36c8aead55ddae2a108d2c8f8c want more data nodes ? update the yaml ( https://gist.github.com/mccraigmccraig/92168f36c8aead55ddae2a108d2c8f8c#file-gistfile1-txt-L53 ) and apply

lemme think - seems like there’s something here, but I’m not sure how to put it into word…

…in the meantime, so is it that something like “perform a database backup” is also an example of a target to being “operationalized”?

@U6L5FFL95 yes - here's a cassandra operator which does just that : https://docs.k8ssandra.io/tasks/backup-restore/

(i haven't used it - cassandra operators were too new when i set up our cluster)

k8s is almost certainly overkill for a straightforward API+DB project - but if you throw in a few extra components - kafka and websockets (e.g. gnatsd) and batch jobs and a desire to avoid coupling yourself to a particular cloud then it makes a lot of sense

oh nevermind… nothing a quick web search can’t solve: https://eskala.io/blog/what-are-the-differences-between-helm-and-operators/

(I also like the stackoverflow ans with the tagline: “CRD + Controller = operator” (ok, maybe a bit more complex than that - operator also “encode and crystalize operational expertise”, whatever that means))

though I’d also say that operator brings to mind the idea of a long-lived (at an org level) service

the kind that you dedicate staff to babysit

yeah that definition ( CRD + Controller = operator ) makes sense to me... and when you look at what resources an operator creates, it's usually just a deployment or statefulset for the controller, so turtles all the way down

@U6L5FFL95 the operator defines new resource types (CRDs) and the controller wraps complex ops processes on those CRDs into the k8s verbs

oh i see… so you’re saying that the power of the CRD + controller combo increases once you begin to actually, err, “Program” in it

in a potentially recursive manner (not sure what the right word is here)

lol - not sure 🙂 - for me the power is that all manner of complex ops processes can be handled in a simple and uniform manner

the idea is actually pretty nice - it is when you implement it on a platform as sprawling and byzantine as kubernetes that makes it (ohno!)

but then, the container orchestration war is already in history textbook, it’s too late if anyone don’t like k8s for whatever reasons

ha, in true https://www.juliadonaldson.co.uk/books/picture-books/a-squash-and-a-squeeze/ style you should try AWS CloudFormation for a bit and then go back to k8s 🙂

😰

(though the children’s book look good)

i would go as far as 😱

haven’t tried that. though my friends over there find that terraform state can get stuck…

(not exactly terraform’s fault though… likely to be the specific cloud vendor provider’s)

it’s not fun I can tell

anyway, I think my main complain on operator is that their docs (for some of the most prominent ones) doesn’t do a good job of illustrating things with a self-contained, end-to-end example to highlight operator’s advantages

yeah @U6L5FFL95 i won't even try and argue with that - my view on k8s was not acquired from the docs - iirc i came across the https://strimzi.io/ operator for kafka before i ever tried k8s, and seeing what it could do perked my interest

although the current k8s explanation of operators is pretty good : https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

yup, so maybe rant a bit more (and off track too) - crossplane, now that’s the real 😱

I mean, as complexity of k8s approaches infinity, perhaps what ppl actually wants, in the limit, is their own self-hostable PaaS to get away from cloud vendor dependence

though it turns out that k8s itself is not a PaaS per se, but could be a substrate to build a PaaS on top of (the horror…)

I am at times puzzled by why it seems that in spite of all the advances over the year, something as “basic” as cloud/PaaS/IaaS etc remains essentially an unsolved problem once you go beyond the surface

ya thanks a lots @U0524B4UW - I came across this https://operatorframework.io/what/ a long time ago but don’t really know what’s going on for the tiniest bit… with your explanation I seem to begin to “get it” (by an epsilon :rolling_on_the_floor_laughing:)

@U6L5FFL95 my current company (sona - took over yapster) runs on k8s - they ported their app from azure to GCP in a day, with no (well, the story says "no", so perhaps "approximately no") changes - so it worked pretty well as a portable PaaS for them

though I have doubts on how often does someone need to build its own operator. Seems like the use case will be much broader if “I have my bespoke custom fullstack application that I want to automate operations on in an end-to-end manner” is a valid case for a custom operator

i think building an operator goes hand-in-hand with building an architectural component - it makes sense for e.g. datastax to build a cassandra operator, or confluence to build a kafka operator (or even AWS to build operators, since they are providing services at scale)... could also make sense for a company with a well-defined application to build their own operator for that application

but you don't need to rush in to building an operator ... you can go a long way with other operators and inbuild k8s nouns - i never considered building an operator for yapster, although now i'm thinking about it there are some ops tasks which require a bunch of kubectl commands in the right order, which is probably the smell indicating you should build an operator

ya that’s exactly what I’m worrying about - many ppl learn about things in a hands-on way, so if building an operator is culturally seem as “something only big-name org do”, it would makes it harder to reach for ordinary folks

even more so if there is a legitimate reason for that

maybe for something at this high a level of abstraction, it’s like what you said - on-ramping in an organic manner naturally is better

i.e. start with just using operators for well known components like DB and kafka (crucial to have clear doc), then gradually evolves towards having your own operator when the pattern emerges

(:require [time.greeting :as tg])

(def ugt ::tg/universal)