Fork me on GitHub
#off-topic
<
2020-05-18
>
Empperi12:05:54

I have a question which is very much related to Clojure but since it is is fundamentally not strictly Clojure specific I decided to ask it here. I am currently trying to create development tooling for kubernetes based system with several pieces. Not everything is currently in kubernetes but some are still running in ECS but this would be the first step in putting those last few pieces into kubernetes world. This system works just fine in production but the development workflow is far from perfect. So, I want to enable the team to start the whole system locally easily. For this developers would use Docker Desktop with it's Kubernetes support or local minikube. All is nice and fine. This would enable us to run full system tests locally and so on which is nice. There's several tools available which make this kind of stuff easier to do such as Skaffold, Garden etc. However things get hairy when I would want to do some actual development work in such a way that rest of the system has been brought up by this tool but not the one service I'm currently developing. The way Skaffold and friends solve this is that they listen for code changes and trigger builds+deploy flow to the local kubernetes so that you get your changes there running quickly. However, this concept breaks completely when either the builds take quite long or if your fundamental workflow is just different to that (hello Clojure and REPL). So, I'm trying to find a solution which would allow me to control how stuff gets pushed into that local kubernetes. Meaning that I could say: "start everything in kube except service X". Even better if I could apply different configurations easily for these services so that they could possibly see the locally running service instance being developed. In our use case however we are transitioning towards queue based comms between services which would make this kind of stuff less relevant since all we would have to do is expose that queue (Kafka in our case) outside the local cube for localhost which is relatively simple to do. Then it doesn't matter where that service is when it is pushing it's messages into the queue. I have just discovered Skaffold and Garden and been looking into those for one working day now but it looks like none of the tools really support this use case. So my fellow clojurians, how have you solved this problem in your day to day work with kubernetes? :edit: Oh right, and we already have a service which needs kubernetes (it starts new pods dynamically) so forgetting kubernetes and using docker-compose (which does support this use case out-of-the-box) is kinda not an option

Darin Douglass13:05:10

heya. at our work we're actually building this exact thing at the moment. we have some plans on open-sourcing it when it's ready. i don't really have a timeline on it (soon™), but almost all of our ecosystem is deployable with the library.

Darin Douglass13:05:31

it is: • clojure code • runnable with babashka • defines the kube resources in terms of EDN (or YAML or JSON) data • a kube pre-processor (i.e has ideas of enviroment, metadata, references, etc)

Empperi17:05:51

Please tell me when it is finished! I'm very much interested

wombawomba20:05:39

I built something ike this

wombawomba21:05:59

Basically: • We use a monorepo (makes this stuff a lot easier). • I use a Makefile with targets for every common dev task for every service (`run-$service`, repl-$service, test-$service, etc), as well as targets for all databases (`init-postgres`, wipe-postgres, init-kafka, wipe-kafka, etc) and libraries (`install-$lib`, install-dependencies-for, etc.). Some targets have options (e.g. BUILT=true/false for running a service using a pre-built Docker image vs building it from code; DETACH=true/false for running in the background or foreground). This makes it easy to run multiple things at once (for instance, DETACH=true BUILT=true make init-postgres init-kafka run-foo run-bar). Plus you get auto-completion in most shells. • Each service has a Helm chart with the necessary variables for all use-cases (running using a built service Docker image vs running from code; running a REPL with tty and stdin vs running the service ‘normally’ with a livenessProbe and whatnot vs performing some non-interactive task like cleaning/linting/testing). There’s also both a values.yaml (for production usage) and values-dev.yaml (for dev usage) for each service. In values.yaml I specify a pre-built service image (from the CI), and in values-dev.yaml I specify an image that has whatever dev tooling is needed for the service (for Clojure projects, this means clojure:openjdk-11-lein). • For the ‘running from code’ scenario, I just mount the relevant code directory as a local volume to /app in the container (Note: local volumes not supported with all ‘local k8s’ setups — I mostly use Virtualbox-backed Minikube). • Both the ‘built’ Docker images and the service directories have scripts in bin for running all supported tasks (e.g. bin/repl, bin/run, etc.). Keeping this stuff consistent across services and images makes helps keep the scripts simple. • The Makefile really just delegates all work to a Bash script (e.g. bin/manage-dev run $service ./bin/repl or bin/manage-dev init kafka), which in turn makes the necessary calls to minikube, kubectl, and helm.

wombawomba21:05:29

In my day-to-day dev I usually just run whichever services I care about as REPLs and start them from there (I prefer standalone terminal REPLs to integrating with my editor, especially when I’m working on multiple services at once — which is almost always). All services are standalone and mostly use Kafka for communication, so there’s little need to spin up services I’m not working on directly.

wombawomba21:05:55

I’ve also set all my Clojure services up such that I can just call (refresh!) to reload the code or (stop-refresh-start!) to reload the code and restart the service, which makes this setup a lot smoother.

drewverlee03:05:59

I wonder, if your services are running in mini kube, for local development, would it be sufficient, assuming your using something like component or integrant to just hot reload the running code. Then commit changes when your happy and push them into what ever the next stage is.

Empperi06:05:51

Nice suggestions. In our case we have several languages in use, Clojure being just one of those. There's few python services and some javascript going on too

Empperi12:05:09

Which looks like exactly what I need. It's very early and most likely has bunch of bugs but since my use case would be (for now!) only for making local development environment setup easier that is ok

papachan17:05:41

reddit is down?