First Month of Kube
I knew going into this that Kubernetes was going to be a massive learning curve but this has been an incredibly tedious month.
Realistically, Kubernetes has been around since 2004 if you consider Google’s Borg which is the closed source predecessor of Kubernetes1. It became open source in 2014 so it’s been around for close to a decade. It is an incredibly mature and well developed platform… but it seems like a lot of what goes into it is the opposite. I don’t actually have a problem with Kubernetes itself, just all the things required to make it go.
Containers
I think containers are a brilliant idea as the ability to carve up OS resources to create segregated applications can offer opportunities for scaling, better security and better management.
Docker is probably the most recognisable container virtualisation platform but there was also it’s predecessor LXC which isn’t strictly the same thing, but more of a framework for containerisation.
Scaling is more easily achieved in containers than regular virtual machines because the container only needs the bare bones of system resources required to make a specific application (or set of applications, or even OS) run and can be spun up quicker than a VM with less host resources. This is different to a virtual machine which can basically be considered a copy of an OS running in virtualisation. Typically, this means a container is smaller than a VM but this is not always the case. The main point of difference between a VM and a container is the communication between the virtualised entity and the host it is running on. Amazon discusses the differences perfectly here.
Virtualisation offers security through isolation. Of course, this can be circumvented with a poor configuration. In a secure environment, the host cannot directly change the contents of the container’s environment such as OS files, app files, etc. The containers file system is separate and can be considered read-only from the hosts perspective. Conversely, the container cannot read or modify the host’s file system unless it has been poorly configured. An example of this is if the container can perform a type of VM escape attack2. Docker provides a way to create networks and lock down communication between containers, the host and the internet. This software defined networking can be very useful but again, it can be affected by the same misconfiguration issues. Like everything in IT, we must be cognizant of the security ramifications of what we implement and containerisation can be a double edged sword if not carefully and knowingly configured.
YAML
Literally standing for Yet Another Markup Language, YAML is my most hated data format. I don’t like any data format which has the necessity of whitespace in parsing, especially when it is so opinionated that it must be exactly 2 spaces. Why not 1 space, or a tab? Why must I be at the mercy of a parsing algorithm with an arbitrary constraint? RedHat actually called YAML a programming language and I shudder at the thought3.
I’d much rather write an .ini file or mess around in some disgusting XML, which is my second most hated data format, but that should give you an idea of how much I dislike YAML.
TOML is much better to read and reason about so I wish they supported this instead. TOML stands for Tom’s Obvious Markup Language and whoever Tom is, he’s a genius. At some point, Helm, which can be thought of as a package manager for Kubernetes did support TOML but they deprecated it in favour of YAML according to this GitHub issue. This reasons they give can be boiled down to “everybody else was doing YAML”. Lots of people smoke cigarettes, but that doesn’t mean you should take up smoking.
Helm
At first, Helm seems like a great idea because you can install “charts” and get applications and their dependencies spun up with very little effort. This is definitely beneficial in the exploration phase and knowing what you might want in your cluster.
The downsides of this is that there are so many different Helm charts already for the same products. Like all package managers, it has become polluted by opinion. One person creates a chart with all the configuration they believe is necessary, then another comes along and thinks they can do a better or different job and creates a new chart that does something quite similar. Why does this happen? An application is capable of being well-defined. I.e. I am an application, I have data persistence, I have an ingress point, and I have a set of roles and accounts. There is finite set of possibilities for configuration but chart creators find a way to deliver a subset or opinionated modification of that configuration. Kubernetes was already filled with more knobs and dials than a single human brain can comprehend, and Helm allows that difficulty to be magnified.
I don’t want to dive into too much detail on why package managers are evil but this video, https://www.youtube.com/watch?v=5IUj1EZwpJY by Casey Muratori is truly a masterpiece at explaining why. Simply, because of Conway’s Law, a package manager is inadvertently creating barriers against optimisation.
In regards to Helm, don’t let yourself be reliant on this system. Pull apart charts and identify the individual pieces you need for your system and implement them in a way that makes sense and that you control. Your use case is your use case so take ownership of the solution you are implementing.
The Community
The community vibe from users of Kubernetes is generally pretty good. There is a clear differentiation between users though; there are the die hard Kube experts and then there is everyone else.
Most people are very willing to help with Kubernetes but the problem is doing anything with the information gained. The following is entirely fictional dialogue but sums up what it has been like conversing about Kubernetes:
Me: I have a problem, P, that I’m trying to solve in Kubernetes. Should I choose X or Y to do that?
The Kube Expert (TKE): Yeah, it depends… Are you running A or B in your cluster already?
Me: We chose A after reading this article about Kubernetes best practices.
TKE: That makes sense but that was best practices several hours ago. The industry has moved on. Try C instead.
Me: C? Not B? Wasn’t the choice between A or B?
TKE: C? No, don’t use that, it’s deprecated. Implement D. Do you still have problem P?
Me: Yeah, I’ve got D now, so do I pick X or Y?
TKE: That won’t solve P. First, set up GitOps; it’ll be an industry standard.
Me: Ok, I’ve set up GitOps. Now, how does that help me solve P?
TKE: It won’t, but we’ll get to that. Now, sacrifice a Microsoft user on the Shrine of Linus Torvalds so that your GitOps controller works as expected today. Then, pick Helm or Kustomize.
Me: Sacrificed. Let’s say I pick Helm?
TKE: Nope, pick the other one.
Me: For ***** sake… Kustomize?
TKE: Kustomize??? Why wouldn’t you pick Helm, its way easier.
Me: Fine. Helm. Now that everything in my environment is managed by Helm except all those things that need to be Kustomize’d, what do I do to solve P?
TKE: P? Kubernetes is a great choice except for problems like P. You might want to rethink Kubernetes in your environment.
Me: Ummmm, ok… I’ll work out how to solve P without Kubernetes then.
TKE: Why wouldn’t you choose Kubernetes? You can solve P with Kubernetes. Kubernetes solves everything.
The main issue here is that the answer to every question regarding Kubernetes is “it depends” and no matter what subsequent responses are, the likely follow-up will also be “it depends” in this horribly recursive cycle of no real answers. No real answers powered by people with strong opinions.
I can see why Kubernetes is like this. There is a plethora of knobs and dials to twist and turn to make things go. Even worse is how many people provide other knobs and dials to replace/augment existing ones. Each knob and dial comes with a shop manual to use it correctly. If read out of order or misunderstood, these manuals are basically spell tomes to summon demons. Read the documentation, twist the dial, watch your cluster shit the bed, read on a buried GitHub issues page that you aren’t meant to twist that dial despite it’s dial-like nature; you were meant to flick it up and down while whispering to it that it’s such a good lil dial. Computer generated documentation is garbage.
Conclusion?
I don’t know. I am not sold on it either way yet. Every one of the problems that I have encountered so far is annoying but expected. These are typical teething problems with something that has such a learning curve and vast problem space.
Though, I still don’t yet see the value proposition of Kubernetes. A VM running Docker could sit on a rack in a DC and engineers could deploy containers to that. There is nothing less or more secure about defining that thing’s setup than deploying a Kubernetes cluster there. I’d wager that the Kubernetes cluster is more likely insecure because of the sheer amount of configuration that could go wrong.
I eagerly await seeing evidence that Kubernetes is the right path for anyone.