Back to the lab

The first articles here started with a single Fedora VM and ended with k3s, a registry and Gitea on one hypervisor. A lot has changed since. This series starts from the same place, a kickstarted Fedora VM, and goes much further: one Kubernetes cluster spread across several machines, with no NAT and no tunnels, its API behind a BGP virtual IP, storage that survives losing a machine, and every change in the cluster recorded.

Then we break it on purpose and measure what happens.

The lab

Everything runs on one Fedora 44 box, so you can follow along at home. Two nested VMs, saturn and jupiter, play two separate machines on two routed networks.

On them runs one Kubernetes cluster, rootless, in Podman, with three control planes and two workers. Its API answers on a single address announced over BGP.

The host itself plays the router and the storage server. BIRD speaks BGP to kube-vip and MetalLB, and SeaweedFS provides shared storage.

An event recorder watches the cluster from the first second of bring-up and writes down every change, so when a node dies we can check exactly what was seen and what was missed.

The whole lab builds from nothing in about 45 minutes, from plain numbered scripts, one per machine. Those scripts are the commands in the articles, so what you read is what was run.

The series

Foundations comes first: refreshed versions of the early articles on Fedora 44. That means trying Fedora 44 with an unattended kickstart install, setting up libvirt, and building a base image with cloud-init.

One cluster, several machines: routed networks with open libvirt networks and no NAT, Kubernetes across two machines, the Kubernetes API behind a BGP virtual IP, and MetalLB services and failover, measured.

Operating it: storage that survives a node with SeaweedFS and its CSI driver, watching every change with audit logs and an event recorder, losing control planes, losing machines (including the node the observer runs on), and rebooting the whole thing.

And finally, what building and breaking it taught me.

Measured, not assumed

Every scenario is checked against something outside the cluster: a probe hitting the API several times a second, and a record of every write the API confirmed. "It recovered" becomes "the API was unreachable for 6.4 seconds, and nothing was lost".

The first articles are on their way.