Running a real Kubernetes cluster inside one microVM

The newest release of smol machines runs a real Kubernetes cluster — k3d, backed by k3s — inside a single machine. That is a stronger claim than it reads, because Kubernetes is one of the most demanding things you can ask a Linux environment to host: it wants its own kernel, a cgroup hierarchy it controls, a working container runtime, and a specific set of device nodes. This post covers why a microVM can host that stack where a container cannot, and the one device node the kubelet refused to start without.

Why Kubernetes-in-a-container fights you

The usual way to run a cluster locally — kind, k3d — runs each Kubernetes node as a container. On a shared-kernel host that hits a wall. The kubelet and the container runtime expect to own a kernel: they configure cgroups, load kernel modules, mount filesystems, and program netfilter. Inside a container they are doing all of that against the host’s kernel, which they do not own and do not fully control. You work around it by granting the node container broad privilege, sharing the host’s cgroup hierarchy, and papering over the parts the container abstraction hides.

On a laptop the whole arrangement is usually running inside Docker Desktop’s Linux VM in the first place — the VM is doing the real work and the container is a formality on top of it. The isolation you actually get is thin, and the setup is easy to break.

A smol machine is a real VM

Each smol machine is its own microVM with its own Linux kernel, booted through a thin hypervisor. It has a real devtmpfs, a real cgroup v2 hierarchy, real netfilter, and real overlayfs — not a set of namespaces carved out of a shared host. Containers inside it run with full capabilities and no seccomp filter, which is what a container runtime and a kubelet need in order to set up cgroups, mount filesystems, and manage networking.

So a k3d node — itself a container — runs against a kernel the workload actually owns, and nested Kubernetes behaves the way it does on a dedicated machine. The full-capability posture inside the guest is not a cross-tenant risk here, because the boundary between workloads is the hypervisor, not the container runtime: each machine is a separate microVM, so nothing a container does can reach a neighbor’s kernel.

The device the kubelet demanded: /dev/kmsg

Even with a real kernel underneath, nested Kubernetes did not start. The failure was specific:

failed to create kubelet: open /dev/kmsg: no such file or directory

The kubelet opens /dev/kmsg — the kernel log device, character device 1:11 — during startup and refuses to run without it. A bare VM has /dev/kmsg: devtmpfs creates it automatically. But a container’s /dev is not the VM’s /dev. The runtime builds each container’s device tree from an explicit allow-list, and that list did not include kmsg — so the k3d node container came up without it and the kubelet bailed before it did anything.

The fix is to add /dev/kmsg to the device set the runtime assembles for every container:

// /dev/kmsg - kernel log device. The kubelet (k3s/k3d, kind, most
// Kubernetes-in-Docker) hard-requires it and refuses to start without it.
// Bare VMs get it from devtmpfs, but the container /dev is built from this
// list, so add it here so nested Kubernetes works out of the box.
OciDevice {
    device_type: "c".to_string(),
    path: "/dev/kmsg".to_string(),
    major: 1,
    minor: 11,
    file_mode: Some(0o644),
    uid: Some(0),
    gid: Some(0),
}

With the node container now holding the device the kubelet expects, the cluster starts. The security reasoning is worth stating plainly, because exposing the kernel ring buffer to a container is normally something you avoid: on a shared host the ring buffer is one buffer for everything on the box, so a container reading it sees its neighbors. Here it does not — each machine is its own microVM, so the kernel log the container reads is its own machine’s. The change ships with a test that asserts /dev/kmsg (1:11) is present in the device set, so a future refactor of the device list cannot silently drop it and break nested Kubernetes again.

Where the container storage has to live

One more thing has to be right for docker-in-a-machine, and therefore for k3d. Docker’s overlay2 graph driver needs a backing filesystem that supports overlay upper directories, and the machine’s root filesystem is already an overlay. You cannot nest overlay2 on top of another overlay — the kernel disables the index that overlay2 relies on. So the machine puts docker’s storage on the ext4 storage disk instead, binding /storage/docker to /var/lib/docker. containerd and overlay2 then build image layers on a real filesystem, which is what k3s and every image the cluster pulls depend on.

The result

Put together, standing up a cluster is a handful of commands against one machine:

# One machine, sized for a control plane
smol machine create --name k8s --net --cpus 4 --mem 4096
smol machine start  --name k8s

# docker's storage goes on the ext4 disk (overlay2 cannot nest on the rootfs overlay)
smol machine exec --name k8s -- \
    dockerd --data-root=/storage/docker --storage-driver=overlay2 &

# stand up a real cluster: k3d nodes are containers running k3s
smol machine exec --name k8s -- k3d cluster create dev
smol machine exec --name k8s -- kubectl get nodes

# NAME               STATUS   ROLES                  AGE   VERSION
# k3d-dev-server-0   Ready    control-plane,master   22s   v1.31.x+k3s1

The node reaches Ready, and from there it is an ordinary cluster — kubectl apply a deployment, expose a service, watch pods schedule. The difference from running the same commands against a container-based node is that everything underneath is a real kernel the machine owns, so the pieces that are fragile inside a container — cgroup delegation, overlay storage, iptables — behave the way they do on a dedicated host.

Why this matters

A personal note on why this one lands for me. I used to work on ECS and Fargate, and the hardest part was never the happy path — it was the edge cases. Iterating on them was brutal, because the runtime and the control plane could almost never run in the same environment. Reproducing a problem meant standing up a large pile of cloud infrastructure that took real effort to assemble and then drifted out from under you between runs. A machine that boots a real kernel and holds an entire cluster — control plane and runtime together — inside one reproducible, disposable artifact is exactly what I wanted back then: run the edge case on your laptop, the same bytes as everywhere else, and throw it away when you are done.

Beyond that, local Kubernetes with real isolation is useful on its own: you can develop and test against a cluster on your laptop without a shared host kernel and without the privileged-container workarounds. But the more interesting consequence is that a cluster is now a disposable, reproducible unit. A machine is a portable, fork-able artifact, so you can bring a cluster up in a golden machine once, fork it per test or per agent task, and tear each copy down on completion — provisioning a whole control plane only when a test needs it, and reclaiming it the moment the test is done.

Nested containers and Kubernetes ship in smol machines today — start with the quickstart, read the machine guide, or take the runtime from GitHub.