Branching running machines and keeping checkpoint history

AI agents build up state as they work: running processes, a half-built repo, a logged-in browser. You often want to try several next steps from the same point, or go back to an earlier point later. Most microVM platforms use Firecracker snapshots for this. smol machines uses a different design. This post covers how it works and where each approach is better.

At a glance

How Firecracker does it

Firecracker's snapshot API does one thing. You pause the VM and it writes two files: guest memory and device state. Disks are not included. The docs say they are "managed by the users".

• Full snapshots write all of memory.
• Diff snapshots write only the pages changed since the last snapshot. A diff can't be restored alone: you rebase the chain onto its base, in order.
• Clones come from restoring one snapshot several times. The memory file is mapped MAP_PRIVATE, so clones share pages until they write to them.
• Uniqueness is up to you. The docs call resuming the same snapshot twice "insecure". VMGenID reseeds the guest kernel's RNG, but IDs, tokens and caches are still duplicated.

PATCH /vm              {"state": "Paused"}
PUT   /snapshot/create {"snapshot_type": "Diff", "mem_file_path": "./diff-2.mem", ...}
PATCH /vm              {"state": "Resumed"}

# later: merge the chain back into one restorable memory file
snapshot-editor edit-memory rebase --memory-path base.mem --diff-path diff-1.mem
snapshot-editor edit-memory rebase --memory-path base.mem --diff-path diff-2.mem

Firecracker gives you the primitive and nothing else. Matching diffs to bases, keeping disks consistent with memory, and defining what a branch is: every team on Firecracker builds this itself.

Branching a running machine

In smol machines, you branch a running machine. Start it with --branchable and its RAM is backed by shared memory. When you branch, the engine pauses the source, freezes its RAM and disks into a shared base, and resumes the source on a new private layer. The child starts from the same moment. On a MacBook, the pause was about 80 ms for a 1 GiB machine.

smolvm machine start  --name agent --branchable
smolvm machine branch --from agent --name try-a
smolvm machine branch --from agent --name try-b

Try it: branch the source while it runs, then write from each side.

Firecracker clones use the same copy-on-write approach. Four things are different:

• The source keeps running, with its processes and connections.
• Disks are copied too, so each child's files match its memory.
• Each child is a new machine: its own name, hostname, machine id and ports. It records which machine it came from.
• It runs on macOS. Firecracker needs Linux and KVM, so it can't branch a machine on a Mac.

Checkpoints capture the whole lineage

A branch stays on the host where it was made. A checkpoint writes the whole machine, disks included, to a file you can restore on another host with the same OS and CPU architecture.

Most snapshots capture one point in time. A smol machines checkpoint also captures its lineage. Each checkpoint records its parent: the checkpoint the machine was last captured to or restored from. Capture a machine several times and you get a chain. Restore an old checkpoint and capture again, and the history branches. Nothing is overwritten. It works like checking out an old git commit and committing on top of it.

smolvm machine checkpoint --name worker --store ./ckpt -o ./ckpt-3

~0   daa66d13daa5  worker  (this checkpoint)
~1   3c6ef3623c6e  worker
~2   9e3779b19e37  worker

smolvm machine checkpoint-log ./ckpt-3
# ~0   9f2c1a7b3e4d  worker  (this checkpoint)
# ~1   51e0d8c2a9b4  worker
# ~2   0c7a44e1fd93  worker

smolvm machine create --name rewind --from ./ckpt-3 --at ~2

Deleting old checkpoint files

Lineage only helps if old points still restore after you delete files. This is the biggest difference from a diff chain.

• A diff chain breaks if you lose a link. Each diff depends on the ones before it. Lose the base or any diff, and every later point stops restoring. Flattening fixes this, but then every point is a full copy.
• A smol machines checkpoint doesn't depend on older files. It holds a full index and hard links into a content-addressed chunk store, plus the indexes of its earlier points. The latest checkpoint restores every point in its history on its own.

Delete files on both sides and see what still restores:

To move the history to another host, --export-from packs a checkpoint and its earlier points into one file. In our demo, three points took 98 MB, compared with 52 MB for one, because only the changes are stored. On Linux, a restore now uses the checkpoint's disk as a shared read-only base instead of copying it. On our test host, that cut restore time from 1.3 s to 0.26 s.

The trade-offs

• Capture cost: Firecracker wins. Diff snapshots write only changed pages, straight from KVM. Our checkpoints still read all of RAM; the store only skips writing chunks that haven't changed. If you capture every few seconds, dirty-page capture is cheaper.
• Scope: different layers. Firecracker handles memory and devices and leaves the rest to you. smol machines checkpoints and branches the whole machine and tracks its history.
• Portability: the same limit. Neither can move a running machine to a different CPU architecture or OS.
• Limits. Branch lineage and disk chains go 32 levels deep. Checkpoint history keeps 32 points by default.

All of this is in smolvm 1.18.0 and later.

Get the runtime on GitHub, or read how checkpoint history works.