---
title: "Branch and checkpoint: scheduled checkpoints, restores and branch points"
description: "Scheduled checkpoints with history, restores, pause and resume, and branch points."
canonical: "https://smolmachines.com/docs/guides/skills/branch-and-checkpoint"
source: "https://github.com/smol-machines/docs/blob/main/guides/skills/branch-and-checkpoint.md"
---

# Branch and checkpoint: scheduled checkpoints, restores and branch points

Saves a running smolvm machine as checkpoints on a schedule, keeps a bounded history, restores any generation in it, pauses and resumes machines without losing running processes, and branches a warm machine into children with a checkpoint of the exact point they started from. Use when a machine's state has to survive the machine; when an agent or a job needs to roll back; when checkpoints must be taken periodically, since smolvm has no scheduler or retention of its own; when fanning a prepared machine out into workers; or when a checkpoint, a pause or a branch fails with an error that names none of its preconditions. Do not use it to ship an environment to another host as a file, which is the pack packet, or for state that only has to survive a stop and start, which is the dev-env packet.

Verified on **smolvm v1.29.0** on macOS arm64, 2026-10-15, and on **v1.18.2** on Linux aarch64, 2026-09-24. Done means a restore
brings back both the disk and the memory you left, the schedule keeps exactly the checkpoints you
asked for and no fewer, and every child of a branch starts from a state you still hold as a
checkpoint after the children are gone.
The Linux runs used the scripts of their date; this version's preflight and cleanup scripts ran
on Linux aarch64 on v1.22.2 on 2026-10-03.

`README.md` says what each of these operations is. This file is how to run them, and the traps.

**Three facts shape everything here.**

- **smolvm has no scheduler and no retention.** `scripts/checkpoint.sh` is the one capture a timer
  runs, with the rules the engine asks a scheduler to follow built in; `references/scheduling.md`
  puts it under cron, systemd or launchd.
- **A branch's captured state stays on this host and cannot be exported.** So a branch point is
  paired with a checkpoint taken at the same moment, which is the copy you keep.
- **`--branchable` at `start` decides what a machine can do later.** A branch source needs it on
  every host, and on macOS so does a stored capture, which is what the scripts here take; Linux
  checkpoints and pauses without it. `README.md` has the errors when it is missing. The scripts
  always start with it.

## Procedure

```
- [ ] 1. preflight.sh               read result= and store_space_ok=
- [ ] 2. create-source.sh           a machine with state in RAM and on disk
- [ ] 3. checkpoint.sh / schedule.sh   captures, with retention
- [ ] 4. restore.sh                 any generation, into a new machine
- [ ] 5. pause and resume           the restored machine, or any running one
- [ ] 6. branch-point.sh            children plus a checkpoint of where they started
- [ ] 7. cleanup.sh                 machines, checkpoints, and smolvm's restore cache
```

**1. Preflight.** Read-only.

```bash
scripts/preflight.sh --store ./store
```

It checks that this binary has `--store`, `--history`, `checkpoint-log`, `pause` and restore
`--at`, says whether this host needs `--branchable`, and reports `scheduler=builtin_none` and
`retention=builtin_none` so nobody goes looking for them. `store_space_ok=no` blocks: a capture
into a full disk publishes nothing, and a machine restored while the disk was full was left with
guest I/O errors that stopped it being deleted normally.

**2. A source with state you can check.**

```bash
scripts/create-source.sh smolskill-src                 # for checkpoints and restores
scripts/create-source.sh smolskill-bp --branch-ready   # for a batch branch
```

The workload writes a counter into `/tmp` every second, which is tmpfs and so RAM, and a marker to
`/root/setup.txt` on disk. A restore that brings the counter back proves memory came back, not only
the disk. `--branch-ready` makes the workload park in `smolvm-branch-ready` after its setup, which a
batch branch waits for.

**Or the user's own machine.** Name it with `--name` in the steps below. On macOS, if it was
started without `--branchable`, the first capture fails with `deferred durable save requires
file-backed guest RAM` on v1.22.2 or `guest RAM has no file-backed regions` before v1.20.0, and the
only fix is a restart with the flag:

```bash
smolvm machine stop  --name worker
smolvm machine start --name worker --branchable
```

That restart is a normal stop and start: disk state survives, and anything in RAM or `/tmp` is
lost, once. Say so to the user before doing it. Every start of that machine needs the flag from
then on for it to stay checkpointable.

A capture also refuses a machine with resources bound to this host: a host mount (`-v`), a
published socket, a remote volume, a host secret reference, custom DNS, a named network, a GPU,
CUDA, Rosetta, `--ssh-agent` or a Docker socket. The error, `the initial portable checkpoint
profile does not support ...; stop or detach these resources before capture`, names which. From
v1.29.0 `machine checkpoint --host-mounts` captures a machine with host mounts, and the checkpoint
restores only on a host where the same paths exist.

**3. Capture, once or on a schedule.**

```bash
scripts/checkpoint.sh --name smolskill-src --store ./store --keep 6 --history 2
scripts/schedule.sh   --name smolskill-src --store ./store --every 10 --times 4 --keep 2 --history 2
```

Each capture goes to a new directory named `<machine>-<label>-<UTC time>.smolcheckpoint` beside the
store, or in the directory `--out <dir>` names, which has to be on the same filesystem as the
store. It is judged published only when `checkpoint-log` lists it as `(this checkpoint)`, and only
then are the oldest beyond `--keep` deleted and the store pruned. A capture started while another
of the same machine runs exits 3 with `result=skipped`. `schedule.sh` is for a session you are
watching; a schedule that outlives it is `checkpoint.sh` under a system timer,
`references/scheduling.md`.

```
run=1 rc=0 took_s=2 ... kept=1 store_size= 59M result=captured
run=2 rc=0 took_s=5 ... kept=2 store_size= 76M result=captured
run=3 rc=0 took_s=2 ... kept=2 store_size= 95M result=captured
run=4 rc=0 took_s=1 ... kept=2 store_size=114M result=captured
captures_ok=4
captures_failed=0
longest_capture_s=5
result=schedule_ok
```

That is the v1.18.2 run. Ten seconds is for watching it work; a schedule uses minutes,
`references/scheduling.md`.

**Set the interval from `longest_capture_s`, not from the pause.** The source pauses for a few
hundredths of a second; the command takes seconds, because retained RAM is hashed and compressed
after the source resumes. **`--keep` alone does not bound the disk**: every kept checkpoint also
retains `--history` earlier generations, 32 by default, so the store above kept growing with two
directories kept. Bound both.

**4. Restore, any generation.**

```bash
smolvm machine checkpoint-log ./smolskill-src-ckpt-<time>.smolcheckpoint
scripts/restore.sh --from <checkpoint> --name smolskill-old --at '~2'
scripts/restore.sh --from <checkpoint> --name smolskill-new
```

`~0` is the checkpoint itself and `~N` is N generations back along its history. `restore.sh`
takes only a name starting with `smolskill-`, so cleanup can find it, and refuses any other with
`name must start with smolskill- so cleanup.sh will delete it`; to restore under a name the user
chooses, run `smolvm machine create --name <name> --from <checkpoint> --at '~N'` and then
`smolvm machine start --name <name> --branchable` yourself. The restore path is
`machine create --from`; **there is no `machine restore`**. A restored machine takes its own name as
its hostname, and its disks are copy-on-write over what the checkpoint holds, so a restore costs
about 29 MB of real disk on macOS for a 243 MiB checkpoint while `du` reports five times that.

**5. Pause and resume.**

```bash
smolvm machine pause  --name smolskill-new
smolvm machine resume --name smolskill-new
```

Pause saves RAM, disks and the running execution and stops the machine; resume brings the same
execution back under the same name. Check a value the workload holds in memory: on both hosts the
counter read 31 before the pause and 34 and 33 after the resume, so it continued where it stopped
and did not run while paused. A paused machine refuses `stop` and `start`; resume it or delete it.
A resume that fails keeps the saved state, and a later resume of the same machine succeeded.

**6. Branch, and keep the branch point.**

```bash
scripts/branch-point.sh --from smolskill-bp --store ./store --count 2 --name-prefix smolskill-w
scripts/branch-point.sh --from smolskill-src --store ./store --name smolskill-one
```

For a batch it waits until a process in the source runs `smolvm-branch-ready`, takes the checkpoint while
the source is parked there, then branches. Every child and that checkpoint then hold the same
state: a random value the source wrote to RAM before parking came back identical in both children
and in a restore of the checkpoint.

```
source_parked_after_s=1
checkpoint=.../smolskill-bp-branchpoint-<time>.smolcheckpoint
child=smolskill-w-0
child=smolskill-w-1
source_state=running
result=branched
```

For one child with `--name` the checkpoint is taken first and the branch a second or two later, so
the two points differ by whatever the source did in between.

**A restored branch point is not a parked source.** On start the helper releases and the workload
runs the child program with an empty `SMOLVM_BRANCH_NAME`, so a batch branch from it waits for a
branch point that never comes. Branch it with `--name`, one child at a time, or restore it once per
worker; either way the child gets the state the original children started from.

**7. Clean up.**

```bash
scripts/cleanup.sh --purge --restore-base --checkpoints ./store ./*.smolcheckpoint
```

It deletes the machines the scripts recorded, children first by `--cascade`, removes the named
checkpoint directories and stores, and with `--restore-base` removes smolvm's copies of restored
checkpoints, which otherwise stay after every machine and checkpoint is gone, printing a
`removed=` or `absent` line for each. It waits up to 20 seconds, polling the machine list, and
says so, because an ephemeral entry retires after its run returns.

## Traps

Full detail in `references/traps.md`. The ones that cost the most:

- **`--output` must end in `.checkpoint` or `.smolcheckpoint`**, with `--store` too, where it names
  a directory. `.checkpoint` is the name from v1.19.1, and the scripts use `.smolcheckpoint`, which
  every release accepts.
- **`pgrep -f smolvm-branch-ready` inside the guest always matches**, because the `sh -c` running it
  contains the string. Read each `/proc/<pid>/cmdline`: from v1.25.4 the workload is not PID 1.
- **`smolvm machine list | grep -q` under `pipefail` can report a machine missing that exists**:
  `grep -q` closes the pipe early and the failed write fails the pipeline. Read the list into a
  variable first.
- **`machine stop --name` on a name that does not exist leaves an empty directory** under the VM
  cache on v1.18.2, and a second stop of it on v1.22.2 leaves the name inside, which reads as a
  leak. `cleanup.sh` stops only names still listed.
- **smolvm keeps restored checkpoints**, memory included, after the checkpoint is deleted
  (`_restore-base` was 243 MB on macOS on v1.18.2); `README.md` has the rest, and step 7's
  `--restore-base` removes them.

## Security defaults, and why they are the defaults

- **A checkpoint is the machine's memory and disks.** Anything the workload held, secrets included,
  is in it; keep stores on a disk only you can read, and remove `_restore-base` and
  `_restore-checkpoints` when you remove the checkpoints.
- **Retention deletes only after the new checkpoint is published**, so a failed capture never
  leaves you with fewer good checkpoints than before.
- **A store is local.** It protects against a bad change, not a lost host; export a checkpoint
  with `machine checkpoint --export-from` and copy it elsewhere for that.
- **Cleanup deletes only machines the scripts recorded under the `smolskill-` prefix, and through
  `--cascade` any machine branched from one of them, whatever its name**, and removes only the
  checkpoint paths you name.
- **Nothing here escalates privilege** or edits smolvm configuration.

## Platform arms

`references/platforms.md` has each arm. In short: **macOS arm64** ran every step here on v1.29.0,
and **Linux aarch64** on v1.18.2 and once on v1.22.2, with the one difference that Linux needs no
`--branchable`. On v1.16.1 Linux aarch64 refused `--store` and froze a branch source; from v1.18.2
it does neither. **Linux x86_64**
was not re-run. **Windows** on v1.22.2 checkpoints to a file, pauses and branches with
`--freeze-source` on every branch, and refuses `--store`, so `checkpoint.sh` cannot run there.
From v1.23.0 a checkpoint from an arm64 Mac restores on arm64 Linux, run once from this Mac to Lima.

## Eval prompts, and what they produced

Run 2026-09-24 PT against v1.18.2 from the published release, under an isolated `HOME`, on macOS
26.6.2 arm64 and Lima `linux-kvm` (Ubuntu 24.04 aarch64), every machine at 1024 MiB. Output is
verbatim unless marked.

**1. "Checkpoint this machine every few minutes, keep only the last few, and show me I can go back
to any of them."**

`schedule.sh` with `--keep 2 --history 2`, four captures ten seconds apart, both hosts:
`captures_ok=4`, `kept=2`, and `checkpoint-log` on the newest listed `~0`, `~1` and `~2`. A marker
written between captures came back as `GEN1` from `--at '~2'` and `GEN3` from the newest, with the
RAM counter restored alongside. A second capture started while one ran:

```
result=skipped
note=a capture of smolskill-src is still running (pid 13935). Run one capture at a time per source: ...
```

**2. "Fan this prepared machine out into workers, and keep a copy of exactly the state they started
from."**

`branch-point.sh --count 2` on a `--branch-ready` source, macOS: the source wrote `8445` to RAM
before parking; both children read `SETUP_DONE`, their own `CHILD=` name and `8445`, and a restore of
the branch-point checkpoint read `8445`. Linux the same with `22642`. `source_state=running` on both.

**3. "Pause the machine I restored and bring it back later without losing what is running."**

The restored machine's counter: `ram before=31 after=34` on macOS and `ram before=31 after=33` on
Linux, across a 10 s pause. Separately, on macOS a restored machine paused while the host disk was
full failed its first resume with `extract paused checkpoint: failed to unpack ...`, and the same
`machine resume` succeeded once space was freed: counter 8 before the pause, 14 after, still
counting.

## Re-verified on v1.29.0

Run 2026-10-15 PT against v1.29.0 from the published release, checksum checked, under a fresh
isolated `HOME` on macOS 27.0.1 arm64, the scripts twice, the second time from a fresh `HOME`, and
the checks beyond them once. Linux was not run, so the Linux stamp stays on its earlier release.

`schedule.sh` gave `captures_ok=4` and `longest_capture_s=1`, `checkpoint-log` listed `~0` to `~2`,
and restores of `~2` and the newest gave `result=restored`. The RAM counter read 32 before a 5 s
pause and 33 after the resume. A batch of two branched with the same RAM token in both children
and in a restore of the branch point, once `branch-point.sh` read every process for the parked
helper (`references/traps.md`, Corrections); before that fix it reported `the source never parked`.
One child with `--name` branched. A capture of a machine with a host mount failed with the profile
error above, and with `--host-mounts` it succeeded.

## Re-verified on v1.23.0

Run 2026-10-04 PT against v1.23.0 from the published release, checksum checked, under a fresh
isolated `HOME` on macOS 27.0.1 arm64, once. On Lima `linux-kvm` (Ubuntu 24.04 aarch64) the
checks named below ran once, so the Linux stamp stays on its earlier release.

macOS: every script. Three scheduled captures (`captures_ok=3`, `longest_capture_s=1`),
`checkpoint-log` listing three generations, restores of `~2` and the newest, the counter at 20
before a pause and 24 after the resume, both children of a batch branch holding the source's token,
and `cleanup.sh --restore-base` removing 258 MB of `_restore-checkpoints`. A capture now reports
when the source resumed, `Source resumed after 0.024s; packaging continued in the background`
([#1519](https://github.com/smol-machines/smolvm/pull/1519)).

**A checkpoint taken on this Mac restored on Linux aarch64**
([#1531](https://github.com/smol-machines/smolvm/pull/1531)). `create-source.sh`, then
`machine checkpoint --name smolskill-mac --output mac.checkpoint` on the Mac with the RAM counter
at 6, which printed `40 MiB written` and `0.066s source pause`. Copied to Lima unchanged,
`machine create --from` and `machine start` there took a second each; the counter read 7 and five
seconds later 12, the RAM token and the disk marker matched the Mac's, and the guest said
`Linux 6.12.95`.

## Re-verified on v1.22.2

Run 2026-10-03 PT against v1.22.2 from the published release, checksum checked, under an isolated
`HOME` on macOS 27.0.1 arm64, twice, the second time from a fresh `HOME`. On Lima `linux-kvm`
(Ubuntu 24.04 aarch64) on 2026-10-03 guests above 2048 MiB timed out, so the Linux lines below are
a single run and the Linux stamp stays on its earlier release.

macOS: every script. Three scheduled captures (`captures_ok=3`, `longest_capture_s=1`),
`checkpoint-log` listing three generations, restores of `~2` and the newest, the counter at 21
before a pause and 25 after the resume, both children of a batch branch holding the source's token,
and `cleanup.sh --restore-base` removing 275 MB of `_restore-checkpoints`. The `--branchable`
facts in "Branchability is decided at start" were measured on releases v1.18.2 to v1.22.2 on
2026-10-03.

Linux aarch64: every script, with `checkpoint_needs_branchable=no`, the counter 63 then 66 and both
children holding the same token.

## What was not run

- **Linux x86_64**, and the scripts on Windows, which are bash.
- **A schedule over hours.** Four captures ten seconds apart on each host, and single captures
  within each run; the growth of a store under `--history 32` over a day was not measured.
- **Restores on another host, beyond Mac to Linux.** Through v1.22.2 a checkpoint restores only on
  a host of the same platform, and another is refused with `checkpoint host platform '...' does not
  match '...'`. From v1.23.0 one taken on an arm64 Mac restores on arm64 Linux, run once above;
  [Moving a running machine from a Mac to Linux](https://github.com/smol-machines/smolvm/blob/main/docs/mac-to-linux.md) has the requirements. Linux to
  Mac, and a stored capture rather than a file, were not run.
- **GPU and CUDA machines**, which a capture refuses, and `--share-weights`.
- **Branch pools** (`--hold`, `branch-release`), and `--freeze-source` outside Windows.
- **The API's restore route**, the `from` field on create; the `local-api` packet covers pause and
  resume over HTTP.

## Related packets

- `dev-env` for state that only has to survive a stop and start.
- `pack` for moving a machine to another host as a file, including a restored one.
- `local-api` for pause and resume over HTTP.
- `teardown` for the wider cleanup, and for what a leak check must exclude.

## Scripts

The files the procedure runs, in the order it runs them. It calls each one by the path in its heading, relative to the directory the procedure is saved in.

### `scripts/preflight.sh`

```bash
#!/usr/bin/env bash
# Report whether this host can checkpoint, restore, pause and branch smolvm
# machines, and whether the place the checkpoints will go has room. Read-only:
# starts no VM, writes no smolvm state, changes no group membership.
#
# usage: preflight.sh [--store <dir>]   (default: the current directory)
#
# Output is one key=value per line so a caller can parse it. The last line is
# always result=ready or result=blocked.

set -uo pipefail

VERIFIED_VERSION="1.29.0"

emit() { printf '%s=%s\n' "$1" "$2"; }

blocked=0
note() { printf 'note=%s\n' "$1"; }

STORE_DIR="."
while [ $# -gt 0 ]; do
    case "$1" in
        --store) STORE_DIR="$2"; shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done

# --- the binary --------------------------------------------------------------

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
    emit smolvm_installed no
    emit smolvm_version ""
    blocked=1
else
    emit smolvm_installed yes
    version="$("$SMOLVM" --version 2>/dev/null | awk '{print $NF}')"
    emit smolvm_version "${version:-unknown}"
fi

emit verified_version "$VERIFIED_VERSION"
if [ -n "${version:-}" ] && [ "$version" != "unknown" ]; then
    if [ "$version" = "$VERIFIED_VERSION" ]; then
        emit version_status match
    else
        newest="$(printf '%s\n%s\n' "$version" "$VERIFIED_VERSION" | sort -V | tail -1)"
        if [ "$newest" = "$version" ]; then
            emit version_status newer
            note "this packet was verified on $VERIFIED_VERSION and the binary is $version; flags and messages move every release, so check the output against the binary before trusting a step here"
        else
            emit version_status older
            note "this packet was verified on $VERIFIED_VERSION and the binary is $version"
        fi
    fi
else
    emit version_status unknown
fi

# --- platform ----------------------------------------------------------------

kernel="$(uname -s)"
arch="$(uname -m)"
case "$arch" in aarch64|arm64) arch=aarch64 ;; esac

case "$kernel" in
    Darwin)
        emit platform "darwin-$arch"
        emit accel hvf
        emit macos_version "$(sw_vers -productVersion)"
        hv="$(sysctl -n kern.hv_support 2>/dev/null)"
        if [ "$hv" = "1" ]; then emit accel_access ok; else emit accel_access denied; blocked=1; fi
        if [ "$arch" != "aarch64" ]; then
            emit hardware_verified no
            note "Intel Mac is not verified by this packet; the installer accepts it and nothing here was run on one"
        else
            emit hardware_verified yes
        fi
        # A VM's agent socket lives under the cache directory. macOS sockaddr_un
        # holds 104 bytes including the terminator.
        sock="$HOME/Library/Caches/smolvm/vms/0123456789abcdef/agent.sock"
        len=${#sock}
        emit socket_path_bytes "$len"
        if [ "$len" -gt 100 ]; then
            emit socket_path_status too_long
            blocked=1
            note "HOME is too deep: every VM start will fail. From v1.28.0 the error names the socket path and its length; before it, krun_start_enter -22, whose text blames disks and device options. Install under a shorter HOME."
        else
            emit socket_path_status ok
        fi
        # v1.20.0 lifted the flag for a checkpoint file and for pause; a stored
        # capture, which is what these scripts take, still needed it on v1.22.2
        # and no longer does on v1.31.1.
        if [ -n "${version:-}" ] && [ "$(printf '%s\n%s\n' "$version" 1.31.1 | sort -V | head -1)" = "1.31.1" ]; then
            emit checkpoint_needs_branchable no
        elif [ -n "${version:-}" ] && [ "$(printf '%s\n%s\n' "$version" 1.20.0 | sort -V | head -1)" = "1.20.0" ]; then
            emit checkpoint_needs_branchable yes
            note "on macOS machine checkpoint --store refuses a machine that was not started with machine start --branchable, with deferred durable save requires file-backed guest RAM, which names neither. A checkpoint file and a pause do not need it from v1.20.0. The flag cannot be added to a running machine; the scripts here always start with it."
        else
            emit checkpoint_needs_branchable yes
            note "on macOS machine checkpoint and machine pause refuse a machine that was not started with machine start --branchable, and the error, guest RAM has no file-backed regions, names neither. The flag cannot be added to a running machine; the scripts here always start with it."
        fi
        ;;
    Linux)
        emit platform "linux-$arch"
        emit accel kvm
        emit socket_path_status n_a
        if [ ! -e /dev/kvm ]; then
            emit accel_access missing
            blocked=1
            note "/dev/kvm does not exist; this host has no KVM"
        elif [ -r /dev/kvm ] && [ -w /dev/kvm ]; then
            emit accel_access ok
        else
            emit accel_access denied
            blocked=1
            note "your user cannot open /dev/kvm. The installer warns and continues, so a successful install says nothing about whether a VM will start. Fix: sudo usermod -aG kvm \$USER, then run the next command through sg kvm -c '...' rather than logging out."
        fi
        emit checkpoint_needs_branchable no
        ;;
    *)
        emit platform "unsupported-$kernel"
        emit accel unknown
        emit accel_access unknown
        blocked=1
        note "this script covers macOS and Linux. On Windows v1.22.2 checkpoints to a file, pauses, and branches with --freeze-source on every branch, and refuses --store, so checkpoint.sh cannot run there; see references/platforms.md."
        ;;
esac

# --- the checkpoint surface this packet uses -----------------------------------

if [ -n "$SMOLVM" ]; then
    ckhelp="$("$SMOLVM" machine checkpoint --help 2>&1)"
    for flag in --store --history --export-from; do
        k="has_$(printf %s "${flag#--}" | tr - _)"
        if grep -q -- "$flag" <<<"$ckhelp"; then emit "$k" yes; else emit "$k" no; blocked=1; fi
    done
    if "$SMOLVM" machine checkpoint-log --help >/dev/null 2>&1; then emit has_checkpoint_log yes; else emit has_checkpoint_log no; blocked=1; fi
    if "$SMOLVM" machine pause --help >/dev/null 2>&1; then emit has_pause yes; else emit has_pause no; fi
    createhelp="$("$SMOLVM" machine create --help 2>&1)"
    if grep -q -- '--at <GENERATION>' <<<"$createhelp"; then emit restore_any_generation yes; else emit restore_any_generation no; blocked=1; fi
    emit scheduler builtin_none
    emit retention builtin_none
fi

# --- memory and disk ---------------------------------------------------------

# The scripts start every machine with --mem 1024. A host that cannot boot that
# inside smolvm's fixed 30 s readiness window fails with "agent did not become
# ready", which names neither memory nor the host.
emit memory_required_mib 1024
mkdir -p "$STORE_DIR" 2>/dev/null
free_kb="$(df -Pk "$STORE_DIR" 2>/dev/null | awk 'NR==2{print $4}')"
if [ -n "$free_kb" ]; then
    emit store_free_mib "$((free_kb / 1024))"
    # A capture of a 1 GiB alpine machine wrote about 55 MiB the first time and
    # about 19 MiB after that; restores, the restore base and the capture
    # staging need more. 2 GiB is headroom, not a measurement of any one step.
    if [ "$((free_kb / 1024))" -lt 2048 ]; then
        emit store_space_ok no
        blocked=1
        note "less than 2 GiB free where the checkpoints go. A capture into a full disk publishes nothing, and a machine restored or resumed while the disk is full can be left with guest I/O errors that stop it from being deleted normally (references/traps.md)."
    else
        emit store_space_ok yes
    fi
fi

if [ "$blocked" -eq 0 ]; then emit result ready; else emit result blocked; fi
```

### `scripts/create-source.sh`

```bash
#!/usr/bin/env bash
# Create and start a machine to checkpoint and branch, with state in RAM and on
# disk that a restore or a child can be checked against.
#
# usage: create-source.sh [<name>] [--branch-ready]
#   <name>          default smolskill-src; must start with smolskill- so cleanup.sh deletes it
#   --branch-ready  the workload does its setup and then parks in smolvm-branch-ready,
#                   which a batch branch (branch-point.sh --count) needs
#
# The workload keeps a counter in /tmp, which is tmpfs and therefore RAM: a
# restore or a resume that brings the counter back proves memory came back, not
# only the disk. /root/setup.txt is the disk marker. The machine is started
# --branchable because on macOS a checkpoint is refused without it and the flag
# cannot be turned on for a running machine.

set -uo pipefail

here="$(cd "$(dirname "$0")" && pwd)"
NAME="smolskill-src"
READY=0
for a in "$@"; do
    case "$a" in
        --branch-ready) READY=1 ;;
        -*) printf 'unknown argument: %s\n' "$a" >&2; exit 2 ;;
        *) NAME="$a" ;;
    esac
done

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
    printf 'smolvm not found; set SMOLVM to its path\n' >&2
    exit 2
fi
case "$NAME" in
    smolskill-*) ;;
    *) printf 'name must start with smolskill- so cleanup.sh will delete it\n' >&2; exit 2 ;;
esac

counter='i=0; while true; do i=$((i+1)); echo $i > /tmp/ram_counter; sleep 1; done'
if [ "$READY" -eq 1 ]; then
    # setup, then park; each child continues into the program after --
    workload="echo SETUP_DONE > /root/setup.txt; echo \$RANDOM > /tmp/ram_token; exec smolvm-branch-ready -- sh -c 'echo CHILD=\$SMOLVM_BRANCH_NAME > /root/child.txt; $counter'"
else
    workload="echo SETUP_DONE > /root/setup.txt; echo \$RANDOM > /tmp/ram_token; $counter"
fi

"$SMOLVM" machine create --name "$NAME" --net --mem 1024 --image alpine -- sh -c "$workload" 2>&1 | sed 's/^/  /'
"$here/cleanup.sh" --record "$NAME"
"$SMOLVM" machine start --name "$NAME" --branchable 2>&1 | sed 's/^/  /'

# Wait for a value from the workload, not for the start command to return.
for i in $(seq 1 60); do
    v="$("$SMOLVM" machine exec --name "$NAME" -- cat /root/setup.txt 2>/dev/null)"
    if [ "$v" = "SETUP_DONE" ]; then
        printf 'workload_ready_after_s=%s\n' "$i"
        printf 'ram_token=%s\n' "$("$SMOLVM" machine exec --name "$NAME" -- cat /tmp/ram_token 2>/dev/null)"
        printf 'result=up\n'
        exit 0
    fi
    sleep 1
done
printf 'result=FAILED the workload never wrote /root/setup.txt\n'
exit 1
```

### `scripts/checkpoint.sh`

```bash
#!/usr/bin/env bash
# One capture of a running machine into a checkpoint store, under a unique name,
# then optional retention. This is the command a timer runs: smolvm has no
# checkpoint scheduler and no retention of its own.
#
# usage: checkpoint.sh --name <machine> --store <dir> [--out <dir>] [--label <word>]
#                      [--keep <K>] [--history <N>]
#   --store <dir>    the checkpoint store, created if missing
#   --out <dir>      where the checkpoint directories go (default: the store's
#                    parent); must be on the same filesystem as the store
#   --label <word>   goes into the output name (default: ckpt)
#   --keep <K>       keep the K newest (K at least 1) checkpoints with this label
#                    in --out, delete the older ones, then run checkpoint-prune
#   --history <N>    earlier generations each new checkpoint retains (smolvm's
#                    default is 32). Retention frees space only for generations
#                    no kept checkpoint still retains, so bound this too.
#
# What it enforces, each from the engine's own documentation of periodic use:
#   - one capture at a time per source: a run that finds another in progress
#     exits 3 without capturing
#   - a unique output each time: <name>-<label>-<UTC timestamp>.smolcheckpoint
#   - the previous checkpoint stays until the new one is published, and
#     "published" is read back from checkpoint-log, not taken from the exit code
#   - retention removes whole checkpoint directories, then prunes the store

set -uo pipefail

NAME=""
STORE=""
OUT=""
LABEL="ckpt"
KEEP=""
HISTORY=""

while [ $# -gt 0 ]; do
    case "$1" in
        --name)    NAME="$2"; shift ;;
        --store)   STORE="$2"; shift ;;
        --out)     OUT="$2"; shift ;;
        --label)   LABEL="$2"; shift ;;
        --keep)    KEEP="$2"; shift ;;
        --history) HISTORY="$2"; shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done

if [ -z "$NAME" ] || [ -z "$STORE" ]; then
    printf 'usage: checkpoint.sh --name <machine> --store <dir> [--out <dir>] [--keep <K>] [--history <N>]\n' >&2
    exit 2
fi
if [ -n "$KEEP" ]; then
    case "$KEEP" in
        *[!0-9]*) printf 'result=FAILED --keep takes a whole number of at least 1, not %s\n' "$KEEP"; exit 2 ;;
    esac
    [ "$KEEP" -ge 1 ] || { printf 'result=FAILED --keep takes a whole number of at least 1, not %s\n' "$KEEP"; exit 2; }
fi
STAMP_GLOB='[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]T[0-9][0-9][0-9][0-9][0-9][0-9]Z'

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
    printf 'smolvm not found; set SMOLVM to its path\n' >&2
    exit 2
fi

mkdir -p "$STORE"
STORE="$(cd "$STORE" && pwd)"
[ -n "$OUT" ] || OUT="$(dirname "$STORE")"
mkdir -p "$OUT"
OUT="$(cd "$OUT" && pwd)"

# One capture at a time per source. mkdir is atomic on every filesystem this
# runs on, and macOS has no flock(1). A lock whose owner is gone is taken over.
LOCK="$STORE/.smolskill-capture-$NAME.lock"
if ! mkdir "$LOCK" 2>/dev/null; then
    owner="$(cat "$LOCK/pid" 2>/dev/null)"
    if [ -n "$owner" ] && kill -0 "$owner" 2>/dev/null; then
        printf 'result=skipped\n'
        printf 'note=a capture of %s is still running (pid %s). Run one capture at a time per source: overlapping captures compete for memory and I/O and finish no sooner. Lengthen the interval.\n' "$NAME" "$owner"
        exit 3
    fi
    rm -rf "$LOCK"
    mkdir "$LOCK" || { printf 'result=FAILED could not take the capture lock %s\n' "$LOCK"; exit 1; }
fi
printf '%s\n' "$$" > "$LOCK/pid"
trap 'rm -rf "$LOCK"' EXIT

stamp="$(date -u +%Y%m%dT%H%M%SZ)"
output="$OUT/$NAME-$LABEL-$stamp.smolcheckpoint"
if [ -e "$output" ]; then
    printf 'result=FAILED %s already exists; smolvm never overwrites an output, and two captures in one second collide on this name\n' "$output"
    exit 1
fi

hist=()
[ -n "$HISTORY" ] && hist=(--history "$HISTORY")

start="$(date +%s)"
out="$("$SMOLVM" machine checkpoint --name "$NAME" --store "$STORE" --output "$output" ${hist[@]+"${hist[@]}"} 2>&1)"
rc=$?
end="$(date +%s)"
printf '%s\n' "$out" | sed 's/^/  /'
printf 'output=%s\n' "$output"
printf 'capture_s=%s\n' "$((end - start))"

# The exit code is not the assertion. A checkpoint is usable once the store
# lists it as its own generation, so read that back.
log="$("$SMOLVM" machine checkpoint-log "$output" 2>&1)"
if [ "$rc" -ne 0 ] || ! printf '%s' "$log" | grep -q '(this checkpoint)'; then
    printf 'published=no\n'
    printf 'result=FAILED the capture did not publish; the previous checkpoint is untouched\n'
    case "$out" in
        *"no file-backed regions"*|*"requires file-backed guest RAM"*)
            printf 'note=on macOS a stored checkpoint needs the machine started with machine start --branchable, and this one was not. The flag cannot be turned on for a running machine.\n' ;;
        *"No space left"*)
            printf 'note=the disk is full. A failed capture publishes nothing, so delete old checkpoints and run machine checkpoint-prune before the next attempt.\n' ;;
    esac
    exit 1
fi
printf 'published=yes\n'
printf 'generation=%s\n' "$(printf '%s\n' "$log" | awk '$1=="~0"{print $2}')"
printf 'generations_retained=%s\n' "$(printf '%s\n' "$log" | grep -c '^~')"

# Retention: only after the new checkpoint is published, and by whole
# directory. Output names sort by time because the stamp is UTC and fixed width,
# and a glob expands sorted and unsplit, so the oldest come first and a space in
# the path stays inside one name. Matching the stamp's shape keeps another label
# that starts with this one, such as ckpt-daily, out of the count.
if [ -n "$KEEP" ]; then
    set -- "$OUT/$NAME-$LABEL"-$STAMP_GLOB.smolcheckpoint
    [ -e "$1" ] || set --
    while [ "$#" -gt "$KEEP" ]; do
        rm -rf -- "$1" && printf 'deleted=%s\n' "$1"
        shift
    done
    "$SMOLVM" machine checkpoint-prune --store "$STORE" 2>&1 | sed 's/^/  /'
    set -- "$OUT/$NAME-$LABEL"-$STAMP_GLOB.smolcheckpoint
    [ -e "$1" ] || set --
    printf 'kept=%s\n' "$#"
fi

printf 'store_size=%s\n' "$(du -sh "$STORE" 2>/dev/null | cut -f1)"
printf 'result=captured\n'
```

### `scripts/schedule.sh`

```bash
#!/usr/bin/env bash
# Run checkpoint.sh on an interval, in the foreground, and say whether the
# interval is long enough. For a test or a session an agent is watching; for a
# schedule that outlives the session, put checkpoint.sh under cron, a systemd
# timer or launchd instead (references/scheduling.md).
#
# usage: schedule.sh --name <machine> --store <dir> --every <seconds> --times <N>
#                    [--keep <K>] [--history <N>] [--out <dir>]

set -uo pipefail

here="$(cd "$(dirname "$0")" && pwd)"
EVERY=""
TIMES=""
pass=()
while [ $# -gt 0 ]; do
    case "$1" in
        --every) EVERY="$2"; shift ;;
        --times) TIMES="$2"; shift ;;
        --name|--store|--keep|--history|--out|--label) pass+=("$1" "$2"); shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done
if [ -z "$EVERY" ] || [ -z "$TIMES" ]; then
    printf 'usage: schedule.sh --name <machine> --store <dir> --every <seconds> --times <N> [--keep <K>]\n' >&2
    exit 2
fi

ok=0
failed=0
longest=0
for run in $(seq 1 "$TIMES"); do
    start="$(date +%s)"
    out="$("$here/checkpoint.sh" ${pass[@]+"${pass[@]}"} 2>&1)"
    rc=$?
    took=$(( $(date +%s) - start ))
    [ "$took" -gt "$longest" ] && longest="$took"
    printf 'run=%s rc=%s took_s=%s %s\n' "$run" "$rc" "$took" "$(printf '%s\n' "$out" | grep -E '^(output|result|kept|store_size)=' | tr '\n' ' ')"
    if [ "$rc" -eq 0 ]; then ok=$((ok + 1)); else failed=$((failed + 1)); printf '%s\n' "$out" | sed 's/^/  /'; fi
    if [ "$took" -ge "$EVERY" ]; then
        printf 'note=this capture took %ss and the interval is %ss. Set the interval from the complete capture time, not the source pause.\n' "$took" "$EVERY"
    fi
    [ "$run" -lt "$TIMES" ] && sleep $(( EVERY > took ? EVERY - took : 0 ))
done
printf 'captures_ok=%s\n' "$ok"
printf 'captures_failed=%s\n' "$failed"
printf 'longest_capture_s=%s\n' "$longest"
if [ "$failed" -eq 0 ]; then printf 'result=schedule_ok\n'; else printf 'result=FAILED\n'; exit 1; fi
```

### `scripts/restore.sh`

```bash
#!/usr/bin/env bash
# Restore a checkpoint, or an earlier generation it retains, into a new machine
# and start it.
#
# usage: restore.sh --from <checkpoint> --name <new> [--at <~N|id>]
#
# The restore path is `machine create --from`; there is no `machine restore`.
# The new machine is started --branchable so it can itself be checkpointed,
# paused and branched on every host.

set -uo pipefail

here="$(cd "$(dirname "$0")" && pwd)"
FROM=""
NAME=""
AT=""
while [ $# -gt 0 ]; do
    case "$1" in
        --from) FROM="$2"; shift ;;
        --name) NAME="$2"; shift ;;
        --at)   AT="$2"; shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done
if [ -z "$FROM" ] || [ -z "$NAME" ]; then
    printf 'usage: restore.sh --from <checkpoint> --name <new> [--at <~N|id>]\n' >&2
    exit 2
fi
case "$NAME" in smolskill-*) ;; *) printf 'name must start with smolskill- so cleanup.sh will delete it\n' >&2; exit 2 ;; esac

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
[ -n "$SMOLVM" ] || { printf 'smolvm not found; set SMOLVM to its path\n' >&2; exit 2; }

at=()
[ -n "$AT" ] && at=(--at "$AT")
# create refuses a name that exists, and the listing below would then find the
# old machine, so the create's own status decides; pipefail carries it.
if ! "$SMOLVM" machine create --name "$NAME" --from "$FROM" ${at[@]+"${at[@]}"} 2>&1 | sed 's/^/  /'; then
    printf 'result=FAILED machine create --from did not succeed, so nothing was restored\n'
    exit 1
fi
# Read the list first: under pipefail, grep -q closing the pipe early can fail
# the pipeline and report a machine missing that exists.
listing="$("$SMOLVM" machine list 2>/dev/null)"
if ! printf '%s\n' "$listing" | grep -q "^$NAME "; then
    printf 'result=FAILED no machine was created\n'
    exit 1
fi
"$here/cleanup.sh" --record "$NAME"
start="$(date +%s)"
"$SMOLVM" machine start --name "$NAME" --branchable 2>&1 | sed 's/^/  /'
printf 'start_s=%s\n' "$(( $(date +%s) - start ))"
state="$("$SMOLVM" machine list 2>/dev/null | awk -v n="$NAME" '$1==n{print $2}')"
printf 'state=%s\n' "$state"
[ "$state" = running ] && printf 'result=restored\n' || { printf 'result=FAILED\n'; exit 1; }
```

### `scripts/branch-point.sh`

```bash
#!/usr/bin/env bash
# Branch a machine and take a checkpoint of the same point, so the state the
# children started from outlives them. A branch's captured generation lives on
# this host only; the checkpoint is the copy you can keep, export and restore.
#
# usage: branch-point.sh --from <source> --store <dir> --count <N> --name-prefix <prefix>
#        branch-point.sh --from <source> --store <dir> --name <child>
#   --count/--name-prefix  a batch. The source's workload must park in
#                          smolvm-branch-ready; the checkpoint is taken while it
#                          is parked, so it is the exact state every child starts from.
#   --name                 one child. The checkpoint is taken first and the
#                          branch follows, a second or two later, so the two
#                          points differ by whatever the source did in between.

set -uo pipefail

here="$(cd "$(dirname "$0")" && pwd)"
FROM=""
STORE=""
COUNT=""
PREFIX=""
CHILD=""
while [ $# -gt 0 ]; do
    case "$1" in
        --from)        FROM="$2"; shift ;;
        --store)       STORE="$2"; shift ;;
        --count)       COUNT="$2"; shift ;;
        --name-prefix) PREFIX="$2"; shift ;;
        --name)        CHILD="$2"; shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done
if [ -z "$FROM" ] || [ -z "$STORE" ] || { [ -z "$CHILD" ] && { [ -z "$COUNT" ] || [ -z "$PREFIX" ]; }; }; then
    printf 'usage: branch-point.sh --from <source> --store <dir> (--count <N> --name-prefix <p> | --name <child>)\n' >&2
    exit 2
fi
case "${PREFIX}${CHILD}" in
    smolskill-*) ;;
    *) printf 'children must be named smolskill-... so cleanup.sh deletes them\n' >&2; exit 2 ;;
esac

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
[ -n "$SMOLVM" ] || { printf 'smolvm not found; set SMOLVM to its path\n' >&2; exit 2; }

if [ -n "$COUNT" ]; then
    # Parked means a process's command line starts with the helper. pgrep -f is
    # not a test for it: it matches the sh -c that runs the check, whose text
    # names the helper. From v1.25.4 the workload runs under smolvm's own init
    # rather than as PID 1, so read every process.
    parked=no
    for i in $(seq 1 120); do
        if "$SMOLVM" machine exec --name "$FROM" -- sh -c 'for f in /proc/[0-9]*/cmdline; do tr "\0" " " < "$f"; echo; done' 2>/dev/null | grep -q '^smolvm-branch-ready'; then
            parked=yes; printf 'source_parked_after_s=%s\n' "$i"; break
        fi
        sleep 1
    done
    if [ "$parked" != yes ]; then
        printf 'result=FAILED the source never parked in smolvm-branch-ready, so a batch branch would wait for a branch point that is not coming. Its workload has to run smolvm-branch-ready -- <program> after its setup (create-source.sh --branch-ready does), or branch one child with --name.\n'
        exit 1
    fi
fi

ck="$("$here/checkpoint.sh" --name "$FROM" --store "$STORE" --label branchpoint 2>&1)"
printf '%s\n' "$ck" | sed 's/^/  /'
checkpoint="$(printf '%s\n' "$ck" | sed -n 's/^output=//p')"
if ! printf '%s' "$ck" | grep -q '^result=captured'; then
    printf 'result=FAILED no checkpoint of the branch point, so no branch was taken\n'
    exit 1
fi
printf 'checkpoint=%s\n' "$checkpoint"

if [ -n "$COUNT" ]; then
    out="$("$SMOLVM" machine branch --from "$FROM" --count "$COUNT" --name-prefix "$PREFIX" 2>&1)"
    children="$(seq 0 $((COUNT - 1)) | sed "s/^/$PREFIX-/")"
else
    out="$("$SMOLVM" machine branch --from "$FROM" --name "$CHILD" 2>&1)"
    children="$CHILD"
fi
printf '%s\n' "$out" | sed 's/^/  /'
made=0
# Read the list once: under pipefail, grep -q closing the pipe early can fail
# the pipeline and report a child missing that exists.
listing="$("$SMOLVM" machine list 2>/dev/null)"
for c in $children; do
    "$here/cleanup.sh" --record "$c"
    if printf '%s\n' "$listing" | grep -q "^$c "; then made=$((made + 1)); printf 'child=%s\n' "$c"; fi
done
# The source is frozen on some hosts after a branch and running on others; read
# which, rather than assuming.
printf 'source_state=%s\n' "$(printf '%s\n' "$listing" | awk -v n="$FROM" '$1==n{print $2}')"
if [ "$made" -eq "$(printf '%s\n' "$children" | grep -c .)" ]; then
    printf 'result=branched\n'
else
    printf 'result=FAILED %s of the children exist\n' "$made"
    exit 1
fi
```

### `scripts/cleanup.sh`

```bash
#!/usr/bin/env bash
# Delete the machines this packet's scripts created, then prove the host is clean.
#
# Only machines recorded in the state file are deleted, with --cascade, so any
# machine branched from one of them goes too, whatever its name. Any other
# machine you or another session created by hand is never deleted. Scripts
# record a name by calling: cleanup.sh --record <name>
#
# usage: cleanup.sh [--record <name>] [--reap] [--purge] [--restore-base] [--checkpoints <dir>...]
#   --record <name>       add a machine name to the state file and exit
#   --reap                kill every VM process under this HOME; run without it first
#   --purge               also remove the state file once the list is empty
#   --restore-base        also remove smolvm's copies of recently restored checkpoints
#   --checkpoints <dir>   also remove these checkpoint directories and stores; last
#
# Checkpoints hold the machine's memory and disks, so they are sensitive, and
# smolvm keeps copies of its own: vms/_restore-base is a clone of the most
# recently restored checkpoint, and from v1.22.0 vms/_restore-checkpoints holds
# the last three restored, both kept to make the next restore cheaper, and both
# outlive every machine and every checkpoint file.

set -uo pipefail

PACKET="branch-and-checkpoint"
PREFIX="smolskill-"

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
STATE_DIR="${SMOLVM_SKILL_STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/smolvm-skills}"
STATE_FILE="$STATE_DIR/$PACKET.machines"

reap=0
purge=0
restore_base=0
while [ $# -gt 0 ]; do
    case "$1" in
        --record)
            mkdir -p "$STATE_DIR"
            printf '%s\n' "$2" >> "$STATE_FILE"
            exit 0
            ;;
        --reap)  reap=1 ;;
        --purge) purge=1 ;;
        --restore-base) restore_base=1 ;;
        --checkpoints) shift; break ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done

if [ -z "$SMOLVM" ]; then
    printf 'smolvm not found; set SMOLVM to its path\n' >&2
    exit 2
fi

# With SMOLVM_DATA_DIR set, smolvm runs with HOME there on Linux, so its cache
# is under .cache in it.
case "$(uname -s)" in
    Darwin) VMS_DIR="$HOME/Library/Caches/smolvm/vms" ;;
    *)      cache_root="${SMOLVM_DATA_DIR:+$SMOLVM_DATA_DIR/.cache}"
            VMS_DIR="${cache_root:-${XDG_CACHE_HOME:-$HOME/.cache}}/smolvm/vms" ;;
esac
VMS_DIR="${SMOLVM_VMS_DIR:-$VMS_DIR}"
SMOLVM_PREFIX="${SMOLVM_PREFIX:-$HOME/.smolvm}"
if [ -n "${SMOLVM:-}" ]; then
    case "$(readlink "$SMOLVM" 2>/dev/null || printf '%s' "$SMOLVM")" in
        "$SMOLVM_PREFIX"/*) ;;
        *) printf 'note=%s is not under %s, so the process scan cannot see its forked VMs (all of its VMs on macOS); set SMOLVM_PREFIX to its directory\n' "$SMOLVM" "$SMOLVM_PREFIX" ;;
    esac
fi

# List this HOME's VM processes as "pid marker". The plain run path execs a
# `_boot-vm` child that carries its boot config; the pack-run path forks one that
# carries none. Linux names both `libkrun VM`, or `VM:<hostname>` when HOSTNAME
# is exported; on macOS the executable path and the parent chain scope the
# search. The teardown packet's traps have the why.
list_vm_processes() {
    case "$(uname -s)" in
        Linux)
            for p in /proc/[0-9]*; do
                case "$(cat "$p/comm" 2>/dev/null)" in "libkrun VM"|VM:*) ;; *) continue ;; esac
                pid="${p#/proc/}"
                cfg="$(tr '\0' '\n' < "$p/cmdline" 2>/dev/null | sed -n '3p')"
                case "$cfg" in
                    "$VMS_DIR"/*) printf '%s %s\n' "$pid" "$cfg"; continue ;;
                esac
                case "$(readlink "$p/exe" 2>/dev/null)" in
                    "$SMOLVM_PREFIX"/*) printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
                esac
            done
            ;;
        Darwin)
            # shellcheck disable=SC2009  # pgrep -f would match this script.
            procs="$(ps -axo pid=,ppid=,command= 2>/dev/null)"
            printf '%s\n' "$procs" | while read -r pid ppid rest; do
                case "$rest" in "$SMOLVM_PREFIX"/smolvm-bin*) ;; *) continue ;; esac
                case "$rest" in
                    *" _boot-vm "*) printf '%s %s\n' "$pid" "${rest#* _boot-vm }"; continue ;;
                esac
                # An orphan counts only if it is a run; a fork has its parent's command line.
                if [ "$ppid" = 1 ]; then
                    case "$rest" in
                        *" machine run "*|*" vm run "*|*" pack run "*)
                            printf '%s orphaned-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
                    esac
                else
                    parent="$(printf '%s\n' "$procs" | while read -r q qp qrest; do [ "$q" = "$ppid" ] && { printf '%s' "$qrest"; break; }; done)"
                    [ "$parent" = "$rest" ] && printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX"
                fi
            done
            ;;
    esac
}

# 1. Delete recorded machines. Without --force a delete prompts and defaults to
# No; --cascade removes branch children, which otherwise block it.
if [ -s "$STATE_FILE" ]; then
    while read -r name; do
        [ -n "$name" ] || continue
        case "$name" in "$PREFIX"*) ;; *)
            printf 'skipping %s: not created by this packet (no %s prefix)\n' "$name" "$PREFIX"
            continue ;;
        esac
        # A child is gone once its source was deleted with --cascade, and on
        # v1.18.2 `machine stop` on a name that does not exist leaves an empty
        # directory under vms/. Act only on names still listed.
        "$SMOLVM" machine list </dev/null 2>/dev/null | awk 'NR>2{print $1}' > "$STATE_DIR/.listed" 2>/dev/null
        grep -qx -- "$name" "$STATE_DIR/.listed" || continue
        # Only names still listed: a second stop of a missing name leaves a
        # directory that reads as a leak. The list is read first because grep -q
        # under pipefail can fail the pipeline and skip a listed machine.
        listed="$("$SMOLVM" machine list </dev/null 2>/dev/null | awk 'NR>2{print $1}')"
        grep -qx -- "$name" <<<"$listed" || continue
        "$SMOLVM" machine stop   --name "$name" </dev/null >/dev/null 2>&1
        "$SMOLVM" machine delete --name "$name" --force --cascade </dev/null 2>&1 | sed 's/^/  /'
    done < "$STATE_FILE"
fi

# 1b. Checkpoints and stores named on the command line, then the restore base.
# A store is known by the objects directory and .lock its first capture makes, not by its name.
for d in "$@"; do
    case "$d" in
        *.smolcheckpoint|*.checkpoint) ;;
        *) if [ ! -d "$d/objects" ] || [ ! -e "$d/.lock" ]; then
               printf 'skipping %s: not a .checkpoint, a .smolcheckpoint or a checkpoint store\n' "$d"; continue
           fi ;;
    esac
    [ -e "$d" ] && rm -rf -- "$d" && printf 'removed=%s\n' "$d"
done
for cache in _restore-base _restore-checkpoints; do
    if [ "$restore_base" -eq 1 ] && [ -d "$VMS_DIR/$cache" ]; then
        rm -rf "${VMS_DIR:?}/$cache" "$VMS_DIR/$cache.lock" && printf 'removed=%s\n' "$VMS_DIR/$cache"
    elif [ "$restore_base" -eq 1 ]; then
        printf '%s=absent (nothing to remove)\n' "$cache"
    elif [ -d "$VMS_DIR/$cache" ]; then
        printf '%s=present %s (restored checkpoints kept by smolvm; --restore-base removes it)\n' "$cache" "$(du -sh "$VMS_DIR/$cache" 2>/dev/null | cut -f1)"
    fi
done

rm -f "$STATE_DIR/.listed"

# 2. An ephemeral machine's entry retires after its run returns: poll, 20 s at most.
printf 'waiting=up to 20s for ephemeral entries to retire before asserting\n'
waited=0
listing="$("$SMOLVM" machine list 2>&1)"
while ! grep -q 'No machines found' <<<"$listing" && [ "$waited" -lt 20 ]; do
    sleep 1; waited=$((waited + 1))
    listing="$("$SMOLVM" machine list 2>&1)"
done
printf 'waited=%ss\n' "$waited"

# 3. Assert the value, not the exit code.
if grep -q 'No machines found' <<<"$listing"; then
    printf 'machines=clean\n'
    [ "$purge" -eq 1 ] && rm -f "$STATE_FILE"
else
    printf 'machines=remaining\n'
    printf '%s\n' "$listing" | sed 's/^/  /'
    printf 'note=this packet did not create these, so it will not delete them. By name:\n'
    printf '  smolvm machine stop --name <NAME> && smolvm machine delete --name <NAME> --force\n'
    printf '  add --cascade for a machine that was branched from another\n'
fi

# 4. Report VM processes under this HOME's state, such as a killed wrapper's.
# With --reap every one is killed, including a machine another packet kept.
found=0
while read -r pid cfg; do
    [ -n "$pid" ] || continue
    found=1
    printf 'vm_process=%s config=%s\n' "$pid" "$cfg"
    if [ "$reap" -eq 1 ]; then
        kill -9 "$pid" 2>/dev/null && printf '  killed %s\n' "$pid"
    fi
done <<EOF
$(list_vm_processes)
EOF

if [ "$found" -eq 0 ]; then
    printf 'vm_processes=none\n'
elif [ "$reap" -eq 0 ]; then
    printf 'rerun with --reap to kill them\n'
fi
```

## Checkpoint, restore, pause and branch traps

Each entry was hit on v1.18.2 on 2026-09-24 unless it says otherwise.

### `--branchable` is decided at start, and older macOS releases needed it for more than branching

`machine branch` against a machine started without it, on either host:

```
Error: agent operation failed: fork: machine 'smolskill-sfx' was not started as branchable, so it
has no copy-on-write memory to branch from. Restart it with `smolvm machine start --name
smolskill-sfx --branchable`; branchability is decided at start time and cannot be turned on for an
already-running machine.
```

That message is clear. The macOS one for `checkpoint` and `pause` before v1.20.0 is not:

```
Error: agent operation failed: checkpoint machine: libkrun save failed: ERR EIO capture VM: VM
snapshot/restore failed: retain COW guest-memory generation: guest RAM has no file-backed regions
```

`pause` prints the same text, because a pause is a checkpoint underneath, and the machine keeps
running. On Linux aarch64 both succeeded without the flag: a checkpoint of 55 MiB with a 1.242 s
pause, and a pause and resume. On those releases, start every machine you might checkpoint on macOS
with `--branchable`.

**From v1.20.0 macOS takes a checkpoint file and a pause without the flag**, measured on v1.20.0,
v1.20.2, v1.21.1, v1.22.0 and v1.22.2, where v1.19.0 and v1.19.3 still refused. **A stored capture
still needs it.** On v1.22.2 `--store` against a machine started without it fails with:

```
Error: agent operation failed: checkpoint machine: libkrun save failed: ERR ENOTSUP VM
snapshot/restore failed: retain COW guest-memory generation: deferred durable save requires
file-backed guest RAM
```

**On v1.31.1 neither host needs it to save a machine**, measured on macOS and Linux x86_64 on
2026-10-22: a machine started without `--branchable` took a checkpoint file, a stored capture
(`--store`, the second one incremental) and a pause and resume, and a process it was running kept
its PID through each. Only `machine branch` still needs the flag.

### `--output` must end in `.checkpoint` or `.smolcheckpoint`, even when it is a directory

From v1.19.1 the file is a `.checkpoint` in `machine checkpoint --help`, and on v1.22.2 any other
name fails with `output must end in .checkpoint`, while `.smolcheckpoint` is still accepted and
`machine create --help` still names it. The scripts use `.smolcheckpoint` so they run on older
releases too. On v1.18.2:

```
Error: config operation failed: checkpoint machine: output must end in .smolcheckpoint
```

for `-o ./x.checkpoint`, and the same for `--store ./st -o ./inc1`, where the output is a directory
and the help calls it one.

### The exit code of a capture is not the proof

A capture is usable once it has been durably published, which `checkpoint-log <output>` shows as the
`~0` line ending `(this checkpoint)`. `scripts/checkpoint.sh` reads that back before it deletes any
older checkpoint. During this run the host disk filled and an export failed with `tar error: No
space left on device (os error 28)`; a scheduler that trusted a zero exit elsewhere and deleted the
previous checkpoint would have been left with none.

### `--keep` does not bound the store on its own

Each stored checkpoint retains up to `--history` earlier generations, 32 by default, and deleting
the directory of a generation that a kept checkpoint retains frees nothing. With `--keep 2
--history 2` the store grew 59, 76, 95, 114 MB over four captures. `references/scheduling.md` has
how to pick the two together.

### A restored branch point is not a parked source

A checkpoint taken while the source is parked in `smolvm-branch-ready` holds exactly the state its
children start from. Restoring it does not give you a parked source back: on start the helper
releases, and the workload runs the program after `--` with an empty `SMOLVM_BRANCH_NAME`, so
`/root/child.txt` in that test read `CHILD=`. A batch branch from the restored machine then waits
its full `--ready-timeout`, 10 minutes by default, and fails with

```
A batch branch checkpoints the source at a point its workload declares by running
`smolvm-branch-ready` after setup ... either add the call, raise --ready-timeout, or take single
`--name` branches, which checkpoint the source wherever it is.
```

A single `--name` branch from the restored machine worked and carried the source's state. To fan out
again from a kept branch point, branch it one child at a time or restore it once per worker, and
give each its identity some other way, since the branch variables are empty.

### Detecting a parked source: `pgrep -f` always matches

`smolvm machine exec --name src -- sh -c 'pgrep -f smolvm-branch-ready'` succeeds whether or not the
source is parked, because the `sh -c` running it has the string in its own command line. Read PID 1
instead, which is the workload:

```bash
smolvm machine exec --name src -- sh -c 'tr "\0" " " < /proc/1/cmdline' | grep -q '^smolvm-branch-ready'
```

A parked source shows `smolvm-branch-ready -- sh -c ...`; a released one shows the program after
`--`.

### `machine list | grep -q` under `pipefail`

`set -o pipefail` plus `smolvm machine list | grep -q name` can fail for a machine that exists:
`grep -q` exits on the first match, the list's next write hits a closed pipe, and the pipeline's
status is the list's failure. Two scripts in this packet reported `result=FAILED no machine was
created` for machines that were running before this was caught. Read the list into a variable, then
search it.

### A paused machine refuses `stop` and `start`, and a failed resume can be retried

```
Error: agent operation failed: stop: machine has saved execution; use resume or delete
Error: agent operation failed: start: machine has saved execution; use resume
```

A restored machine paused while the host disk was full failed its resume with `extract paused
checkpoint: failed to unpack ...`. After space was freed the same `machine resume` succeeded, and
the counter the workload keeps in RAM read 14 against 8 before the pause, then kept counting.

### A full disk leaves machines that `delete --force` will not remove

A machine restored and started while the disk was full came up with its overlay failing. Its
delete, later, with space available:

```
Error: agent operation failed: stop agent: guest did not confirm filesystem synchronization; left
the VM alive for retry: agent operation failed: shutdown ack: freeze /oldroot/mnt/overlay: I/O error
(os error 5)
```

exit 1, still `running`. `scripts/cleanup.sh --reap` killed its VM process, after which `delete
--force` removed it. **`--reap` kills every VM process under this `HOME`**, not only that one, so
use it when nothing else there should keep running. `scripts/preflight.sh` checks free space for
this reason.

### `du` overstates what a restore costs

Restores are copy-on-write over the checkpoint. On macOS three restores of one checkpoint moved the
volume's free space by 22 MB at create and 65 MB at start, about 29 MB a machine, while `du` on
each machine's directory said 150 to 190 MB. On Linux each restored machine's directory holds small
`qcow2` layers over `.smolcheckpoint-*.raw` bases, 29 MB by `du`. Measure a restore budget with
`df` before and after.

### smolvm keeps restored checkpoints after you delete them

On macOS `vms/_restore-base` under smolvm's cache is, in the source's words, a pristine clone of the
most recently restored checkpoint, kept so the next restore writes only the chunks that differ. It
was 243 MB here, with the checkpoint's `memory.bin` inside, and it stayed after every machine and
every checkpoint file was deleted. `smolvm serve start`'s reclaim did not touch it.
`scripts/cleanup.sh --restore-base` removes it. None was created on Linux aarch64.

From v1.22.0 the cache is `vms/_restore-checkpoints`, which keeps the last three restored
checkpoints, up to 16 GiB, so jumping back to one clones it instead of rebuilding its RAM. After
this packet's procedure on macOS on v1.23.0 it held 258 MB with every machine and checkpoint
deleted, and no `_restore-base` was made. `machine create --from` takes `--restore-cache-entries`
and `--restore-cache-gib`; `0` entries turns it off. `scripts/cleanup.sh --restore-base` removes
both.

### `machine stop` on a missing name leaves an empty directory

On v1.18.2, `smolvm machine stop --name <name>` for a machine that does not exist leaves an empty
directory under the VM cache. A cleanup that stops each recorded name after `--cascade` has already
deleted the children leaves one per child, which a leak check counting directories then reports.
This packet's cleanup stops only names still listed.

### Environment given to a restore reaches `exec`, not the resumed workload

`machine create --from <checkpoint> -e ROLE=seven`: `machine exec` saw `ROLE=seven`, and the resumed
workload's own environment did not have it. The workload is the process that was running when the
checkpoint was taken, and its environment came with it.

### Corrections

- **2026-10-15, v1.29.0 run.** "Read PID 1 instead, which is the workload" above stopped holding
  in v1.25.4 (#1632): PID 1 is now `/run/smolvm/init container-init` and the workload is its child,
  so the check never saw a parked source and `branch-point.sh --count` failed with `the source
  never parked`. Read every process instead:

  ```bash
  smolvm machine exec --name src -- sh -c 'for f in /proc/[0-9]*/cmdline; do tr "\0" " " < "$f"; echo; done' | grep -q '^smolvm-branch-ready'
  ```

  `scripts/branch-point.sh` does this, and on v1.29.0 a batch of two then branched with the same
  RAM token in both children and in a restore of the branch point.