Branch and checkpoint: scheduled checkpoints, restores and branch points

Saves a running smolvm machine as checkpoints on a schedule, keeps a bounded history, restores any generation in it, pauses and resumes machines without losing running processes, and branches a warm machine into children with a checkpoint of the exact point they started from. Use when a machine’s state has to survive the machine; when an agent or a job needs to roll back; when checkpoints must be taken periodically, since smolvm has no scheduler or retention of its own; when fanning a prepared machine out into workers; or when a checkpoint, a pause or a branch fails with an error that names none of its preconditions. Do not use it to ship an environment to another host as a file, which is the pack packet, or for state that only has to survive a stop and start, which is the dev-env packet.

Verified on smolvm v1.34.0 on macOS arm64 and Linux x86_64, 2026-11-09, and on v1.18.2 on Linux aarch64, 2026-09-24. Done means a restore brings back both the disk and the memory you left, the schedule keeps exactly the checkpoints you asked for and no fewer, and every child of a branch starts from a state you still hold as a checkpoint after the children are gone. The Linux aarch64 runs used the scripts of their date; this version’s preflight and cleanup scripts ran on Linux aarch64 on v1.22.2 on 2026-10-03.

README.md says what each of these operations is. This file is how to run them, and the traps.

Three facts shape everything here.

  • smolvm has no scheduler and no retention. scripts/checkpoint.sh is the one capture a timer runs, with the rules the engine asks a scheduler to follow built in; references/scheduling.md puts it under cron, systemd or launchd.
  • A branch’s captured state stays on this host and cannot be exported. So a branch point is paired with a checkpoint taken at the same moment, which is the copy you keep.
  • --branchable at start decides what a machine can do later. A branch source needs it on every host, and on macOS before v1.24.0 so does a stored capture, which is what the scripts here take; Linux checkpoints and pauses without it. README.md has the errors when it is missing. The scripts always start with it.

Procedure

- [ ] 1. preflight.sh               read result= and store_space_ok=
- [ ] 2. create-source.sh           a machine with state in RAM and on disk
- [ ] 3. checkpoint.sh / schedule.sh   captures, with retention
- [ ] 4. restore.sh                 any generation, into a new machine
- [ ] 5. pause and resume           the restored machine, or any running one
- [ ] 6. branch-point.sh            children plus a checkpoint of where they started
- [ ] 7. cleanup.sh                 machines, checkpoints, and smolvm's restore cache

1. Preflight. Read-only.

scripts/preflight.sh --store ./store

It checks that this binary has --store, --history, checkpoint-log, pause and restore --at, says whether this host needs --branchable, and reports scheduler=builtin_none and retention=builtin_none so nobody goes looking for them. store_space_ok=no blocks: a capture into a full disk publishes nothing, and a machine restored while the disk was full was left with guest I/O errors that stopped it being deleted normally.

2. A source with state you can check.

scripts/create-source.sh smolskill-src                 # for checkpoints and restores
scripts/create-source.sh smolskill-bp --branch-ready   # for a batch branch

The workload writes a counter into /dev/shm every second, which is tmpfs and so RAM, and a marker to /root/setup.txt on disk. A restore that brings the counter back proves memory came back, not only the disk. --branch-ready makes the workload park in smolvm-branch-ready after its setup, which a batch branch waits for.

Or the user’s own machine. Name it with --name in the steps below. On macOS before v1.24.0, if it was started without --branchable, the first capture fails with deferred durable save requires file-backed guest RAM (v1.20.0 to v1.23.7) or guest RAM has no file-backed regions (before v1.20.0); from v1.24.0 the capture falls back to a synchronous save (#1584) and works. On the older releases, and for a branch on any release, the only fix is a restart with the flag:

smolvm machine stop  --name worker
smolvm machine start --name worker --branchable

That restart is a normal stop and start: disk state survives, and anything in RAM or /tmp is lost, once. Say so to the user before doing it. Every start of that machine needs the flag from then on for it to stay checkpointable.

A capture also refuses a machine with resources bound to this host: a host mount (-v), a published socket, a remote volume, a host secret reference, custom DNS, a named network, a GPU, CUDA, Rosetta, --ssh-agent or a Docker socket. The error, the initial portable checkpoint profile does not support ...; stop or detach these resources before capture, names which. From v1.29.0 machine checkpoint --host-mounts captures a machine with host mounts, and the checkpoint restores only on a host where the same paths exist.

3. Capture, once or on a schedule.

scripts/checkpoint.sh --name smolskill-src --store ./store --keep 6 --history 2
scripts/schedule.sh   --name smolskill-src --store ./store --every 10 --times 4 --keep 2 --history 2

Each capture goes to a new directory named <machine>-<label>-<UTC time>.smolcheckpoint beside the store, or in the directory --out <dir> names, which has to be on the same filesystem as the store. It is judged published only when checkpoint-log lists it as (this checkpoint), and only then are the oldest beyond --keep deleted and the store pruned. A capture started while another of the same machine runs exits 3 with result=skipped. schedule.sh is for a session you are watching; a schedule that outlives it is checkpoint.sh under a system timer, references/scheduling.md.

run=1 rc=0 took_s=2 ... kept=1 store_size= 59M result=captured
run=2 rc=0 took_s=5 ... kept=2 store_size= 76M result=captured
run=3 rc=0 took_s=2 ... kept=2 store_size= 95M result=captured
run=4 rc=0 took_s=1 ... kept=2 store_size=114M result=captured
captures_ok=4
captures_failed=0
longest_capture_s=5
result=schedule_ok

That is the v1.18.2 run. Ten seconds is for watching it work; a schedule uses minutes, references/scheduling.md.

Set the interval from longest_capture_s, not from the pause. The source pauses for a few hundredths of a second; the command takes seconds, because retained RAM is hashed and compressed after the source resumes. --keep alone does not bound the disk: every kept checkpoint also retains --history earlier generations, 32 by default, so the store above kept growing with two directories kept. Bound both.

4. Restore, any generation.

smolvm machine checkpoint-log ./smolskill-src-ckpt-<time>.smolcheckpoint
scripts/restore.sh --from <checkpoint> --name smolskill-old --at '~2'
scripts/restore.sh --from <checkpoint> --name smolskill-new

~0 is the checkpoint itself and ~N is N generations back along its history. restore.sh takes only a name starting with smolskill-, so cleanup can find it, and refuses any other with name must start with smolskill- so cleanup.sh will delete it; to restore under a name the user chooses, run smolvm machine create --name <name> --from <checkpoint> --at '~N' and then smolvm machine start --name <name> --branchable yourself. The restore path is machine create --from; there is no machine restore. A restored machine takes its own name as its hostname, and its disks are copy-on-write over what the checkpoint holds, so a restore costs about 29 MB of real disk on macOS for a 243 MiB checkpoint while du reports five times that.

5. Pause and resume.

smolvm machine pause  --name smolskill-new
smolvm machine resume --name smolskill-new

Pause saves RAM, disks and the running execution and stops the machine; resume brings the same execution back under the same name. Check a value the workload holds in memory: on both hosts the counter read 31 before the pause and 34 and 33 after the resume, so it continued where it stopped and did not run while paused. A paused machine refuses stop and start; resume it or delete it. A resume that fails keeps the saved state, and a later resume of the same machine succeeded.

6. Branch, and keep the branch point.

scripts/branch-point.sh --from smolskill-bp --store ./store --count 2 --name-prefix smolskill-w
scripts/branch-point.sh --from smolskill-src --store ./store --name smolskill-one

For a batch it waits until a process in the source runs smolvm-branch-ready, takes the checkpoint while the source is parked there, then branches. Every child and that checkpoint then hold the same state: a random value the source wrote to RAM before parking came back identical in both children and in a restore of the checkpoint.

source_parked_after_s=1
checkpoint=.../smolskill-bp-branchpoint-<time>.smolcheckpoint
child=smolskill-w-0
child=smolskill-w-1
source_state=running
result=branched

For one child with --name the checkpoint is taken first and the branch a second or two later, so the two points differ by whatever the source did in between.

A restored branch point is not a parked source. On start the helper releases and the workload runs the child program with an empty SMOLVM_BRANCH_NAME, so a batch branch from it waits for a branch point that never comes. Branch it with --name, one child at a time, or restore it once per worker; either way the child gets the state the original children started from.

7. Clean up.

scripts/cleanup.sh --purge --restore-base --checkpoints ./store ./*.smolcheckpoint

It deletes the machines the scripts recorded, children first by --cascade, removes the named checkpoint directories and stores, and with --restore-base removes smolvm’s copies of restored checkpoints, which otherwise stay after every machine and checkpoint is gone, printing a removed= or absent line for each. It waits up to 20 seconds, polling the machine list, and says so, because an ephemeral entry retires after its run returns.

Traps

Full detail in references/traps.md. The ones that cost the most:

  • --output must end in .checkpoint or .smolcheckpoint, with --store too, where it names a directory. .checkpoint is the name from v1.19.1, and the scripts use .smolcheckpoint, which every release accepts.
  • pgrep -f smolvm-branch-ready inside the guest always matches, because the sh -c running it contains the string. Read each /proc/<pid>/cmdline: from v1.25.4 the workload is not PID 1.
  • smolvm machine list | grep -q under pipefail can report a machine missing that exists: grep -q closes the pipe early and the failed write fails the pipeline. Read the list into a variable first.
  • machine stop --name on a name that does not exist leaves an empty directory under the VM cache on v1.18.2, and a second stop of it on v1.22.2 leaves the name inside, which reads as a leak. cleanup.sh stops only names still listed.
  • smolvm keeps restored checkpoints, memory included, after the checkpoint is deleted (_restore-base was 243 MB on macOS on v1.18.2); README.md has the rest, and step 7’s --restore-base removes them.

Security defaults, and why they are the defaults

  • A checkpoint is the machine’s memory and disks. Anything the workload held, secrets included, is in it; keep stores on a disk only you can read, and remove _restore-base and _restore-checkpoints when you remove the checkpoints.
  • Retention deletes only after the new checkpoint is published, so a failed capture never leaves you with fewer good checkpoints than before.
  • A store is local. It protects against a bad change, not a lost host; export a checkpoint with machine checkpoint --export-from and copy it elsewhere for that.
  • Cleanup deletes only machines the scripts recorded under the smolskill- prefix, and through --cascade any machine branched from one of them, whatever its name, and removes only the checkpoint paths you name.
  • Nothing here escalates privilege or edits smolvm configuration.

Platform arms

references/platforms.md has each arm. In short: macOS arm64 ran every step here on v1.29.0, and Linux aarch64 on v1.18.2 and once on v1.22.2, with the one difference that Linux needs no --branchable. On v1.16.1 Linux aarch64 refused --store and froze a branch source; from v1.18.2 it does neither. Linux x86_64 ran every script on v1.34.0, twice, with the same results. Windows on v1.34.0 checkpoints to a file, pauses and resumes, and restores a machine started with --branchable; without the flag a checkpoint or a pause fails with os error 1784 from v1.28.1, where v1.22.2 to v1.28.0 took them. Every branch needs --freeze-source and --store is refused, so checkpoint.sh cannot run there. From v1.23.0 a checkpoint from an arm64 Mac restores on arm64 Linux, run once from this Mac to Lima.

Eval prompts, and what they produced

Run 2026-09-24 PT against v1.18.2 from the published release, under an isolated HOME, on macOS 26.6.2 arm64 and Lima linux-kvm (Ubuntu 24.04 aarch64), every machine at 1024 MiB. Output is verbatim unless marked.

1. “Checkpoint this machine every few minutes, keep only the last few, and show me I can go back to any of them.”

schedule.sh with --keep 2 --history 2, four captures ten seconds apart, both hosts: captures_ok=4, kept=2, and checkpoint-log on the newest listed ~0, ~1 and ~2. A marker written between captures came back as GEN1 from --at '~2' and GEN3 from the newest, with the RAM counter restored alongside. A second capture started while one ran:

result=skipped
note=a capture of smolskill-src is still running (pid 13935). Run one capture at a time per source: ...

2. “Fan this prepared machine out into workers, and keep a copy of exactly the state they started from.”

branch-point.sh --count 2 on a --branch-ready source, macOS: the source wrote 8445 to RAM before parking; both children read SETUP_DONE, their own CHILD= name and 8445, and a restore of the branch-point checkpoint read 8445. Linux the same with 22642. source_state=running on both.

3. “Pause the machine I restored and bring it back later without losing what is running.”

The restored machine’s counter: ram before=31 after=34 on macOS and ram before=31 after=33 on Linux, across a 10 s pause. Separately, on macOS a restored machine paused while the host disk was full failed its first resume with extract paused checkpoint: failed to unpack ..., and the same machine resume succeeded once space was freed: counter 8 before the pause, 14 after, still counting.

Re-verified on v1.34.0

Run 2026-11-09 PT against v1.34.0 from the published release, checksum checked, under a fresh isolated HOME on macOS 27.2 arm64 (Apple M4), twice, the second time from a fresh HOME, and once on Lima linux-kvm (Ubuntu 24.04 aarch64, 11 GiB). Every step on both hosts: result=schedule_ok, three generations listed, restores of ~2 and the newest holding the source’s RAM token, a pause and resume with the counter at 32 then 33, and both children of a batch branch holding one token. Captures now take under a second, so checkpoint.sh run right after another hit the one-second name collision 4 times in 6 (result=FAILED ... already exists); it now waits for the next second. On macOS a machine started without --branchable took a --store capture and restored with its RAM token on v1.24.0 and v1.34.0, where v1.23.7 refused it.

On Linux x86_64 (Ubuntu 22.04, KVM, an NVIDIA A10 cloud host with 30 cores and 222 GiB), twice on 2026-11-09, each pass from a fresh HOME, every step: result=schedule_ok with 4 of 4 captures, three generations listed, restores of ~2 and the newest holding the source’s RAM token, a pause and resume with the counter at 31 then 32, and both children of a batch branch holding one token.

Re-verified on v1.29.0

Run 2026-10-15 PT against v1.29.0 from the published release, checksum checked, under a fresh isolated HOME on macOS 27.0.1 arm64, the scripts twice, the second time from a fresh HOME, and the checks beyond them once. Linux was not run, so the Linux stamp stays on its earlier release.

schedule.sh gave captures_ok=4 and longest_capture_s=1, checkpoint-log listed ~0 to ~2, and restores of ~2 and the newest gave result=restored. The RAM counter read 32 before a 5 s pause and 33 after the resume. A batch of two branched with the same RAM token in both children and in a restore of the branch point, once branch-point.sh read every process for the parked helper (references/traps.md, Corrections); before that fix it reported the source never parked. One child with --name branched. A capture of a machine with a host mount failed with the profile error above, and with --host-mounts it succeeded.

Re-verified on v1.23.0

Run 2026-10-04 PT against v1.23.0 from the published release, checksum checked, under a fresh isolated HOME on macOS 27.0.1 arm64, once. On Lima linux-kvm (Ubuntu 24.04 aarch64) the checks named below ran once, so the Linux stamp stays on its earlier release.

macOS: every script. Three scheduled captures (captures_ok=3, longest_capture_s=1), checkpoint-log listing three generations, restores of ~2 and the newest, the counter at 20 before a pause and 24 after the resume, both children of a batch branch holding the source’s token, and cleanup.sh --restore-base removing 258 MB of _restore-checkpoints. A capture now reports when the source resumed, Source resumed after 0.024s; packaging continued in the background (#1519).

A checkpoint taken on this Mac restored on Linux aarch64 (#1531). create-source.sh, then machine checkpoint --name smolskill-mac --output mac.checkpoint on the Mac with the RAM counter at 6, which printed 40 MiB written and 0.066s source pause. Copied to Lima unchanged, machine create --from and machine start there took a second each; the counter read 7 and five seconds later 12, the RAM token and the disk marker matched the Mac’s, and the guest said Linux 6.12.95.

Re-verified on v1.22.2

Run 2026-10-03 PT against v1.22.2 from the published release, checksum checked, under an isolated HOME on macOS 27.0.1 arm64, twice, the second time from a fresh HOME. On Lima linux-kvm (Ubuntu 24.04 aarch64) on 2026-10-03 guests above 2048 MiB timed out, so the Linux lines below are a single run and the Linux stamp stays on its earlier release.

macOS: every script. Three scheduled captures (captures_ok=3, longest_capture_s=1), checkpoint-log listing three generations, restores of ~2 and the newest, the counter at 21 before a pause and 25 after the resume, both children of a batch branch holding the source’s token, and cleanup.sh --restore-base removing 275 MB of _restore-checkpoints. The --branchable facts in “Branchability is decided at start” were measured on releases v1.18.2 to v1.22.2 on 2026-10-03.

Linux aarch64: every script, with checkpoint_needs_branchable=no, the counter 63 then 66 and both children holding the same token.

What was not run

  • The scripts on Windows, which are bash.
  • A schedule over hours. Four captures ten seconds apart on each host, and single captures within each run; the growth of a store under --history 32 over a day was not measured.
  • Restores on another host, beyond Mac to Linux. Through v1.22.2 a checkpoint restores only on a host of the same platform, and another is refused with checkpoint host platform '...' does not match '...'. From v1.23.0 one taken on an arm64 Mac restores on arm64 Linux, run once above; Moving a running machine from a Mac to Linux has the requirements. Linux to Mac, and a stored capture rather than a file, were not run.
  • GPU and CUDA machines, which a capture refuses, and --share-weights.
  • Branch pools (--hold, branch-release), and --freeze-source outside Windows.
  • The API’s restore route, the from field on create; the local-api packet covers pause and resume over HTTP.
  • dev-env for state that only has to survive a stop and start.
  • pack for moving a machine to another host as a file, including a restored one.
  • local-api for pause and resume over HTTP.
  • teardown for the wider cleanup, and for what a leak check must exclude.

Scripts

The files the procedure runs, in the order it runs them. It calls each one by the path in its heading, relative to the directory the procedure is saved in.

scripts/preflight.sh

#!/usr/bin/env bash
# Report whether this host can checkpoint, restore, pause and branch smolvm
# machines, and whether the place the checkpoints will go has room. Read-only:
# starts no VM, writes no smolvm state, changes no group membership.
#
# usage: preflight.sh [--store <dir>]   (default: the current directory)
#
# Output is one key=value per line so a caller can parse it. The last line is
# always result=ready or result=blocked.

set -uo pipefail

VERIFIED_VERSION="1.34.0"

emit() { printf '%s=%s\n' "$1" "$2"; }

blocked=0
note() { printf 'note=%s\n' "$1"; }

STORE_DIR="."
while [ $# -gt 0 ]; do
    case "$1" in
        --store) STORE_DIR="$2"; shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done

# --- the binary --------------------------------------------------------------

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
    emit smolvm_installed no
    emit smolvm_version ""
    blocked=1
else
    emit smolvm_installed yes
    version="$("$SMOLVM" --version 2>/dev/null | awk '{print $NF}')"
    emit smolvm_version "${version:-unknown}"
fi

emit verified_version "$VERIFIED_VERSION"
if [ -n "${version:-}" ] && [ "$version" != "unknown" ]; then
    if [ "$version" = "$VERIFIED_VERSION" ]; then
        emit version_status match
    else
        newest="$(printf '%s\n%s\n' "$version" "$VERIFIED_VERSION" | sort -V | tail -1)"
        if [ "$newest" = "$version" ]; then
            emit version_status newer
            note "this packet was verified on $VERIFIED_VERSION and the binary is $version; flags and messages move every release, so check the output against the binary before trusting a step here"
        else
            emit version_status older
            note "this packet was verified on $VERIFIED_VERSION and the binary is $version"
        fi
    fi
else
    emit version_status unknown
fi

# --- platform ----------------------------------------------------------------

kernel="$(uname -s)"
arch="$(uname -m)"
case "$arch" in aarch64|arm64) arch=aarch64 ;; esac

case "$kernel" in
    Darwin)
        emit platform "darwin-$arch"
        emit accel hvf
        emit macos_version "$(sw_vers -productVersion)"
        hv="$(sysctl -n kern.hv_support 2>/dev/null)"
        if [ "$hv" = "1" ]; then emit accel_access ok; else emit accel_access denied; blocked=1; fi
        if [ "$arch" != "aarch64" ]; then
            emit hardware_verified no
            note "Intel Mac is not verified by this packet; the installer accepts it and nothing here was run on one"
        else
            emit hardware_verified yes
        fi
        # A VM's agent socket lives under the cache directory. macOS sockaddr_un
        # holds 104 bytes including the terminator.
        sock="$HOME/Library/Caches/smolvm/vms/0123456789abcdef/agent.sock"
        len=${#sock}
        emit socket_path_bytes "$len"
        if [ "$len" -gt 100 ]; then
            emit socket_path_status too_long
            blocked=1
            note "HOME is too deep: every VM start will fail. From v1.28.0 the error names the socket path and its length; before it, krun_start_enter -22, whose text blames disks and device options. Install under a shorter HOME."
        else
            emit socket_path_status ok
        fi
        # v1.20.0 lifted the flag for a checkpoint file and for pause, and v1.24.0
        # for a stored capture, which falls back to a synchronous save (#1584).
        if [ -n "${version:-}" ] && [ "$(printf '%s\n%s\n' "$version" 1.24.0 | sort -V | head -1)" = "1.24.0" ]; then
            emit checkpoint_needs_branchable no
            note "on macOS from v1.24.0 a checkpoint, a stored capture and a pause all work without machine start --branchable; a branch still needs it, and the flag cannot be added to a running machine. The scripts here always start with it."
        elif [ -n "${version:-}" ] && [ "$(printf '%s\n%s\n' "$version" 1.20.0 | sort -V | head -1)" = "1.20.0" ]; then
            emit checkpoint_needs_branchable yes
            note "on macOS machine checkpoint --store refuses a machine that was not started with machine start --branchable, with deferred durable save requires file-backed guest RAM, which names neither. A checkpoint file and a pause do not need it from v1.20.0. The flag cannot be added to a running machine; the scripts here always start with it."
        else
            emit checkpoint_needs_branchable yes
            note "on macOS machine checkpoint and machine pause refuse a machine that was not started with machine start --branchable, and the error, guest RAM has no file-backed regions, names neither. The flag cannot be added to a running machine; the scripts here always start with it."
        fi
        ;;
    Linux)
        emit platform "linux-$arch"
        emit accel kvm
        emit socket_path_status n_a
        if [ ! -e /dev/kvm ]; then
            emit accel_access missing
            blocked=1
            note "/dev/kvm does not exist; this host has no KVM"
        elif [ -r /dev/kvm ] && [ -w /dev/kvm ]; then
            emit accel_access ok
        else
            emit accel_access denied
            blocked=1
            note "your user cannot open /dev/kvm. The installer warns and continues, so a successful install says nothing about whether a VM will start. Fix: sudo usermod -aG kvm \$USER, then run the next command through sg kvm -c '...' rather than logging out."
        fi
        emit checkpoint_needs_branchable no
        ;;
    *)
        emit platform "unsupported-$kernel"
        emit accel unknown
        emit accel_access unknown
        blocked=1
        note "this script covers macOS and Linux. On Windows v1.22.2 checkpoints to a file, pauses, and branches with --freeze-source on every branch, and refuses --store, so checkpoint.sh cannot run there; see references/platforms.md."
        ;;
esac

# --- the checkpoint surface this packet uses -----------------------------------

if [ -n "$SMOLVM" ]; then
    ckhelp="$("$SMOLVM" machine checkpoint --help 2>&1)"
    for flag in --store --history --export-from; do
        k="has_$(printf %s "${flag#--}" | tr - _)"
        if grep -q -- "$flag" <<<"$ckhelp"; then emit "$k" yes; else emit "$k" no; blocked=1; fi
    done
    if "$SMOLVM" machine checkpoint-log --help >/dev/null 2>&1; then emit has_checkpoint_log yes; else emit has_checkpoint_log no; blocked=1; fi
    if "$SMOLVM" machine pause --help >/dev/null 2>&1; then emit has_pause yes; else emit has_pause no; fi
    createhelp="$("$SMOLVM" machine create --help 2>&1)"
    if grep -q -- '--at <GENERATION>' <<<"$createhelp"; then emit restore_any_generation yes; else emit restore_any_generation no; blocked=1; fi
    emit scheduler builtin_none
    emit retention builtin_none
fi

# --- memory and disk ---------------------------------------------------------

# The scripts start every machine with --mem 1024. A host that cannot boot that
# inside smolvm's fixed 30 s readiness window fails with "agent did not become
# ready", which names neither memory nor the host.
emit memory_required_mib 1024
mkdir -p "$STORE_DIR" 2>/dev/null
free_kb="$(df -Pk "$STORE_DIR" 2>/dev/null | awk 'NR==2{print $4}')"
if [ -n "$free_kb" ]; then
    emit store_free_mib "$((free_kb / 1024))"
    # A capture of a 1 GiB alpine machine wrote about 55 MiB the first time and
    # about 19 MiB after that; restores, the restore base and the capture
    # staging need more. 2 GiB is headroom, not a measurement of any one step.
    if [ "$((free_kb / 1024))" -lt 2048 ]; then
        emit store_space_ok no
        blocked=1
        note "less than 2 GiB free where the checkpoints go. A capture into a full disk publishes nothing, and a machine restored or resumed while the disk is full can be left with guest I/O errors that stop it from being deleted normally (references/traps.md)."
    else
        emit store_space_ok yes
    fi
fi

if [ "$blocked" -eq 0 ]; then emit result ready; else emit result blocked; fi

scripts/create-source.sh

#!/usr/bin/env bash
# Create and start a machine to checkpoint and branch, with state in RAM and on
# disk that a restore or a child can be checked against.
#
# usage: create-source.sh [<name>] [--branch-ready]
#   <name>          default smolskill-src; must start with smolskill- so cleanup.sh deletes it
#   --branch-ready  the workload does its setup and then parks in smolvm-branch-ready,
#                   which a batch branch (branch-point.sh --count) needs
#
# The workload keeps a counter in /dev/shm, which is tmpfs and therefore RAM: a
# restore or a resume that brings the counter back proves memory came back, not
# only the disk. /root/setup.txt is the disk marker. The machine is started
# --branchable because on macOS a checkpoint is refused without it and the flag
# cannot be turned on for a running machine.

set -uo pipefail

here="$(cd "$(dirname "$0")" && pwd)"
NAME="smolskill-src"
READY=0
for a in "$@"; do
    case "$a" in
        --branch-ready) READY=1 ;;
        -*) printf 'unknown argument: %s\n' "$a" >&2; exit 2 ;;
        *) NAME="$a" ;;
    esac
done

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
    printf 'smolvm not found; set SMOLVM to its path\n' >&2
    exit 2
fi
case "$NAME" in
    smolskill-*) ;;
    *) printf 'name must start with smolskill- so cleanup.sh will delete it\n' >&2; exit 2 ;;
esac

counter='i=0; while true; do i=$((i+1)); echo $i > /dev/shm/ram_counter; sleep 1; done'
if [ "$READY" -eq 1 ]; then
    # setup, then park; each child continues into the program after --
    workload="echo SETUP_DONE > /root/setup.txt; echo \$RANDOM > /dev/shm/ram_token; exec smolvm-branch-ready -- sh -c 'echo CHILD=\$SMOLVM_BRANCH_NAME > /root/child.txt; $counter'"
else
    workload="echo SETUP_DONE > /root/setup.txt; echo \$RANDOM > /dev/shm/ram_token; $counter"
fi

"$SMOLVM" machine create --name "$NAME" --net --mem 1024 --image alpine -- sh -c "$workload" 2>&1 | sed 's/^/  /'
"$here/cleanup.sh" --record "$NAME"
"$SMOLVM" machine start --name "$NAME" --branchable 2>&1 | sed 's/^/  /'

# Wait for a value from the workload, not for the start command to return.
for i in $(seq 1 60); do
    v="$("$SMOLVM" machine exec --name "$NAME" -- cat /root/setup.txt 2>/dev/null)"
    if [ "$v" = "SETUP_DONE" ]; then
        printf 'workload_ready_after_s=%s\n' "$i"
        printf 'ram_token=%s\n' "$("$SMOLVM" machine exec --name "$NAME" -- cat /dev/shm/ram_token 2>/dev/null)"
        printf 'result=up\n'
        exit 0
    fi
    sleep 1
done
printf 'result=FAILED the workload never wrote /root/setup.txt\n'
exit 1

scripts/checkpoint.sh

#!/usr/bin/env bash
# One capture of a running machine into a checkpoint store, under a unique name,
# then optional retention. This is the command a timer runs: smolvm has no
# checkpoint scheduler and no retention of its own.
#
# usage: checkpoint.sh --name <machine> --store <dir> [--out <dir>] [--label <word>]
#                      [--keep <K>] [--history <N>]
#   --store <dir>    the checkpoint store, created if missing
#   --out <dir>      where the checkpoint directories go (default: the store's
#                    parent); must be on the same filesystem as the store
#   --label <word>   goes into the output name (default: ckpt)
#   --keep <K>       keep the K newest (K at least 1) checkpoints with this label
#                    in --out, delete the older ones, then run checkpoint-prune
#   --history <N>    earlier generations each new checkpoint retains (smolvm's
#                    default is 32). Retention frees space only for generations
#                    no kept checkpoint still retains, so bound this too.
#
# What it enforces, each from the engine's own documentation of periodic use:
#   - one capture at a time per source: a run that finds another in progress
#     exits 3 without capturing
#   - a unique output each time: <name>-<label>-<UTC timestamp>.smolcheckpoint
#   - the previous checkpoint stays until the new one is published, and
#     "published" is read back from checkpoint-log, not taken from the exit code
#   - retention removes whole checkpoint directories, then prunes the store

set -uo pipefail

NAME=""
STORE=""
OUT=""
LABEL="ckpt"
KEEP=""
HISTORY=""

while [ $# -gt 0 ]; do
    case "$1" in
        --name)    NAME="$2"; shift ;;
        --store)   STORE="$2"; shift ;;
        --out)     OUT="$2"; shift ;;
        --label)   LABEL="$2"; shift ;;
        --keep)    KEEP="$2"; shift ;;
        --history) HISTORY="$2"; shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done

if [ -z "$NAME" ] || [ -z "$STORE" ]; then
    printf 'usage: checkpoint.sh --name <machine> --store <dir> [--out <dir>] [--keep <K>] [--history <N>]\n' >&2
    exit 2
fi
if [ -n "$KEEP" ]; then
    case "$KEEP" in
        *[!0-9]*) printf 'result=FAILED --keep takes a whole number of at least 1, not %s\n' "$KEEP"; exit 2 ;;
    esac
    [ "$KEEP" -ge 1 ] || { printf 'result=FAILED --keep takes a whole number of at least 1, not %s\n' "$KEEP"; exit 2; }
fi
STAMP_GLOB='[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]T[0-9][0-9][0-9][0-9][0-9][0-9]Z'

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
    printf 'smolvm not found; set SMOLVM to its path\n' >&2
    exit 2
fi

mkdir -p "$STORE"
STORE="$(cd "$STORE" && pwd)"
[ -n "$OUT" ] || OUT="$(dirname "$STORE")"
mkdir -p "$OUT"
OUT="$(cd "$OUT" && pwd)"

# One capture at a time per source. mkdir is atomic on every filesystem this
# runs on, and macOS has no flock(1). A lock whose owner is gone is taken over.
LOCK="$STORE/.smolskill-capture-$NAME.lock"
if ! mkdir "$LOCK" 2>/dev/null; then
    owner="$(cat "$LOCK/pid" 2>/dev/null)"
    if [ -n "$owner" ] && kill -0 "$owner" 2>/dev/null; then
        printf 'result=skipped\n'
        printf 'note=a capture of %s is still running (pid %s). Run one capture at a time per source: overlapping captures compete for memory and I/O and finish no sooner. Lengthen the interval.\n' "$NAME" "$owner"
        exit 3
    fi
    rm -rf "$LOCK"
    mkdir "$LOCK" || { printf 'result=FAILED could not take the capture lock %s\n' "$LOCK"; exit 1; }
fi
printf '%s\n' "$$" > "$LOCK/pid"
trap 'rm -rf "$LOCK"' EXIT

# The name has one-second resolution and a capture can take less than that, so a
# capture right after another waits for the next second rather than collide.
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
output="$OUT/$NAME-$LABEL-$stamp.smolcheckpoint"
if [ -e "$output" ]; then
    sleep 1
    stamp="$(date -u +%Y%m%dT%H%M%SZ)"
    output="$OUT/$NAME-$LABEL-$stamp.smolcheckpoint"
fi
if [ -e "$output" ]; then
    printf 'result=FAILED %s already exists; smolvm never overwrites an output, and two captures in one second collide on this name\n' "$output"
    exit 1
fi

hist=()
[ -n "$HISTORY" ] && hist=(--history "$HISTORY")

start="$(date +%s)"
out="$("$SMOLVM" machine checkpoint --name "$NAME" --store "$STORE" --output "$output" ${hist[@]+"${hist[@]}"} 2>&1)"
rc=$?
end="$(date +%s)"
printf '%s\n' "$out" | sed 's/^/  /'
printf 'output=%s\n' "$output"
printf 'capture_s=%s\n' "$((end - start))"

# The exit code is not the assertion. A checkpoint is usable once the store
# lists it as its own generation, so read that back.
log="$("$SMOLVM" machine checkpoint-log "$output" 2>&1)"
if [ "$rc" -ne 0 ] || ! printf '%s' "$log" | grep -q '(this checkpoint)'; then
    printf 'published=no\n'
    printf 'result=FAILED the capture did not publish; the previous checkpoint is untouched\n'
    case "$out" in
        *"no file-backed regions"*|*"requires file-backed guest RAM"*)
            printf 'note=on macOS before v1.24.0 a stored checkpoint needs the machine started with machine start --branchable, and this one was not. The flag cannot be turned on for a running machine.\n' ;;
        *"No space left"*)
            printf 'note=the disk is full. A failed capture publishes nothing, so delete old checkpoints and run machine checkpoint-prune before the next attempt.\n' ;;
    esac
    exit 1
fi
printf 'published=yes\n'
printf 'generation=%s\n' "$(printf '%s\n' "$log" | awk '$1=="~0"{print $2}')"
printf 'generations_retained=%s\n' "$(printf '%s\n' "$log" | grep -c '^~')"

# Retention: only after the new checkpoint is published, and by whole
# directory. Output names sort by time because the stamp is UTC and fixed width,
# and a glob expands sorted and unsplit, so the oldest come first and a space in
# the path stays inside one name. Matching the stamp's shape keeps another label
# that starts with this one, such as ckpt-daily, out of the count.
if [ -n "$KEEP" ]; then
    set -- "$OUT/$NAME-$LABEL"-$STAMP_GLOB.smolcheckpoint
    [ -e "$1" ] || set --
    while [ "$#" -gt "$KEEP" ]; do
        rm -rf -- "$1" && printf 'deleted=%s\n' "$1"
        shift
    done
    "$SMOLVM" machine checkpoint-prune --store "$STORE" 2>&1 | sed 's/^/  /'
    set -- "$OUT/$NAME-$LABEL"-$STAMP_GLOB.smolcheckpoint
    [ -e "$1" ] || set --
    printf 'kept=%s\n' "$#"
fi

printf 'store_size=%s\n' "$(du -sh "$STORE" 2>/dev/null | cut -f1)"
printf 'result=captured\n'

scripts/schedule.sh

#!/usr/bin/env bash
# Run checkpoint.sh on an interval, in the foreground, and say whether the
# interval is long enough. For a test or a session an agent is watching; for a
# schedule that outlives the session, put checkpoint.sh under cron, a systemd
# timer or launchd instead (references/scheduling.md).
#
# usage: schedule.sh --name <machine> --store <dir> --every <seconds> --times <N>
#                    [--keep <K>] [--history <N>] [--out <dir>]

set -uo pipefail

here="$(cd "$(dirname "$0")" && pwd)"
EVERY=""
TIMES=""
pass=()
while [ $# -gt 0 ]; do
    case "$1" in
        --every) EVERY="$2"; shift ;;
        --times) TIMES="$2"; shift ;;
        --name|--store|--keep|--history|--out|--label) pass+=("$1" "$2"); shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done
if [ -z "$EVERY" ] || [ -z "$TIMES" ]; then
    printf 'usage: schedule.sh --name <machine> --store <dir> --every <seconds> --times <N> [--keep <K>]\n' >&2
    exit 2
fi

ok=0
failed=0
longest=0
for run in $(seq 1 "$TIMES"); do
    start="$(date +%s)"
    out="$("$here/checkpoint.sh" ${pass[@]+"${pass[@]}"} 2>&1)"
    rc=$?
    took=$(( $(date +%s) - start ))
    [ "$took" -gt "$longest" ] && longest="$took"
    printf 'run=%s rc=%s took_s=%s %s\n' "$run" "$rc" "$took" "$(printf '%s\n' "$out" | grep -E '^(output|result|kept|store_size)=' | tr '\n' ' ')"
    if [ "$rc" -eq 0 ]; then ok=$((ok + 1)); else failed=$((failed + 1)); printf '%s\n' "$out" | sed 's/^/  /'; fi
    if [ "$took" -ge "$EVERY" ]; then
        printf 'note=this capture took %ss and the interval is %ss. Set the interval from the complete capture time, not the source pause.\n' "$took" "$EVERY"
    fi
    [ "$run" -lt "$TIMES" ] && sleep $(( EVERY > took ? EVERY - took : 0 ))
done
printf 'captures_ok=%s\n' "$ok"
printf 'captures_failed=%s\n' "$failed"
printf 'longest_capture_s=%s\n' "$longest"
if [ "$failed" -eq 0 ]; then printf 'result=schedule_ok\n'; else printf 'result=FAILED\n'; exit 1; fi

scripts/restore.sh

#!/usr/bin/env bash
# Restore a checkpoint, or an earlier generation it retains, into a new machine
# and start it.
#
# usage: restore.sh --from <checkpoint> --name <new> [--at <~N|id>]
#
# The restore path is `machine create --from`; there is no `machine restore`.
# The new machine is started --branchable so it can itself be checkpointed,
# paused and branched on every host.

set -uo pipefail

here="$(cd "$(dirname "$0")" && pwd)"
FROM=""
NAME=""
AT=""
while [ $# -gt 0 ]; do
    case "$1" in
        --from) FROM="$2"; shift ;;
        --name) NAME="$2"; shift ;;
        --at)   AT="$2"; shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done
if [ -z "$FROM" ] || [ -z "$NAME" ]; then
    printf 'usage: restore.sh --from <checkpoint> --name <new> [--at <~N|id>]\n' >&2
    exit 2
fi
case "$NAME" in smolskill-*) ;; *) printf 'name must start with smolskill- so cleanup.sh will delete it\n' >&2; exit 2 ;; esac

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
[ -n "$SMOLVM" ] || { printf 'smolvm not found; set SMOLVM to its path\n' >&2; exit 2; }

at=()
[ -n "$AT" ] && at=(--at "$AT")
# create refuses a name that exists, and the listing below would then find the
# old machine, so the create's own status decides; pipefail carries it.
if ! "$SMOLVM" machine create --name "$NAME" --from "$FROM" ${at[@]+"${at[@]}"} 2>&1 | sed 's/^/  /'; then
    printf 'result=FAILED machine create --from did not succeed, so nothing was restored\n'
    exit 1
fi
# Read the list first: under pipefail, grep -q closing the pipe early can fail
# the pipeline and report a machine missing that exists.
listing="$("$SMOLVM" machine list 2>/dev/null)"
if ! printf '%s\n' "$listing" | grep -q "^$NAME "; then
    printf 'result=FAILED no machine was created\n'
    exit 1
fi
"$here/cleanup.sh" --record "$NAME"
start="$(date +%s)"
"$SMOLVM" machine start --name "$NAME" --branchable 2>&1 | sed 's/^/  /'
printf 'start_s=%s\n' "$(( $(date +%s) - start ))"
state="$("$SMOLVM" machine list 2>/dev/null | awk -v n="$NAME" '$1==n{print $2}')"
printf 'state=%s\n' "$state"
[ "$state" = running ] && printf 'result=restored\n' || { printf 'result=FAILED\n'; exit 1; }

scripts/branch-point.sh

#!/usr/bin/env bash
# Branch a machine and take a checkpoint of the same point, so the state the
# children started from outlives them. A branch's captured generation lives on
# this host only; the checkpoint is the copy you can keep, export and restore.
#
# usage: branch-point.sh --from <source> --store <dir> --count <N> --name-prefix <prefix>
#        branch-point.sh --from <source> --store <dir> --name <child>
#   --count/--name-prefix  a batch. The source's workload must park in
#                          smolvm-branch-ready; the checkpoint is taken while it
#                          is parked, so it is the exact state every child starts from.
#   --name                 one child. The checkpoint is taken first and the
#                          branch follows, a second or two later, so the two
#                          points differ by whatever the source did in between.

set -uo pipefail

here="$(cd "$(dirname "$0")" && pwd)"
FROM=""
STORE=""
COUNT=""
PREFIX=""
CHILD=""
while [ $# -gt 0 ]; do
    case "$1" in
        --from)        FROM="$2"; shift ;;
        --store)       STORE="$2"; shift ;;
        --count)       COUNT="$2"; shift ;;
        --name-prefix) PREFIX="$2"; shift ;;
        --name)        CHILD="$2"; shift ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done
if [ -z "$FROM" ] || [ -z "$STORE" ] || { [ -z "$CHILD" ] && { [ -z "$COUNT" ] || [ -z "$PREFIX" ]; }; }; then
    printf 'usage: branch-point.sh --from <source> --store <dir> (--count <N> --name-prefix <p> | --name <child>)\n' >&2
    exit 2
fi
case "${PREFIX}${CHILD}" in
    smolskill-*) ;;
    *) printf 'children must be named smolskill-... so cleanup.sh deletes them\n' >&2; exit 2 ;;
esac

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
[ -n "$SMOLVM" ] || { printf 'smolvm not found; set SMOLVM to its path\n' >&2; exit 2; }

if [ -n "$COUNT" ]; then
    # Parked means a process's command line starts with the helper. pgrep -f is
    # not a test for it: it matches the sh -c that runs the check, whose text
    # names the helper. From v1.25.4 the workload runs under smolvm's own init
    # rather than as PID 1, so read every process.
    parked=no
    for i in $(seq 1 120); do
        if "$SMOLVM" machine exec --name "$FROM" -- sh -c 'for f in /proc/[0-9]*/cmdline; do tr "\0" " " < "$f"; echo; done' 2>/dev/null | grep -q '^smolvm-branch-ready'; then
            parked=yes; printf 'source_parked_after_s=%s\n' "$i"; break
        fi
        sleep 1
    done
    if [ "$parked" != yes ]; then
        printf 'result=FAILED the source never parked in smolvm-branch-ready, so a batch branch would wait for a branch point that is not coming. Its workload has to run smolvm-branch-ready -- <program> after its setup (create-source.sh --branch-ready does), or branch one child with --name.\n'
        exit 1
    fi
fi

ck="$("$here/checkpoint.sh" --name "$FROM" --store "$STORE" --label branchpoint 2>&1)"
printf '%s\n' "$ck" | sed 's/^/  /'
checkpoint="$(printf '%s\n' "$ck" | sed -n 's/^output=//p')"
if ! printf '%s' "$ck" | grep -q '^result=captured'; then
    printf 'result=FAILED no checkpoint of the branch point, so no branch was taken\n'
    exit 1
fi
printf 'checkpoint=%s\n' "$checkpoint"

if [ -n "$COUNT" ]; then
    out="$("$SMOLVM" machine branch --from "$FROM" --count "$COUNT" --name-prefix "$PREFIX" 2>&1)"
    children="$(seq 0 $((COUNT - 1)) | sed "s/^/$PREFIX-/")"
else
    out="$("$SMOLVM" machine branch --from "$FROM" --name "$CHILD" 2>&1)"
    children="$CHILD"
fi
printf '%s\n' "$out" | sed 's/^/  /'
made=0
# Read the list once: under pipefail, grep -q closing the pipe early can fail
# the pipeline and report a child missing that exists.
listing="$("$SMOLVM" machine list 2>/dev/null)"
for c in $children; do
    "$here/cleanup.sh" --record "$c"
    if printf '%s\n' "$listing" | grep -q "^$c "; then made=$((made + 1)); printf 'child=%s\n' "$c"; fi
done
# The source is frozen on some hosts after a branch and running on others; read
# which, rather than assuming.
printf 'source_state=%s\n' "$(printf '%s\n' "$listing" | awk -v n="$FROM" '$1==n{print $2}')"
if [ "$made" -eq "$(printf '%s\n' "$children" | grep -c .)" ]; then
    printf 'result=branched\n'
else
    printf 'result=FAILED %s of the children exist\n' "$made"
    exit 1
fi

scripts/cleanup.sh

#!/usr/bin/env bash
# Delete the machines this packet's scripts created, then prove the host is clean.
#
# Only machines recorded in the state file are deleted, with --cascade, so any
# machine branched from one of them goes too, whatever its name. Any other
# machine you or another session created by hand is never deleted. Scripts
# record a name by calling: cleanup.sh --record <name>
#
# usage: cleanup.sh [--record <name>] [--reap] [--purge] [--restore-base] [--checkpoints <dir>...]
#   --record <name>       add a machine name to the state file and exit
#   --reap                kill every VM process under this HOME; run without it first
#   --purge               also remove the state file once the list is empty
#   --restore-base        also remove smolvm's copies of recently restored checkpoints
#   --checkpoints <dir>   also remove these checkpoint directories and stores; last
#
# Checkpoints hold the machine's memory and disks, so they are sensitive, and
# smolvm keeps copies of its own: vms/_restore-base is a clone of the most
# recently restored checkpoint, and from v1.22.0 vms/_restore-checkpoints holds
# the last three restored, both kept to make the next restore cheaper, and both
# outlive every machine and every checkpoint file.

set -uo pipefail

PACKET="branch-and-checkpoint"
PREFIX="smolskill-"

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
STATE_DIR="${SMOLVM_SKILL_STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/smolvm-skills}"
STATE_FILE="$STATE_DIR/$PACKET.machines"

reap=0
purge=0
restore_base=0
while [ $# -gt 0 ]; do
    case "$1" in
        --record)
            mkdir -p "$STATE_DIR"
            printf '%s\n' "$2" >> "$STATE_FILE"
            exit 0
            ;;
        --reap)  reap=1 ;;
        --purge) purge=1 ;;
        --restore-base) restore_base=1 ;;
        --checkpoints) shift; break ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done

if [ -z "$SMOLVM" ]; then
    printf 'smolvm not found; set SMOLVM to its path\n' >&2
    exit 2
fi

# With SMOLVM_DATA_DIR set, smolvm runs with HOME there on Linux, so its cache
# is under .cache in it.
case "$(uname -s)" in
    Darwin) VMS_DIR="$HOME/Library/Caches/smolvm/vms" ;;
    *)      cache_root="${SMOLVM_DATA_DIR:+$SMOLVM_DATA_DIR/.cache}"
            VMS_DIR="${cache_root:-${XDG_CACHE_HOME:-$HOME/.cache}}/smolvm/vms" ;;
esac
VMS_DIR="${SMOLVM_VMS_DIR:-$VMS_DIR}"
SMOLVM_PREFIX="${SMOLVM_PREFIX:-$HOME/.smolvm}"
if [ -n "${SMOLVM:-}" ]; then
    case "$(readlink "$SMOLVM" 2>/dev/null || printf '%s' "$SMOLVM")" in
        "$SMOLVM_PREFIX"/*) ;;
        *) printf 'note=%s is not under %s, so the process scan cannot see its forked VMs (all of its VMs on macOS); set SMOLVM_PREFIX to its directory\n' "$SMOLVM" "$SMOLVM_PREFIX" ;;
    esac
fi

# List this HOME's VM processes as "pid marker". The plain run path execs a
# `_boot-vm` child that carries its boot config; the pack-run path forks one that
# carries none. Linux names both `libkrun VM`, or `VM:<hostname>` when HOSTNAME
# is exported; on macOS the executable path and the parent chain scope the
# search. The teardown packet's traps have the why.
list_vm_processes() {
    case "$(uname -s)" in
        Linux)
            for p in /proc/[0-9]*; do
                case "$(cat "$p/comm" 2>/dev/null)" in "libkrun VM"|VM:*) ;; *) continue ;; esac
                pid="${p#/proc/}"
                cfg="$(tr '\0' '\n' < "$p/cmdline" 2>/dev/null | sed -n '3p')"
                case "$cfg" in
                    "$VMS_DIR"/*) printf '%s %s\n' "$pid" "$cfg"; continue ;;
                esac
                case "$(readlink "$p/exe" 2>/dev/null)" in
                    "$SMOLVM_PREFIX"/*) printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
                esac
            done
            ;;
        Darwin)
            # shellcheck disable=SC2009  # pgrep -f would match this script.
            procs="$(ps -axo pid=,ppid=,command= 2>/dev/null)"
            printf '%s\n' "$procs" | while read -r pid ppid rest; do
                case "$rest" in "$SMOLVM_PREFIX"/smolvm-bin*) ;; *) continue ;; esac
                case "$rest" in
                    *" _boot-vm "*) printf '%s %s\n' "$pid" "${rest#* _boot-vm }"; continue ;;
                esac
                # An orphan counts only if it is a run; a fork has its parent's command line.
                if [ "$ppid" = 1 ]; then
                    case "$rest" in
                        *" machine run "*|*" vm run "*|*" pack run "*)
                            printf '%s orphaned-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
                    esac
                else
                    parent="$(printf '%s\n' "$procs" | while read -r q qp qrest; do [ "$q" = "$ppid" ] && { printf '%s' "$qrest"; break; }; done)"
                    [ "$parent" = "$rest" ] && printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX"
                fi
            done
            ;;
    esac
}

# 1. Delete recorded machines. Without --force a delete prompts and defaults to
# No; --cascade removes branch children, which otherwise block it.
if [ -s "$STATE_FILE" ]; then
    while read -r name; do
        [ -n "$name" ] || continue
        case "$name" in "$PREFIX"*) ;; *)
            printf 'skipping %s: not created by this packet (no %s prefix)\n' "$name" "$PREFIX"
            continue ;;
        esac
        # A child is gone once its source was deleted with --cascade, and on
        # v1.18.2 `machine stop` on a name that does not exist leaves an empty
        # directory under vms/. Act only on names still listed.
        "$SMOLVM" machine list </dev/null 2>/dev/null | awk 'NR>2{print $1}' > "$STATE_DIR/.listed" 2>/dev/null
        grep -qx -- "$name" "$STATE_DIR/.listed" || continue
        # Only names still listed: a second stop of a missing name leaves a
        # directory that reads as a leak. The list is read first because grep -q
        # under pipefail can fail the pipeline and skip a listed machine.
        listed="$("$SMOLVM" machine list </dev/null 2>/dev/null | awk 'NR>2{print $1}')"
        grep -qx -- "$name" <<<"$listed" || continue
        "$SMOLVM" machine stop   --name "$name" </dev/null >/dev/null 2>&1
        "$SMOLVM" machine delete --name "$name" --force --cascade </dev/null 2>&1 | sed 's/^/  /'
    done < "$STATE_FILE"
fi

# 1b. Checkpoints and stores named on the command line, then the restore base.
# A store is known by the objects directory and .lock its first capture makes, not by its name.
for d in "$@"; do
    case "$d" in
        *.smolcheckpoint|*.checkpoint) ;;
        *) if [ ! -d "$d/objects" ] || [ ! -e "$d/.lock" ]; then
               printf 'skipping %s: not a .checkpoint, a .smolcheckpoint or a checkpoint store\n' "$d"; continue
           fi ;;
    esac
    [ -e "$d" ] && rm -rf -- "$d" && printf 'removed=%s\n' "$d"
done
for cache in _restore-base _restore-checkpoints; do
    if [ "$restore_base" -eq 1 ] && [ -d "$VMS_DIR/$cache" ]; then
        rm -rf "${VMS_DIR:?}/$cache" "$VMS_DIR/$cache.lock" && printf 'removed=%s\n' "$VMS_DIR/$cache"
    elif [ "$restore_base" -eq 1 ]; then
        printf '%s=absent (nothing to remove)\n' "$cache"
    elif [ -d "$VMS_DIR/$cache" ]; then
        printf '%s=present %s (restored checkpoints kept by smolvm; --restore-base removes it)\n' "$cache" "$(du -sh "$VMS_DIR/$cache" 2>/dev/null | cut -f1)"
    fi
done

rm -f "$STATE_DIR/.listed"

# 2. An ephemeral machine's entry retires after its run returns: poll, 20 s at most.
printf 'waiting=up to 20s for ephemeral entries to retire before asserting\n'
waited=0
listing="$("$SMOLVM" machine list 2>&1)"
while ! grep -q 'No machines found' <<<"$listing" && [ "$waited" -lt 20 ]; do
    sleep 1; waited=$((waited + 1))
    listing="$("$SMOLVM" machine list 2>&1)"
done
printf 'waited=%ss\n' "$waited"

# 3. Assert the value, not the exit code.
if grep -q 'No machines found' <<<"$listing"; then
    printf 'machines=clean\n'
    [ "$purge" -eq 1 ] && rm -f "$STATE_FILE"
else
    printf 'machines=remaining\n'
    printf '%s\n' "$listing" | sed 's/^/  /'
    printf 'note=this packet did not create these, so it will not delete them. By name:\n'
    printf '  smolvm machine stop --name <NAME> && smolvm machine delete --name <NAME> --force\n'
    printf '  add --cascade for a machine that was branched from another\n'
fi

# 4. Report VM processes under this HOME's state, such as a killed wrapper's.
# With --reap every one is killed, including a machine another packet kept.
found=0
while read -r pid cfg; do
    [ -n "$pid" ] || continue
    found=1
    printf 'vm_process=%s config=%s\n' "$pid" "$cfg"
    if [ "$reap" -eq 1 ]; then
        kill -9 "$pid" 2>/dev/null && printf '  killed %s\n' "$pid"
    fi
done <<EOF
$(list_vm_processes)
EOF

if [ "$found" -eq 0 ]; then
    printf 'vm_processes=none\n'
elif [ "$reap" -eq 0 ]; then
    printf 'rerun with --reap to kill them\n'
fi

Checkpoint, restore, pause and branch traps

Each entry was hit on v1.18.2 on 2026-09-24 unless it says otherwise.

--branchable is decided at start, and older macOS releases needed it for more than branching

machine branch against a machine started without it, on either host:

Error: agent operation failed: fork: machine 'smolskill-sfx' was not started as branchable, so it
has no copy-on-write memory to branch from. Restart it with `smolvm machine start --name
smolskill-sfx --branchable`; branchability is decided at start time and cannot be turned on for an
already-running machine.

That message is clear. The macOS one for checkpoint and pause before v1.20.0 is not:

Error: agent operation failed: checkpoint machine: libkrun save failed: ERR EIO capture VM: VM
snapshot/restore failed: retain COW guest-memory generation: guest RAM has no file-backed regions

pause prints the same text, because a pause is a checkpoint underneath, and the machine keeps running. On Linux aarch64 both succeeded without the flag: a checkpoint of 55 MiB with a 1.242 s pause, and a pause and resume. On those releases, start every machine you might checkpoint on macOS with --branchable.

From v1.20.0 macOS takes a checkpoint file and a pause without the flag, measured on v1.20.0, v1.20.2, v1.21.1, v1.22.0 and v1.22.2, where v1.19.0 and v1.19.3 still refused. A stored capture needs it through v1.23.7. There --store against a machine started without it fails with:

Error: agent operation failed: checkpoint machine: libkrun save failed: ERR ENOTSUP VM
snapshot/restore failed: retain COW guest-memory generation: deferred durable save requires
file-backed guest RAM

From v1.24.0 the capture falls back to a synchronous save (#1584): on v1.24.0 and v1.34.0 the same --store capture published, and a machine restored from it held the source’s RAM token.

--output must end in .checkpoint or .smolcheckpoint, even when it is a directory

From v1.19.1 the file is a .checkpoint in machine checkpoint --help, and on v1.22.2 any other name fails with output must end in .checkpoint, while .smolcheckpoint is still accepted and machine create --help still names it. The scripts use .smolcheckpoint so they run on older releases too. On v1.18.2:

Error: config operation failed: checkpoint machine: output must end in .smolcheckpoint

for -o ./x.checkpoint, and the same for --store ./st -o ./inc1, where the output is a directory and the help calls it one.

The exit code of a capture is not the proof

A capture is usable once it has been durably published, which checkpoint-log <output> shows as the ~0 line ending (this checkpoint). scripts/checkpoint.sh reads that back before it deletes any older checkpoint. During this run the host disk filled and an export failed with tar error: No space left on device (os error 28); a scheduler that trusted a zero exit elsewhere and deleted the previous checkpoint would have been left with none.

--keep does not bound the store on its own

Each stored checkpoint retains up to --history earlier generations, 32 by default, and deleting the directory of a generation that a kept checkpoint retains frees nothing. With --keep 2 --history 2 the store grew 59, 76, 95, 114 MB over four captures. references/scheduling.md has how to pick the two together.

A restored branch point is not a parked source

A checkpoint taken while the source is parked in smolvm-branch-ready holds exactly the state its children start from. Restoring it does not give you a parked source back: on start the helper releases, and the workload runs the program after -- with an empty SMOLVM_BRANCH_NAME, so /root/child.txt in that test read CHILD=. A batch branch from the restored machine then waits its full --ready-timeout, 10 minutes by default, and fails with

A batch branch checkpoints the source at a point its workload declares by running
`smolvm-branch-ready` after setup ... either add the call, raise --ready-timeout, or take single
`--name` branches, which checkpoint the source wherever it is.

A single --name branch from the restored machine worked and carried the source’s state. To fan out again from a kept branch point, branch it one child at a time or restore it once per worker, and give each its identity some other way, since the branch variables are empty.

Detecting a parked source: pgrep -f always matches

smolvm machine exec --name src -- sh -c 'pgrep -f smolvm-branch-ready' succeeds whether or not the source is parked, because the sh -c running it has the string in its own command line. Read PID 1 instead, which is the workload:

smolvm machine exec --name src -- sh -c 'tr "\0" " " < /proc/1/cmdline' | grep -q '^smolvm-branch-ready'

A parked source shows smolvm-branch-ready -- sh -c ...; a released one shows the program after --.

machine list | grep -q under pipefail

set -o pipefail plus smolvm machine list | grep -q name can fail for a machine that exists: grep -q exits on the first match, the list’s next write hits a closed pipe, and the pipeline’s status is the list’s failure. Two scripts in this packet reported result=FAILED no machine was created for machines that were running before this was caught. Read the list into a variable, then search it.

A paused machine refuses stop and start, and a failed resume can be retried

Error: agent operation failed: stop: machine has saved execution; use resume or delete
Error: agent operation failed: start: machine has saved execution; use resume

A restored machine paused while the host disk was full failed its resume with extract paused checkpoint: failed to unpack .... After space was freed the same machine resume succeeded, and the counter the workload keeps in RAM read 14 against 8 before the pause, then kept counting.

A full disk leaves machines that delete --force will not remove

A machine restored and started while the disk was full came up with its overlay failing. Its delete, later, with space available:

Error: agent operation failed: stop agent: guest did not confirm filesystem synchronization; left
the VM alive for retry: agent operation failed: shutdown ack: freeze /oldroot/mnt/overlay: I/O error
(os error 5)

exit 1, still running. scripts/cleanup.sh --reap killed its VM process, after which delete --force removed it. --reap kills every VM process under this HOME, not only that one, so use it when nothing else there should keep running. scripts/preflight.sh checks free space for this reason.

du overstates what a restore costs

Restores are copy-on-write over the checkpoint. On macOS three restores of one checkpoint moved the volume’s free space by 22 MB at create and 65 MB at start, about 29 MB a machine, while du on each machine’s directory said 150 to 190 MB. On Linux each restored machine’s directory holds small qcow2 layers over .smolcheckpoint-*.raw bases, 29 MB by du. Measure a restore budget with df before and after.

smolvm keeps restored checkpoints after you delete them

On macOS vms/_restore-base under smolvm’s cache is, in the source’s words, a pristine clone of the most recently restored checkpoint, kept so the next restore writes only the chunks that differ. It was 243 MB here, with the checkpoint’s memory.bin inside, and it stayed after every machine and every checkpoint file was deleted. smolvm serve start’s reclaim did not touch it. scripts/cleanup.sh --restore-base removes it. None was created on Linux aarch64.

From v1.22.0 the cache is vms/_restore-checkpoints, which keeps the last three restored checkpoints, up to 16 GiB, so jumping back to one clones it instead of rebuilding its RAM. After this packet’s procedure on macOS on v1.23.0 it held 258 MB with every machine and checkpoint deleted, and no _restore-base was made. machine create --from takes --restore-cache-entries and --restore-cache-gib; 0 entries turns it off. scripts/cleanup.sh --restore-base removes both.

machine stop on a missing name leaves an empty directory

On v1.18.2, smolvm machine stop --name <name> for a machine that does not exist leaves an empty directory under the VM cache. A cleanup that stops each recorded name after --cascade has already deleted the children leaves one per child, which a leak check counting directories then reports. This packet’s cleanup stops only names still listed.

Environment given to a restore reaches exec, not the resumed workload

machine create --from <checkpoint> -e ROLE=seven: machine exec saw ROLE=seven, and the resumed workload’s own environment did not have it. The workload is the process that was running when the checkpoint was taken, and its environment came with it.

Corrections

  • 2026-10-15, v1.29.0 run. “Read PID 1 instead, which is the workload” above stopped holding in v1.25.4 (#1632): PID 1 is now /run/smolvm/init container-init and the workload is its child, so the check never saw a parked source and branch-point.sh --count failed with the source never parked. Read every process instead:

    smolvm machine exec --name src -- sh -c 'for f in /proc/[0-9]*/cmdline; do tr "\0" " " < "$f"; echo; done' | grep -q '^smolvm-branch-ready'

    scripts/branch-point.sh does this, and on v1.29.0 a batch of two then branched with the same RAM token in both children and in a restore of the branch point.