Branch and checkpoint: scheduled checkpoints, restores and branch points
Saves a running smolvm machine as checkpoints on a schedule, keeps a bounded history, restores any generation in it, pauses and resumes machines without losing running processes, and branches a warm machine into children with a checkpoint of the exact point they started from. Use when a machine’s state has to survive the machine; when an agent or a job needs to roll back; when checkpoints must be taken periodically, since smolvm has no scheduler or retention of its own; when fanning a prepared machine out into workers; or when a checkpoint, a pause or a branch fails with an error that names none of its preconditions. Do not use it to ship an environment to another host as a file, which is the pack packet, or for state that only has to survive a stop and start, which is the dev-env packet.
Verified on smolvm v1.34.0 on macOS arm64 and Linux x86_64, 2026-11-09, and on v1.18.2 on Linux aarch64, 2026-09-24. Done means a restore brings back both the disk and the memory you left, the schedule keeps exactly the checkpoints you asked for and no fewer, and every child of a branch starts from a state you still hold as a checkpoint after the children are gone. The Linux aarch64 runs used the scripts of their date; this version’s preflight and cleanup scripts ran on Linux aarch64 on v1.22.2 on 2026-10-03.
README.md says what each of these operations is. This file is how to run them, and the traps.
Three facts shape everything here.
- smolvm has no scheduler and no retention.
scripts/checkpoint.shis the one capture a timer runs, with the rules the engine asks a scheduler to follow built in;references/scheduling.mdputs it under cron, systemd or launchd. - A branch’s captured state stays on this host and cannot be exported. So a branch point is paired with a checkpoint taken at the same moment, which is the copy you keep.
--branchableatstartdecides what a machine can do later. A branch source needs it on every host, and on macOS before v1.24.0 so does a stored capture, which is what the scripts here take; Linux checkpoints and pauses without it.README.mdhas the errors when it is missing. The scripts always start with it.
Procedure
- [ ] 1. preflight.sh read result= and store_space_ok=
- [ ] 2. create-source.sh a machine with state in RAM and on disk
- [ ] 3. checkpoint.sh / schedule.sh captures, with retention
- [ ] 4. restore.sh any generation, into a new machine
- [ ] 5. pause and resume the restored machine, or any running one
- [ ] 6. branch-point.sh children plus a checkpoint of where they started
- [ ] 7. cleanup.sh machines, checkpoints, and smolvm's restore cache 1. Preflight. Read-only.
scripts/preflight.sh --store ./store It checks that this binary has --store, --history, checkpoint-log, pause and restore --at, says whether this host needs --branchable, and reports scheduler=builtin_none and retention=builtin_none so nobody goes looking for them. store_space_ok=no blocks: a capture
into a full disk publishes nothing, and a machine restored while the disk was full was left with
guest I/O errors that stopped it being deleted normally.
2. A source with state you can check.
scripts/create-source.sh smolskill-src # for checkpoints and restores
scripts/create-source.sh smolskill-bp --branch-ready # for a batch branch The workload writes a counter into /dev/shm every second, which is tmpfs and so RAM, and a marker to /root/setup.txt on disk. A restore that brings the counter back proves memory came back, not only
the disk. --branch-ready makes the workload park in smolvm-branch-ready after its setup, which a
batch branch waits for.
Or the user’s own machine. Name it with --name in the steps below. On macOS before v1.24.0,
if it was started without --branchable, the first capture fails with deferred durable save requires file-backed guest RAM (v1.20.0 to v1.23.7) or guest RAM has no file-backed regions (before v1.20.0); from v1.24.0 the capture falls back to a synchronous save (#1584) and works. On
the older releases, and for a branch on any release, the only fix is a restart with the flag:
smolvm machine stop --name worker
smolvm machine start --name worker --branchable That restart is a normal stop and start: disk state survives, and anything in RAM or /tmp is
lost, once. Say so to the user before doing it. Every start of that machine needs the flag from
then on for it to stay checkpointable.
A capture also refuses a machine with resources bound to this host: a host mount (-v), a
published socket, a remote volume, a host secret reference, custom DNS, a named network, a GPU,
CUDA, Rosetta, --ssh-agent or a Docker socket. The error, the initial portable checkpoint profile does not support ...; stop or detach these resources before capture, names which. From
v1.29.0 machine checkpoint --host-mounts captures a machine with host mounts, and the checkpoint
restores only on a host where the same paths exist.
3. Capture, once or on a schedule.
scripts/checkpoint.sh --name smolskill-src --store ./store --keep 6 --history 2
scripts/schedule.sh --name smolskill-src --store ./store --every 10 --times 4 --keep 2 --history 2 Each capture goes to a new directory named <machine>-<label>-<UTC time>.smolcheckpoint beside the
store, or in the directory --out <dir> names, which has to be on the same filesystem as the
store. It is judged published only when checkpoint-log lists it as (this checkpoint), and only
then are the oldest beyond --keep deleted and the store pruned. A capture started while another
of the same machine runs exits 3 with result=skipped. schedule.sh is for a session you are
watching; a schedule that outlives it is checkpoint.sh under a system timer, references/scheduling.md.
run=1 rc=0 took_s=2 ... kept=1 store_size= 59M result=captured
run=2 rc=0 took_s=5 ... kept=2 store_size= 76M result=captured
run=3 rc=0 took_s=2 ... kept=2 store_size= 95M result=captured
run=4 rc=0 took_s=1 ... kept=2 store_size=114M result=captured
captures_ok=4
captures_failed=0
longest_capture_s=5
result=schedule_ok That is the v1.18.2 run. Ten seconds is for watching it work; a schedule uses minutes, references/scheduling.md.
Set the interval from longest_capture_s, not from the pause. The source pauses for a few
hundredths of a second; the command takes seconds, because retained RAM is hashed and compressed
after the source resumes. --keep alone does not bound the disk: every kept checkpoint also
retains --history earlier generations, 32 by default, so the store above kept growing with two
directories kept. Bound both.
4. Restore, any generation.
smolvm machine checkpoint-log ./smolskill-src-ckpt-<time>.smolcheckpoint
scripts/restore.sh --from <checkpoint> --name smolskill-old --at '~2'
scripts/restore.sh --from <checkpoint> --name smolskill-new ~0 is the checkpoint itself and ~N is N generations back along its history. restore.sh takes only a name starting with smolskill-, so cleanup can find it, and refuses any other with name must start with smolskill- so cleanup.sh will delete it; to restore under a name the user
chooses, run smolvm machine create --name <name> --from <checkpoint> --at '~N' and then smolvm machine start --name <name> --branchable yourself. The restore path is machine create --from; there is no machine restore. A restored machine takes its own name as
its hostname, and its disks are copy-on-write over what the checkpoint holds, so a restore costs
about 29 MB of real disk on macOS for a 243 MiB checkpoint while du reports five times that.
5. Pause and resume.
smolvm machine pause --name smolskill-new
smolvm machine resume --name smolskill-new Pause saves RAM, disks and the running execution and stops the machine; resume brings the same
execution back under the same name. Check a value the workload holds in memory: on both hosts the
counter read 31 before the pause and 34 and 33 after the resume, so it continued where it stopped
and did not run while paused. A paused machine refuses stop and start; resume it or delete it.
A resume that fails keeps the saved state, and a later resume of the same machine succeeded.
6. Branch, and keep the branch point.
scripts/branch-point.sh --from smolskill-bp --store ./store --count 2 --name-prefix smolskill-w
scripts/branch-point.sh --from smolskill-src --store ./store --name smolskill-one For a batch it waits until a process in the source runs smolvm-branch-ready, takes the checkpoint while
the source is parked there, then branches. Every child and that checkpoint then hold the same
state: a random value the source wrote to RAM before parking came back identical in both children
and in a restore of the checkpoint.
source_parked_after_s=1
checkpoint=.../smolskill-bp-branchpoint-<time>.smolcheckpoint
child=smolskill-w-0
child=smolskill-w-1
source_state=running
result=branched For one child with --name the checkpoint is taken first and the branch a second or two later, so
the two points differ by whatever the source did in between.
A restored branch point is not a parked source. On start the helper releases and the workload
runs the child program with an empty SMOLVM_BRANCH_NAME, so a batch branch from it waits for a
branch point that never comes. Branch it with --name, one child at a time, or restore it once per
worker; either way the child gets the state the original children started from.
7. Clean up.
scripts/cleanup.sh --purge --restore-base --checkpoints ./store ./*.smolcheckpoint It deletes the machines the scripts recorded, children first by --cascade, removes the named
checkpoint directories and stores, and with --restore-base removes smolvm’s copies of restored
checkpoints, which otherwise stay after every machine and checkpoint is gone, printing a removed= or absent line for each. It waits up to 20 seconds, polling the machine list, and
says so, because an ephemeral entry retires after its run returns.
Traps
Full detail in references/traps.md. The ones that cost the most:
--outputmust end in.checkpointor.smolcheckpoint, with--storetoo, where it names a directory..checkpointis the name from v1.19.1, and the scripts use.smolcheckpoint, which every release accepts.pgrep -f smolvm-branch-readyinside the guest always matches, because thesh -crunning it contains the string. Read each/proc/<pid>/cmdline: from v1.25.4 the workload is not PID 1.smolvm machine list | grep -qunderpipefailcan report a machine missing that exists:grep -qcloses the pipe early and the failed write fails the pipeline. Read the list into a variable first.machine stop --nameon a name that does not exist leaves an empty directory under the VM cache on v1.18.2, and a second stop of it on v1.22.2 leaves the name inside, which reads as a leak.cleanup.shstops only names still listed.- smolvm keeps restored checkpoints, memory included, after the checkpoint is deleted
(
_restore-basewas 243 MB on macOS on v1.18.2);README.mdhas the rest, and step 7’s--restore-baseremoves them.
Security defaults, and why they are the defaults
- A checkpoint is the machine’s memory and disks. Anything the workload held, secrets included,
is in it; keep stores on a disk only you can read, and remove
_restore-baseand_restore-checkpointswhen you remove the checkpoints. - Retention deletes only after the new checkpoint is published, so a failed capture never leaves you with fewer good checkpoints than before.
- A store is local. It protects against a bad change, not a lost host; export a checkpoint
with
machine checkpoint --export-fromand copy it elsewhere for that. - Cleanup deletes only machines the scripts recorded under the
smolskill-prefix, and through--cascadeany machine branched from one of them, whatever its name, and removes only the checkpoint paths you name. - Nothing here escalates privilege or edits smolvm configuration.
Platform arms
references/platforms.md has each arm. In short: macOS arm64 ran every step here on v1.29.0,
and Linux aarch64 on v1.18.2 and once on v1.22.2, with the one difference that Linux needs no --branchable. On v1.16.1 Linux aarch64 refused --store and froze a branch source; from v1.18.2
it does neither. Linux x86_64 ran every script on v1.34.0, twice, with the same results. Windows on v1.34.0 checkpoints to a file, pauses and resumes, and restores a
machine started with --branchable; without the flag a checkpoint or a pause fails with os error 1784 from v1.28.1, where v1.22.2 to v1.28.0 took them. Every branch needs --freeze-source and --store is refused, so checkpoint.sh cannot run there.
From v1.23.0 a checkpoint from an arm64 Mac restores on arm64 Linux, run once from this Mac to Lima.
Eval prompts, and what they produced
Run 2026-09-24 PT against v1.18.2 from the published release, under an isolated HOME, on macOS
26.6.2 arm64 and Lima linux-kvm (Ubuntu 24.04 aarch64), every machine at 1024 MiB. Output is
verbatim unless marked.
1. “Checkpoint this machine every few minutes, keep only the last few, and show me I can go back to any of them.”
schedule.sh with --keep 2 --history 2, four captures ten seconds apart, both hosts: captures_ok=4, kept=2, and checkpoint-log on the newest listed ~0, ~1 and ~2. A marker
written between captures came back as GEN1 from --at '~2' and GEN3 from the newest, with the
RAM counter restored alongside. A second capture started while one ran:
result=skipped
note=a capture of smolskill-src is still running (pid 13935). Run one capture at a time per source: ... 2. “Fan this prepared machine out into workers, and keep a copy of exactly the state they started from.”
branch-point.sh --count 2 on a --branch-ready source, macOS: the source wrote 8445 to RAM
before parking; both children read SETUP_DONE, their own CHILD= name and 8445, and a restore of
the branch-point checkpoint read 8445. Linux the same with 22642. source_state=running on both.
3. “Pause the machine I restored and bring it back later without losing what is running.”
The restored machine’s counter: ram before=31 after=34 on macOS and ram before=31 after=33 on
Linux, across a 10 s pause. Separately, on macOS a restored machine paused while the host disk was
full failed its first resume with extract paused checkpoint: failed to unpack ..., and the same machine resume succeeded once space was freed: counter 8 before the pause, 14 after, still
counting.
Re-verified on v1.34.0
Run 2026-11-09 PT against v1.34.0 from the published release, checksum checked, under a fresh
isolated HOME on macOS 27.2 arm64 (Apple M4), twice, the second time from a fresh HOME, and once
on Lima linux-kvm (Ubuntu 24.04 aarch64, 11 GiB). Every step on both hosts: result=schedule_ok,
three generations listed, restores of ~2 and the newest holding the source’s RAM token, a pause
and resume with the counter at 32 then 33, and both children of a batch branch holding one token.
Captures now take under a second, so checkpoint.sh run right after another hit the one-second name
collision 4 times in 6 (result=FAILED ... already exists); it now waits for the next second. On
macOS a machine started without --branchable took a --store capture and restored with its RAM
token on v1.24.0 and v1.34.0, where v1.23.7 refused it.
On Linux x86_64 (Ubuntu 22.04, KVM, an NVIDIA A10 cloud host with 30 cores and 222 GiB), twice on
2026-11-09, each pass from a fresh HOME, every step: result=schedule_ok with 4 of 4 captures,
three generations listed, restores of ~2 and the newest holding the source’s RAM token, a pause
and resume with the counter at 31 then 32, and both children of a batch branch holding one token.
Re-verified on v1.29.0
Run 2026-10-15 PT against v1.29.0 from the published release, checksum checked, under a fresh
isolated HOME on macOS 27.0.1 arm64, the scripts twice, the second time from a fresh HOME, and
the checks beyond them once. Linux was not run, so the Linux stamp stays on its earlier release.
schedule.sh gave captures_ok=4 and longest_capture_s=1, checkpoint-log listed ~0 to ~2,
and restores of ~2 and the newest gave result=restored. The RAM counter read 32 before a 5 s
pause and 33 after the resume. A batch of two branched with the same RAM token in both children
and in a restore of the branch point, once branch-point.sh read every process for the parked
helper (references/traps.md, Corrections); before that fix it reported the source never parked.
One child with --name branched. A capture of a machine with a host mount failed with the profile
error above, and with --host-mounts it succeeded.
Re-verified on v1.23.0
Run 2026-10-04 PT against v1.23.0 from the published release, checksum checked, under a fresh
isolated HOME on macOS 27.0.1 arm64, once. On Lima linux-kvm (Ubuntu 24.04 aarch64) the
checks named below ran once, so the Linux stamp stays on its earlier release.
macOS: every script. Three scheduled captures (captures_ok=3, longest_capture_s=1), checkpoint-log listing three generations, restores of ~2 and the newest, the counter at 20
before a pause and 24 after the resume, both children of a batch branch holding the source’s token,
and cleanup.sh --restore-base removing 258 MB of _restore-checkpoints. A capture now reports
when the source resumed, Source resumed after 0.024s; packaging continued in the background (#1519).
A checkpoint taken on this Mac restored on Linux aarch64 (#1531). create-source.sh, then machine checkpoint --name smolskill-mac --output mac.checkpoint on the Mac with the RAM counter
at 6, which printed 40 MiB written and 0.066s source pause. Copied to Lima unchanged, machine create --from and machine start there took a second each; the counter read 7 and five
seconds later 12, the RAM token and the disk marker matched the Mac’s, and the guest said Linux 6.12.95.
Re-verified on v1.22.2
Run 2026-10-03 PT against v1.22.2 from the published release, checksum checked, under an isolated HOME on macOS 27.0.1 arm64, twice, the second time from a fresh HOME. On Lima linux-kvm (Ubuntu 24.04 aarch64) on 2026-10-03 guests above 2048 MiB timed out, so the Linux lines below are
a single run and the Linux stamp stays on its earlier release.
macOS: every script. Three scheduled captures (captures_ok=3, longest_capture_s=1), checkpoint-log listing three generations, restores of ~2 and the newest, the counter at 21
before a pause and 25 after the resume, both children of a batch branch holding the source’s token,
and cleanup.sh --restore-base removing 275 MB of _restore-checkpoints. The --branchable facts in “Branchability is decided at start” were measured on releases v1.18.2 to v1.22.2 on
2026-10-03.
Linux aarch64: every script, with checkpoint_needs_branchable=no, the counter 63 then 66 and both
children holding the same token.
What was not run
- The scripts on Windows, which are bash.
- A schedule over hours. Four captures ten seconds apart on each host, and single captures
within each run; the growth of a store under
--history 32over a day was not measured. - Restores on another host, beyond Mac to Linux. Through v1.22.2 a checkpoint restores only on
a host of the same platform, and another is refused with
checkpoint host platform '...' does not match '...'. From v1.23.0 one taken on an arm64 Mac restores on arm64 Linux, run once above; Moving a running machine from a Mac to Linux has the requirements. Linux to Mac, and a stored capture rather than a file, were not run. - GPU and CUDA machines, which a capture refuses, and
--share-weights. - Branch pools (
--hold,branch-release), and--freeze-sourceoutside Windows. - The API’s restore route, the
fromfield on create; thelocal-apipacket covers pause and resume over HTTP.
Related packets
dev-envfor state that only has to survive a stop and start.packfor moving a machine to another host as a file, including a restored one.local-apifor pause and resume over HTTP.teardownfor the wider cleanup, and for what a leak check must exclude.
Scripts
The files the procedure runs, in the order it runs them. It calls each one by the path in its heading, relative to the directory the procedure is saved in.
scripts/preflight.sh
#!/usr/bin/env bash
# Report whether this host can checkpoint, restore, pause and branch smolvm
# machines, and whether the place the checkpoints will go has room. Read-only:
# starts no VM, writes no smolvm state, changes no group membership.
#
# usage: preflight.sh [--store <dir>] (default: the current directory)
#
# Output is one key=value per line so a caller can parse it. The last line is
# always result=ready or result=blocked.
set -uo pipefail
VERIFIED_VERSION="1.34.0"
emit() { printf '%s=%s\n' "$1" "$2"; }
blocked=0
note() { printf 'note=%s\n' "$1"; }
STORE_DIR="."
while [ $# -gt 0 ]; do
case "$1" in
--store) STORE_DIR="$2"; shift ;;
*) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
esac
shift
done
# --- the binary --------------------------------------------------------------
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
emit smolvm_installed no
emit smolvm_version ""
blocked=1
else
emit smolvm_installed yes
version="$("$SMOLVM" --version 2>/dev/null | awk '{print $NF}')"
emit smolvm_version "${version:-unknown}"
fi
emit verified_version "$VERIFIED_VERSION"
if [ -n "${version:-}" ] && [ "$version" != "unknown" ]; then
if [ "$version" = "$VERIFIED_VERSION" ]; then
emit version_status match
else
newest="$(printf '%s\n%s\n' "$version" "$VERIFIED_VERSION" | sort -V | tail -1)"
if [ "$newest" = "$version" ]; then
emit version_status newer
note "this packet was verified on $VERIFIED_VERSION and the binary is $version; flags and messages move every release, so check the output against the binary before trusting a step here"
else
emit version_status older
note "this packet was verified on $VERIFIED_VERSION and the binary is $version"
fi
fi
else
emit version_status unknown
fi
# --- platform ----------------------------------------------------------------
kernel="$(uname -s)"
arch="$(uname -m)"
case "$arch" in aarch64|arm64) arch=aarch64 ;; esac
case "$kernel" in
Darwin)
emit platform "darwin-$arch"
emit accel hvf
emit macos_version "$(sw_vers -productVersion)"
hv="$(sysctl -n kern.hv_support 2>/dev/null)"
if [ "$hv" = "1" ]; then emit accel_access ok; else emit accel_access denied; blocked=1; fi
if [ "$arch" != "aarch64" ]; then
emit hardware_verified no
note "Intel Mac is not verified by this packet; the installer accepts it and nothing here was run on one"
else
emit hardware_verified yes
fi
# A VM's agent socket lives under the cache directory. macOS sockaddr_un
# holds 104 bytes including the terminator.
sock="$HOME/Library/Caches/smolvm/vms/0123456789abcdef/agent.sock"
len=${#sock}
emit socket_path_bytes "$len"
if [ "$len" -gt 100 ]; then
emit socket_path_status too_long
blocked=1
note "HOME is too deep: every VM start will fail. From v1.28.0 the error names the socket path and its length; before it, krun_start_enter -22, whose text blames disks and device options. Install under a shorter HOME."
else
emit socket_path_status ok
fi
# v1.20.0 lifted the flag for a checkpoint file and for pause, and v1.24.0
# for a stored capture, which falls back to a synchronous save (#1584).
if [ -n "${version:-}" ] && [ "$(printf '%s\n%s\n' "$version" 1.24.0 | sort -V | head -1)" = "1.24.0" ]; then
emit checkpoint_needs_branchable no
note "on macOS from v1.24.0 a checkpoint, a stored capture and a pause all work without machine start --branchable; a branch still needs it, and the flag cannot be added to a running machine. The scripts here always start with it."
elif [ -n "${version:-}" ] && [ "$(printf '%s\n%s\n' "$version" 1.20.0 | sort -V | head -1)" = "1.20.0" ]; then
emit checkpoint_needs_branchable yes
note "on macOS machine checkpoint --store refuses a machine that was not started with machine start --branchable, with deferred durable save requires file-backed guest RAM, which names neither. A checkpoint file and a pause do not need it from v1.20.0. The flag cannot be added to a running machine; the scripts here always start with it."
else
emit checkpoint_needs_branchable yes
note "on macOS machine checkpoint and machine pause refuse a machine that was not started with machine start --branchable, and the error, guest RAM has no file-backed regions, names neither. The flag cannot be added to a running machine; the scripts here always start with it."
fi
;;
Linux)
emit platform "linux-$arch"
emit accel kvm
emit socket_path_status n_a
if [ ! -e /dev/kvm ]; then
emit accel_access missing
blocked=1
note "/dev/kvm does not exist; this host has no KVM"
elif [ -r /dev/kvm ] && [ -w /dev/kvm ]; then
emit accel_access ok
else
emit accel_access denied
blocked=1
note "your user cannot open /dev/kvm. The installer warns and continues, so a successful install says nothing about whether a VM will start. Fix: sudo usermod -aG kvm \$USER, then run the next command through sg kvm -c '...' rather than logging out."
fi
emit checkpoint_needs_branchable no
;;
*)
emit platform "unsupported-$kernel"
emit accel unknown
emit accel_access unknown
blocked=1
note "this script covers macOS and Linux. On Windows v1.22.2 checkpoints to a file, pauses, and branches with --freeze-source on every branch, and refuses --store, so checkpoint.sh cannot run there; see references/platforms.md."
;;
esac
# --- the checkpoint surface this packet uses -----------------------------------
if [ -n "$SMOLVM" ]; then
ckhelp="$("$SMOLVM" machine checkpoint --help 2>&1)"
for flag in --store --history --export-from; do
k="has_$(printf %s "${flag#--}" | tr - _)"
if grep -q -- "$flag" <<<"$ckhelp"; then emit "$k" yes; else emit "$k" no; blocked=1; fi
done
if "$SMOLVM" machine checkpoint-log --help >/dev/null 2>&1; then emit has_checkpoint_log yes; else emit has_checkpoint_log no; blocked=1; fi
if "$SMOLVM" machine pause --help >/dev/null 2>&1; then emit has_pause yes; else emit has_pause no; fi
createhelp="$("$SMOLVM" machine create --help 2>&1)"
if grep -q -- '--at <GENERATION>' <<<"$createhelp"; then emit restore_any_generation yes; else emit restore_any_generation no; blocked=1; fi
emit scheduler builtin_none
emit retention builtin_none
fi
# --- memory and disk ---------------------------------------------------------
# The scripts start every machine with --mem 1024. A host that cannot boot that
# inside smolvm's fixed 30 s readiness window fails with "agent did not become
# ready", which names neither memory nor the host.
emit memory_required_mib 1024
mkdir -p "$STORE_DIR" 2>/dev/null
free_kb="$(df -Pk "$STORE_DIR" 2>/dev/null | awk 'NR==2{print $4}')"
if [ -n "$free_kb" ]; then
emit store_free_mib "$((free_kb / 1024))"
# A capture of a 1 GiB alpine machine wrote about 55 MiB the first time and
# about 19 MiB after that; restores, the restore base and the capture
# staging need more. 2 GiB is headroom, not a measurement of any one step.
if [ "$((free_kb / 1024))" -lt 2048 ]; then
emit store_space_ok no
blocked=1
note "less than 2 GiB free where the checkpoints go. A capture into a full disk publishes nothing, and a machine restored or resumed while the disk is full can be left with guest I/O errors that stop it from being deleted normally (references/traps.md)."
else
emit store_space_ok yes
fi
fi
if [ "$blocked" -eq 0 ]; then emit result ready; else emit result blocked; fi scripts/create-source.sh
#!/usr/bin/env bash
# Create and start a machine to checkpoint and branch, with state in RAM and on
# disk that a restore or a child can be checked against.
#
# usage: create-source.sh [<name>] [--branch-ready]
# <name> default smolskill-src; must start with smolskill- so cleanup.sh deletes it
# --branch-ready the workload does its setup and then parks in smolvm-branch-ready,
# which a batch branch (branch-point.sh --count) needs
#
# The workload keeps a counter in /dev/shm, which is tmpfs and therefore RAM: a
# restore or a resume that brings the counter back proves memory came back, not
# only the disk. /root/setup.txt is the disk marker. The machine is started
# --branchable because on macOS a checkpoint is refused without it and the flag
# cannot be turned on for a running machine.
set -uo pipefail
here="$(cd "$(dirname "$0")" && pwd)"
NAME="smolskill-src"
READY=0
for a in "$@"; do
case "$a" in
--branch-ready) READY=1 ;;
-*) printf 'unknown argument: %s\n' "$a" >&2; exit 2 ;;
*) NAME="$a" ;;
esac
done
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
printf 'smolvm not found; set SMOLVM to its path\n' >&2
exit 2
fi
case "$NAME" in
smolskill-*) ;;
*) printf 'name must start with smolskill- so cleanup.sh will delete it\n' >&2; exit 2 ;;
esac
counter='i=0; while true; do i=$((i+1)); echo $i > /dev/shm/ram_counter; sleep 1; done'
if [ "$READY" -eq 1 ]; then
# setup, then park; each child continues into the program after --
workload="echo SETUP_DONE > /root/setup.txt; echo \$RANDOM > /dev/shm/ram_token; exec smolvm-branch-ready -- sh -c 'echo CHILD=\$SMOLVM_BRANCH_NAME > /root/child.txt; $counter'"
else
workload="echo SETUP_DONE > /root/setup.txt; echo \$RANDOM > /dev/shm/ram_token; $counter"
fi
"$SMOLVM" machine create --name "$NAME" --net --mem 1024 --image alpine -- sh -c "$workload" 2>&1 | sed 's/^/ /'
"$here/cleanup.sh" --record "$NAME"
"$SMOLVM" machine start --name "$NAME" --branchable 2>&1 | sed 's/^/ /'
# Wait for a value from the workload, not for the start command to return.
for i in $(seq 1 60); do
v="$("$SMOLVM" machine exec --name "$NAME" -- cat /root/setup.txt 2>/dev/null)"
if [ "$v" = "SETUP_DONE" ]; then
printf 'workload_ready_after_s=%s\n' "$i"
printf 'ram_token=%s\n' "$("$SMOLVM" machine exec --name "$NAME" -- cat /dev/shm/ram_token 2>/dev/null)"
printf 'result=up\n'
exit 0
fi
sleep 1
done
printf 'result=FAILED the workload never wrote /root/setup.txt\n'
exit 1 scripts/checkpoint.sh
#!/usr/bin/env bash
# One capture of a running machine into a checkpoint store, under a unique name,
# then optional retention. This is the command a timer runs: smolvm has no
# checkpoint scheduler and no retention of its own.
#
# usage: checkpoint.sh --name <machine> --store <dir> [--out <dir>] [--label <word>]
# [--keep <K>] [--history <N>]
# --store <dir> the checkpoint store, created if missing
# --out <dir> where the checkpoint directories go (default: the store's
# parent); must be on the same filesystem as the store
# --label <word> goes into the output name (default: ckpt)
# --keep <K> keep the K newest (K at least 1) checkpoints with this label
# in --out, delete the older ones, then run checkpoint-prune
# --history <N> earlier generations each new checkpoint retains (smolvm's
# default is 32). Retention frees space only for generations
# no kept checkpoint still retains, so bound this too.
#
# What it enforces, each from the engine's own documentation of periodic use:
# - one capture at a time per source: a run that finds another in progress
# exits 3 without capturing
# - a unique output each time: <name>-<label>-<UTC timestamp>.smolcheckpoint
# - the previous checkpoint stays until the new one is published, and
# "published" is read back from checkpoint-log, not taken from the exit code
# - retention removes whole checkpoint directories, then prunes the store
set -uo pipefail
NAME=""
STORE=""
OUT=""
LABEL="ckpt"
KEEP=""
HISTORY=""
while [ $# -gt 0 ]; do
case "$1" in
--name) NAME="$2"; shift ;;
--store) STORE="$2"; shift ;;
--out) OUT="$2"; shift ;;
--label) LABEL="$2"; shift ;;
--keep) KEEP="$2"; shift ;;
--history) HISTORY="$2"; shift ;;
*) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
esac
shift
done
if [ -z "$NAME" ] || [ -z "$STORE" ]; then
printf 'usage: checkpoint.sh --name <machine> --store <dir> [--out <dir>] [--keep <K>] [--history <N>]\n' >&2
exit 2
fi
if [ -n "$KEEP" ]; then
case "$KEEP" in
*[!0-9]*) printf 'result=FAILED --keep takes a whole number of at least 1, not %s\n' "$KEEP"; exit 2 ;;
esac
[ "$KEEP" -ge 1 ] || { printf 'result=FAILED --keep takes a whole number of at least 1, not %s\n' "$KEEP"; exit 2; }
fi
STAMP_GLOB='[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]T[0-9][0-9][0-9][0-9][0-9][0-9]Z'
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
printf 'smolvm not found; set SMOLVM to its path\n' >&2
exit 2
fi
mkdir -p "$STORE"
STORE="$(cd "$STORE" && pwd)"
[ -n "$OUT" ] || OUT="$(dirname "$STORE")"
mkdir -p "$OUT"
OUT="$(cd "$OUT" && pwd)"
# One capture at a time per source. mkdir is atomic on every filesystem this
# runs on, and macOS has no flock(1). A lock whose owner is gone is taken over.
LOCK="$STORE/.smolskill-capture-$NAME.lock"
if ! mkdir "$LOCK" 2>/dev/null; then
owner="$(cat "$LOCK/pid" 2>/dev/null)"
if [ -n "$owner" ] && kill -0 "$owner" 2>/dev/null; then
printf 'result=skipped\n'
printf 'note=a capture of %s is still running (pid %s). Run one capture at a time per source: overlapping captures compete for memory and I/O and finish no sooner. Lengthen the interval.\n' "$NAME" "$owner"
exit 3
fi
rm -rf "$LOCK"
mkdir "$LOCK" || { printf 'result=FAILED could not take the capture lock %s\n' "$LOCK"; exit 1; }
fi
printf '%s\n' "$$" > "$LOCK/pid"
trap 'rm -rf "$LOCK"' EXIT
# The name has one-second resolution and a capture can take less than that, so a
# capture right after another waits for the next second rather than collide.
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
output="$OUT/$NAME-$LABEL-$stamp.smolcheckpoint"
if [ -e "$output" ]; then
sleep 1
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
output="$OUT/$NAME-$LABEL-$stamp.smolcheckpoint"
fi
if [ -e "$output" ]; then
printf 'result=FAILED %s already exists; smolvm never overwrites an output, and two captures in one second collide on this name\n' "$output"
exit 1
fi
hist=()
[ -n "$HISTORY" ] && hist=(--history "$HISTORY")
start="$(date +%s)"
out="$("$SMOLVM" machine checkpoint --name "$NAME" --store "$STORE" --output "$output" ${hist[@]+"${hist[@]}"} 2>&1)"
rc=$?
end="$(date +%s)"
printf '%s\n' "$out" | sed 's/^/ /'
printf 'output=%s\n' "$output"
printf 'capture_s=%s\n' "$((end - start))"
# The exit code is not the assertion. A checkpoint is usable once the store
# lists it as its own generation, so read that back.
log="$("$SMOLVM" machine checkpoint-log "$output" 2>&1)"
if [ "$rc" -ne 0 ] || ! printf '%s' "$log" | grep -q '(this checkpoint)'; then
printf 'published=no\n'
printf 'result=FAILED the capture did not publish; the previous checkpoint is untouched\n'
case "$out" in
*"no file-backed regions"*|*"requires file-backed guest RAM"*)
printf 'note=on macOS before v1.24.0 a stored checkpoint needs the machine started with machine start --branchable, and this one was not. The flag cannot be turned on for a running machine.\n' ;;
*"No space left"*)
printf 'note=the disk is full. A failed capture publishes nothing, so delete old checkpoints and run machine checkpoint-prune before the next attempt.\n' ;;
esac
exit 1
fi
printf 'published=yes\n'
printf 'generation=%s\n' "$(printf '%s\n' "$log" | awk '$1=="~0"{print $2}')"
printf 'generations_retained=%s\n' "$(printf '%s\n' "$log" | grep -c '^~')"
# Retention: only after the new checkpoint is published, and by whole
# directory. Output names sort by time because the stamp is UTC and fixed width,
# and a glob expands sorted and unsplit, so the oldest come first and a space in
# the path stays inside one name. Matching the stamp's shape keeps another label
# that starts with this one, such as ckpt-daily, out of the count.
if [ -n "$KEEP" ]; then
set -- "$OUT/$NAME-$LABEL"-$STAMP_GLOB.smolcheckpoint
[ -e "$1" ] || set --
while [ "$#" -gt "$KEEP" ]; do
rm -rf -- "$1" && printf 'deleted=%s\n' "$1"
shift
done
"$SMOLVM" machine checkpoint-prune --store "$STORE" 2>&1 | sed 's/^/ /'
set -- "$OUT/$NAME-$LABEL"-$STAMP_GLOB.smolcheckpoint
[ -e "$1" ] || set --
printf 'kept=%s\n' "$#"
fi
printf 'store_size=%s\n' "$(du -sh "$STORE" 2>/dev/null | cut -f1)"
printf 'result=captured\n' scripts/schedule.sh
#!/usr/bin/env bash
# Run checkpoint.sh on an interval, in the foreground, and say whether the
# interval is long enough. For a test or a session an agent is watching; for a
# schedule that outlives the session, put checkpoint.sh under cron, a systemd
# timer or launchd instead (references/scheduling.md).
#
# usage: schedule.sh --name <machine> --store <dir> --every <seconds> --times <N>
# [--keep <K>] [--history <N>] [--out <dir>]
set -uo pipefail
here="$(cd "$(dirname "$0")" && pwd)"
EVERY=""
TIMES=""
pass=()
while [ $# -gt 0 ]; do
case "$1" in
--every) EVERY="$2"; shift ;;
--times) TIMES="$2"; shift ;;
--name|--store|--keep|--history|--out|--label) pass+=("$1" "$2"); shift ;;
*) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
esac
shift
done
if [ -z "$EVERY" ] || [ -z "$TIMES" ]; then
printf 'usage: schedule.sh --name <machine> --store <dir> --every <seconds> --times <N> [--keep <K>]\n' >&2
exit 2
fi
ok=0
failed=0
longest=0
for run in $(seq 1 "$TIMES"); do
start="$(date +%s)"
out="$("$here/checkpoint.sh" ${pass[@]+"${pass[@]}"} 2>&1)"
rc=$?
took=$(( $(date +%s) - start ))
[ "$took" -gt "$longest" ] && longest="$took"
printf 'run=%s rc=%s took_s=%s %s\n' "$run" "$rc" "$took" "$(printf '%s\n' "$out" | grep -E '^(output|result|kept|store_size)=' | tr '\n' ' ')"
if [ "$rc" -eq 0 ]; then ok=$((ok + 1)); else failed=$((failed + 1)); printf '%s\n' "$out" | sed 's/^/ /'; fi
if [ "$took" -ge "$EVERY" ]; then
printf 'note=this capture took %ss and the interval is %ss. Set the interval from the complete capture time, not the source pause.\n' "$took" "$EVERY"
fi
[ "$run" -lt "$TIMES" ] && sleep $(( EVERY > took ? EVERY - took : 0 ))
done
printf 'captures_ok=%s\n' "$ok"
printf 'captures_failed=%s\n' "$failed"
printf 'longest_capture_s=%s\n' "$longest"
if [ "$failed" -eq 0 ]; then printf 'result=schedule_ok\n'; else printf 'result=FAILED\n'; exit 1; fi scripts/restore.sh
#!/usr/bin/env bash
# Restore a checkpoint, or an earlier generation it retains, into a new machine
# and start it.
#
# usage: restore.sh --from <checkpoint> --name <new> [--at <~N|id>]
#
# The restore path is `machine create --from`; there is no `machine restore`.
# The new machine is started --branchable so it can itself be checkpointed,
# paused and branched on every host.
set -uo pipefail
here="$(cd "$(dirname "$0")" && pwd)"
FROM=""
NAME=""
AT=""
while [ $# -gt 0 ]; do
case "$1" in
--from) FROM="$2"; shift ;;
--name) NAME="$2"; shift ;;
--at) AT="$2"; shift ;;
*) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
esac
shift
done
if [ -z "$FROM" ] || [ -z "$NAME" ]; then
printf 'usage: restore.sh --from <checkpoint> --name <new> [--at <~N|id>]\n' >&2
exit 2
fi
case "$NAME" in smolskill-*) ;; *) printf 'name must start with smolskill- so cleanup.sh will delete it\n' >&2; exit 2 ;; esac
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
[ -n "$SMOLVM" ] || { printf 'smolvm not found; set SMOLVM to its path\n' >&2; exit 2; }
at=()
[ -n "$AT" ] && at=(--at "$AT")
# create refuses a name that exists, and the listing below would then find the
# old machine, so the create's own status decides; pipefail carries it.
if ! "$SMOLVM" machine create --name "$NAME" --from "$FROM" ${at[@]+"${at[@]}"} 2>&1 | sed 's/^/ /'; then
printf 'result=FAILED machine create --from did not succeed, so nothing was restored\n'
exit 1
fi
# Read the list first: under pipefail, grep -q closing the pipe early can fail
# the pipeline and report a machine missing that exists.
listing="$("$SMOLVM" machine list 2>/dev/null)"
if ! printf '%s\n' "$listing" | grep -q "^$NAME "; then
printf 'result=FAILED no machine was created\n'
exit 1
fi
"$here/cleanup.sh" --record "$NAME"
start="$(date +%s)"
"$SMOLVM" machine start --name "$NAME" --branchable 2>&1 | sed 's/^/ /'
printf 'start_s=%s\n' "$(( $(date +%s) - start ))"
state="$("$SMOLVM" machine list 2>/dev/null | awk -v n="$NAME" '$1==n{print $2}')"
printf 'state=%s\n' "$state"
[ "$state" = running ] && printf 'result=restored\n' || { printf 'result=FAILED\n'; exit 1; } scripts/branch-point.sh
#!/usr/bin/env bash
# Branch a machine and take a checkpoint of the same point, so the state the
# children started from outlives them. A branch's captured generation lives on
# this host only; the checkpoint is the copy you can keep, export and restore.
#
# usage: branch-point.sh --from <source> --store <dir> --count <N> --name-prefix <prefix>
# branch-point.sh --from <source> --store <dir> --name <child>
# --count/--name-prefix a batch. The source's workload must park in
# smolvm-branch-ready; the checkpoint is taken while it
# is parked, so it is the exact state every child starts from.
# --name one child. The checkpoint is taken first and the
# branch follows, a second or two later, so the two
# points differ by whatever the source did in between.
set -uo pipefail
here="$(cd "$(dirname "$0")" && pwd)"
FROM=""
STORE=""
COUNT=""
PREFIX=""
CHILD=""
while [ $# -gt 0 ]; do
case "$1" in
--from) FROM="$2"; shift ;;
--store) STORE="$2"; shift ;;
--count) COUNT="$2"; shift ;;
--name-prefix) PREFIX="$2"; shift ;;
--name) CHILD="$2"; shift ;;
*) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
esac
shift
done
if [ -z "$FROM" ] || [ -z "$STORE" ] || { [ -z "$CHILD" ] && { [ -z "$COUNT" ] || [ -z "$PREFIX" ]; }; }; then
printf 'usage: branch-point.sh --from <source> --store <dir> (--count <N> --name-prefix <p> | --name <child>)\n' >&2
exit 2
fi
case "${PREFIX}${CHILD}" in
smolskill-*) ;;
*) printf 'children must be named smolskill-... so cleanup.sh deletes them\n' >&2; exit 2 ;;
esac
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
[ -n "$SMOLVM" ] || { printf 'smolvm not found; set SMOLVM to its path\n' >&2; exit 2; }
if [ -n "$COUNT" ]; then
# Parked means a process's command line starts with the helper. pgrep -f is
# not a test for it: it matches the sh -c that runs the check, whose text
# names the helper. From v1.25.4 the workload runs under smolvm's own init
# rather than as PID 1, so read every process.
parked=no
for i in $(seq 1 120); do
if "$SMOLVM" machine exec --name "$FROM" -- sh -c 'for f in /proc/[0-9]*/cmdline; do tr "\0" " " < "$f"; echo; done' 2>/dev/null | grep -q '^smolvm-branch-ready'; then
parked=yes; printf 'source_parked_after_s=%s\n' "$i"; break
fi
sleep 1
done
if [ "$parked" != yes ]; then
printf 'result=FAILED the source never parked in smolvm-branch-ready, so a batch branch would wait for a branch point that is not coming. Its workload has to run smolvm-branch-ready -- <program> after its setup (create-source.sh --branch-ready does), or branch one child with --name.\n'
exit 1
fi
fi
ck="$("$here/checkpoint.sh" --name "$FROM" --store "$STORE" --label branchpoint 2>&1)"
printf '%s\n' "$ck" | sed 's/^/ /'
checkpoint="$(printf '%s\n' "$ck" | sed -n 's/^output=//p')"
if ! printf '%s' "$ck" | grep -q '^result=captured'; then
printf 'result=FAILED no checkpoint of the branch point, so no branch was taken\n'
exit 1
fi
printf 'checkpoint=%s\n' "$checkpoint"
if [ -n "$COUNT" ]; then
out="$("$SMOLVM" machine branch --from "$FROM" --count "$COUNT" --name-prefix "$PREFIX" 2>&1)"
children="$(seq 0 $((COUNT - 1)) | sed "s/^/$PREFIX-/")"
else
out="$("$SMOLVM" machine branch --from "$FROM" --name "$CHILD" 2>&1)"
children="$CHILD"
fi
printf '%s\n' "$out" | sed 's/^/ /'
made=0
# Read the list once: under pipefail, grep -q closing the pipe early can fail
# the pipeline and report a child missing that exists.
listing="$("$SMOLVM" machine list 2>/dev/null)"
for c in $children; do
"$here/cleanup.sh" --record "$c"
if printf '%s\n' "$listing" | grep -q "^$c "; then made=$((made + 1)); printf 'child=%s\n' "$c"; fi
done
# The source is frozen on some hosts after a branch and running on others; read
# which, rather than assuming.
printf 'source_state=%s\n' "$(printf '%s\n' "$listing" | awk -v n="$FROM" '$1==n{print $2}')"
if [ "$made" -eq "$(printf '%s\n' "$children" | grep -c .)" ]; then
printf 'result=branched\n'
else
printf 'result=FAILED %s of the children exist\n' "$made"
exit 1
fi scripts/cleanup.sh
#!/usr/bin/env bash
# Delete the machines this packet's scripts created, then prove the host is clean.
#
# Only machines recorded in the state file are deleted, with --cascade, so any
# machine branched from one of them goes too, whatever its name. Any other
# machine you or another session created by hand is never deleted. Scripts
# record a name by calling: cleanup.sh --record <name>
#
# usage: cleanup.sh [--record <name>] [--reap] [--purge] [--restore-base] [--checkpoints <dir>...]
# --record <name> add a machine name to the state file and exit
# --reap kill every VM process under this HOME; run without it first
# --purge also remove the state file once the list is empty
# --restore-base also remove smolvm's copies of recently restored checkpoints
# --checkpoints <dir> also remove these checkpoint directories and stores; last
#
# Checkpoints hold the machine's memory and disks, so they are sensitive, and
# smolvm keeps copies of its own: vms/_restore-base is a clone of the most
# recently restored checkpoint, and from v1.22.0 vms/_restore-checkpoints holds
# the last three restored, both kept to make the next restore cheaper, and both
# outlive every machine and every checkpoint file.
set -uo pipefail
PACKET="branch-and-checkpoint"
PREFIX="smolskill-"
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
STATE_DIR="${SMOLVM_SKILL_STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/smolvm-skills}"
STATE_FILE="$STATE_DIR/$PACKET.machines"
reap=0
purge=0
restore_base=0
while [ $# -gt 0 ]; do
case "$1" in
--record)
mkdir -p "$STATE_DIR"
printf '%s\n' "$2" >> "$STATE_FILE"
exit 0
;;
--reap) reap=1 ;;
--purge) purge=1 ;;
--restore-base) restore_base=1 ;;
--checkpoints) shift; break ;;
*) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
esac
shift
done
if [ -z "$SMOLVM" ]; then
printf 'smolvm not found; set SMOLVM to its path\n' >&2
exit 2
fi
# With SMOLVM_DATA_DIR set, smolvm runs with HOME there on Linux, so its cache
# is under .cache in it.
case "$(uname -s)" in
Darwin) VMS_DIR="$HOME/Library/Caches/smolvm/vms" ;;
*) cache_root="${SMOLVM_DATA_DIR:+$SMOLVM_DATA_DIR/.cache}"
VMS_DIR="${cache_root:-${XDG_CACHE_HOME:-$HOME/.cache}}/smolvm/vms" ;;
esac
VMS_DIR="${SMOLVM_VMS_DIR:-$VMS_DIR}"
SMOLVM_PREFIX="${SMOLVM_PREFIX:-$HOME/.smolvm}"
if [ -n "${SMOLVM:-}" ]; then
case "$(readlink "$SMOLVM" 2>/dev/null || printf '%s' "$SMOLVM")" in
"$SMOLVM_PREFIX"/*) ;;
*) printf 'note=%s is not under %s, so the process scan cannot see its forked VMs (all of its VMs on macOS); set SMOLVM_PREFIX to its directory\n' "$SMOLVM" "$SMOLVM_PREFIX" ;;
esac
fi
# List this HOME's VM processes as "pid marker". The plain run path execs a
# `_boot-vm` child that carries its boot config; the pack-run path forks one that
# carries none. Linux names both `libkrun VM`, or `VM:<hostname>` when HOSTNAME
# is exported; on macOS the executable path and the parent chain scope the
# search. The teardown packet's traps have the why.
list_vm_processes() {
case "$(uname -s)" in
Linux)
for p in /proc/[0-9]*; do
case "$(cat "$p/comm" 2>/dev/null)" in "libkrun VM"|VM:*) ;; *) continue ;; esac
pid="${p#/proc/}"
cfg="$(tr '\0' '\n' < "$p/cmdline" 2>/dev/null | sed -n '3p')"
case "$cfg" in
"$VMS_DIR"/*) printf '%s %s\n' "$pid" "$cfg"; continue ;;
esac
case "$(readlink "$p/exe" 2>/dev/null)" in
"$SMOLVM_PREFIX"/*) printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
esac
done
;;
Darwin)
# shellcheck disable=SC2009 # pgrep -f would match this script.
procs="$(ps -axo pid=,ppid=,command= 2>/dev/null)"
printf '%s\n' "$procs" | while read -r pid ppid rest; do
case "$rest" in "$SMOLVM_PREFIX"/smolvm-bin*) ;; *) continue ;; esac
case "$rest" in
*" _boot-vm "*) printf '%s %s\n' "$pid" "${rest#* _boot-vm }"; continue ;;
esac
# An orphan counts only if it is a run; a fork has its parent's command line.
if [ "$ppid" = 1 ]; then
case "$rest" in
*" machine run "*|*" vm run "*|*" pack run "*)
printf '%s orphaned-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
esac
else
parent="$(printf '%s\n' "$procs" | while read -r q qp qrest; do [ "$q" = "$ppid" ] && { printf '%s' "$qrest"; break; }; done)"
[ "$parent" = "$rest" ] && printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX"
fi
done
;;
esac
}
# 1. Delete recorded machines. Without --force a delete prompts and defaults to
# No; --cascade removes branch children, which otherwise block it.
if [ -s "$STATE_FILE" ]; then
while read -r name; do
[ -n "$name" ] || continue
case "$name" in "$PREFIX"*) ;; *)
printf 'skipping %s: not created by this packet (no %s prefix)\n' "$name" "$PREFIX"
continue ;;
esac
# A child is gone once its source was deleted with --cascade, and on
# v1.18.2 `machine stop` on a name that does not exist leaves an empty
# directory under vms/. Act only on names still listed.
"$SMOLVM" machine list </dev/null 2>/dev/null | awk 'NR>2{print $1}' > "$STATE_DIR/.listed" 2>/dev/null
grep -qx -- "$name" "$STATE_DIR/.listed" || continue
# Only names still listed: a second stop of a missing name leaves a
# directory that reads as a leak. The list is read first because grep -q
# under pipefail can fail the pipeline and skip a listed machine.
listed="$("$SMOLVM" machine list </dev/null 2>/dev/null | awk 'NR>2{print $1}')"
grep -qx -- "$name" <<<"$listed" || continue
"$SMOLVM" machine stop --name "$name" </dev/null >/dev/null 2>&1
"$SMOLVM" machine delete --name "$name" --force --cascade </dev/null 2>&1 | sed 's/^/ /'
done < "$STATE_FILE"
fi
# 1b. Checkpoints and stores named on the command line, then the restore base.
# A store is known by the objects directory and .lock its first capture makes, not by its name.
for d in "$@"; do
case "$d" in
*.smolcheckpoint|*.checkpoint) ;;
*) if [ ! -d "$d/objects" ] || [ ! -e "$d/.lock" ]; then
printf 'skipping %s: not a .checkpoint, a .smolcheckpoint or a checkpoint store\n' "$d"; continue
fi ;;
esac
[ -e "$d" ] && rm -rf -- "$d" && printf 'removed=%s\n' "$d"
done
for cache in _restore-base _restore-checkpoints; do
if [ "$restore_base" -eq 1 ] && [ -d "$VMS_DIR/$cache" ]; then
rm -rf "${VMS_DIR:?}/$cache" "$VMS_DIR/$cache.lock" && printf 'removed=%s\n' "$VMS_DIR/$cache"
elif [ "$restore_base" -eq 1 ]; then
printf '%s=absent (nothing to remove)\n' "$cache"
elif [ -d "$VMS_DIR/$cache" ]; then
printf '%s=present %s (restored checkpoints kept by smolvm; --restore-base removes it)\n' "$cache" "$(du -sh "$VMS_DIR/$cache" 2>/dev/null | cut -f1)"
fi
done
rm -f "$STATE_DIR/.listed"
# 2. An ephemeral machine's entry retires after its run returns: poll, 20 s at most.
printf 'waiting=up to 20s for ephemeral entries to retire before asserting\n'
waited=0
listing="$("$SMOLVM" machine list 2>&1)"
while ! grep -q 'No machines found' <<<"$listing" && [ "$waited" -lt 20 ]; do
sleep 1; waited=$((waited + 1))
listing="$("$SMOLVM" machine list 2>&1)"
done
printf 'waited=%ss\n' "$waited"
# 3. Assert the value, not the exit code.
if grep -q 'No machines found' <<<"$listing"; then
printf 'machines=clean\n'
[ "$purge" -eq 1 ] && rm -f "$STATE_FILE"
else
printf 'machines=remaining\n'
printf '%s\n' "$listing" | sed 's/^/ /'
printf 'note=this packet did not create these, so it will not delete them. By name:\n'
printf ' smolvm machine stop --name <NAME> && smolvm machine delete --name <NAME> --force\n'
printf ' add --cascade for a machine that was branched from another\n'
fi
# 4. Report VM processes under this HOME's state, such as a killed wrapper's.
# With --reap every one is killed, including a machine another packet kept.
found=0
while read -r pid cfg; do
[ -n "$pid" ] || continue
found=1
printf 'vm_process=%s config=%s\n' "$pid" "$cfg"
if [ "$reap" -eq 1 ]; then
kill -9 "$pid" 2>/dev/null && printf ' killed %s\n' "$pid"
fi
done <<EOF
$(list_vm_processes)
EOF
if [ "$found" -eq 0 ]; then
printf 'vm_processes=none\n'
elif [ "$reap" -eq 0 ]; then
printf 'rerun with --reap to kill them\n'
fi Checkpoint, restore, pause and branch traps
Each entry was hit on v1.18.2 on 2026-09-24 unless it says otherwise.
--branchable is decided at start, and older macOS releases needed it for more than branching
machine branch against a machine started without it, on either host:
Error: agent operation failed: fork: machine 'smolskill-sfx' was not started as branchable, so it
has no copy-on-write memory to branch from. Restart it with `smolvm machine start --name
smolskill-sfx --branchable`; branchability is decided at start time and cannot be turned on for an
already-running machine. That message is clear. The macOS one for checkpoint and pause before v1.20.0 is not:
Error: agent operation failed: checkpoint machine: libkrun save failed: ERR EIO capture VM: VM
snapshot/restore failed: retain COW guest-memory generation: guest RAM has no file-backed regions pause prints the same text, because a pause is a checkpoint underneath, and the machine keeps
running. On Linux aarch64 both succeeded without the flag: a checkpoint of 55 MiB with a 1.242 s
pause, and a pause and resume. On those releases, start every machine you might checkpoint on macOS
with --branchable.
From v1.20.0 macOS takes a checkpoint file and a pause without the flag, measured on v1.20.0,
v1.20.2, v1.21.1, v1.22.0 and v1.22.2, where v1.19.0 and v1.19.3 still refused. A stored capture
needs it through v1.23.7. There --store against a machine started without it fails with:
Error: agent operation failed: checkpoint machine: libkrun save failed: ERR ENOTSUP VM
snapshot/restore failed: retain COW guest-memory generation: deferred durable save requires
file-backed guest RAM From v1.24.0 the capture falls back to a synchronous save (#1584): on v1.24.0 and v1.34.0 the same --store capture published, and a machine restored from it held the source’s RAM token.
--output must end in .checkpoint or .smolcheckpoint, even when it is a directory
From v1.19.1 the file is a .checkpoint in machine checkpoint --help, and on v1.22.2 any other
name fails with output must end in .checkpoint, while .smolcheckpoint is still accepted and machine create --help still names it. The scripts use .smolcheckpoint so they run on older
releases too. On v1.18.2:
Error: config operation failed: checkpoint machine: output must end in .smolcheckpoint for -o ./x.checkpoint, and the same for --store ./st -o ./inc1, where the output is a directory
and the help calls it one.
The exit code of a capture is not the proof
A capture is usable once it has been durably published, which checkpoint-log <output> shows as the ~0 line ending (this checkpoint). scripts/checkpoint.sh reads that back before it deletes any
older checkpoint. During this run the host disk filled and an export failed with tar error: No space left on device (os error 28); a scheduler that trusted a zero exit elsewhere and deleted the
previous checkpoint would have been left with none.
--keep does not bound the store on its own
Each stored checkpoint retains up to --history earlier generations, 32 by default, and deleting
the directory of a generation that a kept checkpoint retains frees nothing. With --keep 2 --history 2 the store grew 59, 76, 95, 114 MB over four captures. references/scheduling.md has
how to pick the two together.
A restored branch point is not a parked source
A checkpoint taken while the source is parked in smolvm-branch-ready holds exactly the state its
children start from. Restoring it does not give you a parked source back: on start the helper
releases, and the workload runs the program after -- with an empty SMOLVM_BRANCH_NAME, so /root/child.txt in that test read CHILD=. A batch branch from the restored machine then waits
its full --ready-timeout, 10 minutes by default, and fails with
A batch branch checkpoints the source at a point its workload declares by running
`smolvm-branch-ready` after setup ... either add the call, raise --ready-timeout, or take single
`--name` branches, which checkpoint the source wherever it is. A single --name branch from the restored machine worked and carried the source’s state. To fan out
again from a kept branch point, branch it one child at a time or restore it once per worker, and
give each its identity some other way, since the branch variables are empty.
Detecting a parked source: pgrep -f always matches
smolvm machine exec --name src -- sh -c 'pgrep -f smolvm-branch-ready' succeeds whether or not the
source is parked, because the sh -c running it has the string in its own command line. Read PID 1
instead, which is the workload:
smolvm machine exec --name src -- sh -c 'tr "\0" " " < /proc/1/cmdline' | grep -q '^smolvm-branch-ready' A parked source shows smolvm-branch-ready -- sh -c ...; a released one shows the program after --.
machine list | grep -q under pipefail
set -o pipefail plus smolvm machine list | grep -q name can fail for a machine that exists: grep -q exits on the first match, the list’s next write hits a closed pipe, and the pipeline’s
status is the list’s failure. Two scripts in this packet reported result=FAILED no machine was created for machines that were running before this was caught. Read the list into a variable, then
search it.
A paused machine refuses stop and start, and a failed resume can be retried
Error: agent operation failed: stop: machine has saved execution; use resume or delete
Error: agent operation failed: start: machine has saved execution; use resume A restored machine paused while the host disk was full failed its resume with extract paused checkpoint: failed to unpack .... After space was freed the same machine resume succeeded, and
the counter the workload keeps in RAM read 14 against 8 before the pause, then kept counting.
A full disk leaves machines that delete --force will not remove
A machine restored and started while the disk was full came up with its overlay failing. Its delete, later, with space available:
Error: agent operation failed: stop agent: guest did not confirm filesystem synchronization; left
the VM alive for retry: agent operation failed: shutdown ack: freeze /oldroot/mnt/overlay: I/O error
(os error 5) exit 1, still running. scripts/cleanup.sh --reap killed its VM process, after which delete --force removed it. --reap kills every VM process under this HOME, not only that one, so
use it when nothing else there should keep running. scripts/preflight.sh checks free space for
this reason.
du overstates what a restore costs
Restores are copy-on-write over the checkpoint. On macOS three restores of one checkpoint moved the
volume’s free space by 22 MB at create and 65 MB at start, about 29 MB a machine, while du on
each machine’s directory said 150 to 190 MB. On Linux each restored machine’s directory holds small qcow2 layers over .smolcheckpoint-*.raw bases, 29 MB by du. Measure a restore budget with df before and after.
smolvm keeps restored checkpoints after you delete them
On macOS vms/_restore-base under smolvm’s cache is, in the source’s words, a pristine clone of the
most recently restored checkpoint, kept so the next restore writes only the chunks that differ. It
was 243 MB here, with the checkpoint’s memory.bin inside, and it stayed after every machine and
every checkpoint file was deleted. smolvm serve start’s reclaim did not touch it. scripts/cleanup.sh --restore-base removes it. None was created on Linux aarch64.
From v1.22.0 the cache is vms/_restore-checkpoints, which keeps the last three restored
checkpoints, up to 16 GiB, so jumping back to one clones it instead of rebuilding its RAM. After
this packet’s procedure on macOS on v1.23.0 it held 258 MB with every machine and checkpoint
deleted, and no _restore-base was made. machine create --from takes --restore-cache-entries and --restore-cache-gib; 0 entries turns it off. scripts/cleanup.sh --restore-base removes
both.
machine stop on a missing name leaves an empty directory
On v1.18.2, smolvm machine stop --name <name> for a machine that does not exist leaves an empty
directory under the VM cache. A cleanup that stops each recorded name after --cascade has already
deleted the children leaves one per child, which a leak check counting directories then reports.
This packet’s cleanup stops only names still listed.
Environment given to a restore reaches exec, not the resumed workload
machine create --from <checkpoint> -e ROLE=seven: machine exec saw ROLE=seven, and the resumed
workload’s own environment did not have it. The workload is the process that was running when the
checkpoint was taken, and its environment came with it.
Corrections
2026-10-15, v1.29.0 run. “Read PID 1 instead, which is the workload” above stopped holding in v1.25.4 (#1632): PID 1 is now
/run/smolvm/init container-initand the workload is its child, so the check never saw a parked source andbranch-point.sh --countfailed withthe source never parked. Read every process instead:smolvm machine exec --name src -- sh -c 'for f in /proc/[0-9]*/cmdline; do tr "\0" " " < "$f"; echo; done' | grep -q '^smolvm-branch-ready'scripts/branch-point.shdoes this, and on v1.29.0 a batch of two then branched with the same RAM token in both children and in a restore of the branch point.