Dev env: a persistent machine you re-enter
Keeps a persistent smolvm machine with its dependencies already installed and re-enters it cheaply across sessions. Use when a project needs an isolated development environment that survives stop and start; when deciding what belongs in a Smolfile’s init versus what has to run on every boot; when a package installed in a machine has vanished after a restart; or when exec answers “the container smolvm-
Verified on smolvm v1.34.0 on macOS arm64 and Linux x86_64, 2026-11-09, and on v1.18.2 on Linux aarch64, 2026-09-24; the aarch64 runs used
the Smolfile with memory = 1024, for the host reason in “Platform arms”. Done means a second start is fast, skips provisioning, and the packages installed in
the first session are still there.
The Linux aarch64 runs used the scripts of their date; this version’s preflight and cleanup scripts ran
on Linux aarch64 on v1.22.2 on 2026-10-03.
The whole use case turns on one fact: init runs once, not on every start, as smolfile.md says, and as the docs site (smolmachines.com/docs) says under “When
init runs”. smolvm machine create --help reads “Run command on every VM start” at v1.22.2, and
“Run command once, on the machine’s first start” from v1.23.7; provisioning that relies on the old
line is missing on the second boot.
Procedure
1. Preflight.
scripts/preflight.sh Read-only. restart_after_stop=verified on macOS and Linux. The script covers macOS and Linux
only; elsewhere it prints unknown and a note pointing at references/windows.md, where a stopped
machine starts again on v1.14.6 and v1.22.2.
2. Declare the machine. assets/dev.smolfile is a working starting point, and the one verify-persistence.sh is written against: it asserts the app user and the /app workdir that
file sets. examples/python-app/python.smolfile and examples/node-app/node.smolfile in the
repository are plainer examples without them.
image = "python:3.12-alpine"
net = true
cpus = 2
memory = 2048
volumes = ["./src:/app"]
init = [
"sh -c \"id -un > /init-ran-as.txt\"",
"sh -c \"date +%s >> /init-count.txt\"",
"adduser -D app",
]
user = "app"
workdir = "/app" 3. Create and bring it up. Run this from the project directory: the Smolfile mounts ./src,
which is relative to the working directory, and the script creates ./src there if it is missing.
scripts/create-dev-machine.sh # name defaults to smolskill-dev
scripts/create-dev-machine.sh smolskill-myproj ./my.smolfile It creates the machine with an explicit long-lived workload command, records the name for cleanup, starts it, waits for the workload container to answer with a value, and then reports what actually happened rather than that nothing errored:
init_ran=yes
machine_running=yes
workload_ready_after_s=0
workload_ready=yes
init_ran_as=root
exec_user=app
workdir=/app
result=up init_ran_as=root with exec_user=app is the correct outcome, not a bug: init provisions the
machine and runs as root regardless of user, while exec and shell run as user.
4. Work in it.
smolvm machine exec --name smolskill-dev -- pip install --user requests
smolvm machine exec --name smolskill-dev --user root -- sh -c 'mkdir -p /storage/keep'
smolvm machine shell --name smolskill-dev # interactive, lands in workdir as `user` 5. Prove it is worth keeping.
scripts/verify-persistence.sh It installs a package and records its version, seeds one file per filesystem, stops, starts, and then asserts each value against what it recorded. A version comparison is the point: an import that does not crash can be satisfied by a system copy and says nothing about your install. It leaves the machine running.
When the setup is done, smolvm machine stop --name smolskill-dev and give the user its name; the
next machine start brings it back as it was.
6. Clean up, when the machine is no longer wanted. Not at the end of a setup: the machine is the deliverable, so leave it stopped and tell the user its name.
scripts/cleanup.sh --purge cleanup.sh waits up to 20 seconds, polling the machine list, and prints waiting=up to 20s first: an ephemeral machine’s entry retires after its run returns.
What init and user actually do, observed on v1.14.2 and again on v1.22.2
Observed, not inferred:
| question | observed |
|---|---|
when does init run? | on the first start, not on create, and not again |
| second start | prints Init already completed, skipping N command(s) |
which user runs init? | root, even with user = "app" set |
which user runs exec and shell? | the Smolfile user |
| override per command | machine exec --user root works |
is there machine start --init? | no |
What survives a stop and start
| location | survives | why |
|---|---|---|
pip --user packages | yes | under $HOME, on the overlay |
$HOME/... files | yes | overlay |
files written at / | yes | overlay |
/tmp | no | emptied at each start, like a tmpfs |
/storage/... | yes | the ext4 disk itself |
The root filesystem is an overlay whose upper layer lives on the machine’s ext4 disk, which is why
writes to / persist while tmpfs mounts and /tmp do not.
Traps
Full detail in references/traps.md. The ones that cost the most:
initruns once (top of this file). Anything that must hold on every boot, a bind mount above all, runs in the command that needs it.- Give a machine you
execinto a long-lived workload. Without one, anexecright afterstartcan answerthe container `smolvm-<hash>` is not runninginstead of running. machine shelldoes not start a stopped machine on v1.22.2, while its help says “starts it if stopped”.- A machine that runs Tailscale or another carrier NAT VPN needs
--guest-subnetat create, which a Smolfile cannot carry:smolvm machine create --name smolskill-dev -s dev.smolfile --guest-subnet 10.200.0.0/30. createis instant and proves nothing, and a hostvolumesmount was not writable by theappuser.
Security defaults, and why they are the defaults
userin the Smolfile is what your workload runs as, and it is not root.initrunning as root is the provisioning step, deliberately separated from the workload’s identity. Keep them separate rather than settinguser = "root"to make a mount writable: the mount carries host ownership, and running the workload as root to reach it hands root in the guest everything the mount exposes on the host.- A
volumesmount is host authority handed to the guest, so mount the narrowest directory that works and prefer:rofor anything the machine only reads. Treat root in the guest as untrusted, as the security model says: every forwarded mount, port and network permission becomes part of the workload’s authority. net = trueis outbound access for the whole machine. A dev machine that only needs a package index does not need general egress;--allow-hostscopes it.- These scripts wrap the public CLI only. They edit no smolvm configuration and nothing under
~/.smolvm, and cleanup deletes only names it recorded under thesmolskill-prefix.
Platform arms
- Linux aarch64: the scripts were run here on v1.18.2 with the Smolfile’s memory at 1024 MiB,
and once on v1.22.2 at 1024. At 2048 on v1.18.2 the restart failed with
agent did not become ready within 30 seconds: on a small or busy host lowermemorybefore suspecting smolvm, and check the mount source first (theinstallpacket’s traps have the numbers). - macOS arm64: the scripts were run here on v1.34.0 as shipped, and every check passed.
- Linux x86_64: verified on v1.34.0, twice, on an NVIDIA A10 cloud host, with the Smolfile as shipped.
- Windows x86_64:
references/windows.md, re-run on 2026-11-09 against v1.34.0 on Windows 11 Home build 10.0.26200 UBR 9711. Create, three stops and three starts, with a marker read back after each start: a stopped machine starts again and its state survives, as on v1.14.6, where on v1.14.2 it did not. That page also carries the WHP device ceiling a Windows preflight needed through v1.22.2, four-vmounts and three when a port is published; v1.34.0 boots eight with two ports.
Eval prompts, and what they produced
1. “Set me up a Python dev machine I can come back to, and prove the packages survive a restart.” On macOS arm64: the step 3 output above, on v1.22.2.
Then, after the stop and start, verify-persistence.sh:
workload_ready_after_s=0
installed_version=2.34.2
Init already completed, skipping 3 command(s)
workload_ready_after_s=0
init_ran_once=ok (yes)
package_version=ok (2.34.2)
home_file=ok (SURVIVES)
storage_file=ok (SURVIVES)
tmp_file=ok (WIPED)
exec_user=ok (app)
workdir=ok (/app)
result=persistent 2. “My bind mount is gone after I restarted the machine. It is right there in init.”
init ran once. The second start reports it, on v1.22.2 as in eval 1:
Init already completed, skipping 3 command(s) There is no machine start --init, so the mount has to be re-applied by the command that needs
it. docker-in-machine is the packet built entirely around this.
3. ”smolvm machine exec just told me the container is not running, but the machine is
running.”
On v1.14.2, reproduced deliberately by creating the same machine without a workload command, then running 20 execs, on Lima:
3: the container `smolvm-e51bb94127b751bf` is not running
failures=1/20
4: the container `smolvm-32edffb50eaf0d20` is not running
failures_after_restart=1/20 and with the explicit long-lived workload scripts/create-dev-machine.sh passes:
failures=0/20
failures_after_restart=0/20 The changing hash is the tell: the workload container is being relaunched between execs.
Re-verified on v1.34.0
Run 2026-11-09 PT against v1.34.0 from the published release, checksum checked, under a fresh
isolated HOME on macOS 27.2 arm64 (Apple M4), twice, the second time from a fresh HOME, and once
on Lima linux-kvm (Ubuntu 24.04 aarch64, 11 GiB). Every step on both hosts, with the Smolfile as
shipped: result=up, init_ran_once=ok (yes), package_version=ok (2.34.2), tmp_file=ok (WIPED) and result=persistent.
On Linux x86_64 (Ubuntu 22.04, KVM, an NVIDIA A10 cloud host with 30 cores and 222 GiB), twice on
2026-11-09, each pass from a fresh HOME, every step with the Smolfile as shipped: result=up, init_ran_once=ok (yes), package_version=ok (2.34.2), tmp_file=ok (WIPED) and result=persistent.
Re-verified on v1.29.0
Run 2026-10-15 PT against v1.29.0 from the published release, checksum checked, under a fresh
isolated HOME on macOS 27.0.1 arm64, the scripts twice, the second time from a fresh HOME, and
the checks beyond them once. Linux was not run, so the Linux stamp stays on its earlier release.
create-dev-machine.sh gave result=up with init_ran_as=root, exec_user=app and workdir=/app; verify-persistence.sh gave init_ran_once=ok (yes), package_version=ok (2.34.2), home_file=ok (SURVIVES), storage_file=ok (SURVIVES), tmp_file=ok (WIPED) and result=persistent, and the second start printed Init already completed, skipping 3 command(s).
Re-verified on v1.22.2
Run 2026-10-03 PT against v1.22.2 from the published release, checksum checked, under an isolated HOME on macOS 27.0.1 arm64, twice, the second time from a fresh HOME. On Lima linux-kvm (Ubuntu 24.04 aarch64) guests above 2048 MiB timed out on 2026-10-03, so the Linux lines below are
a single run and the Linux stamp stays on its earlier release.
macOS: result=up with init_ran_as=root and exec_user=app, then result=persistent, package_version=ok (2.34.2), tmp_file=ok (WIPED). machine create --help describes --init as
running “on every VM start”, machine shell on a stopped machine answered is not running. Use 'smolvm machine start ...' first while its help says it starts one, and the exec race gave 0 of 20.
Linux aarch64 at memory = 1024: result=persistent, the exec race 0 of 20. Linux was last
verified on v1.18.2, at memory = 1024.
What was not run
- Windows beyond the restart. The v1.22.2 run covered create, three stops and starts, the
marker and
initrunning once; the WHP device ceiling was confirmed by theinstallpacket’s Windows run on 2026-10-03. initafter a checkpoint restore. Nowhere, on any platform.- The
USERinteraction still settling.initruns as root on v1.22.2, and whether the image’s ownUSERstill governs it in some paths is open as smolvm#1189. Nothing here tested an image with aUSERline.
Related packets
installfor the boot this assumes, andteardownfor the cleanup script.docker-in-machinefor the clearest case of theinit-runs-once trap.packfor turning the machine this packet builds into a portable artifact.
Scripts
The files the procedure runs, in the order it runs them. It calls each one by the path in its heading, relative to the directory the procedure is saved in.
scripts/preflight.sh
#!/usr/bin/env bash
# Report whether this host can keep a persistent smolvm dev machine. Read-only:
# starts no VM, writes no smolvm state, changes no group membership.
#
# Output is one key=value per line so a caller can parse it. The last line is
# always result=ready or result=blocked.
set -uo pipefail
VERIFIED_VERSION="1.34.0"
emit() { printf '%s=%s\n' "$1" "$2"; }
blocked=0
note() { printf 'note=%s\n' "$1"; }
# --- the binary --------------------------------------------------------------
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
emit smolvm_installed no
emit smolvm_version ""
blocked=1
else
emit smolvm_installed yes
version="$("$SMOLVM" --version 2>/dev/null | awk '{print $NF}')"
emit smolvm_version "${version:-unknown}"
fi
emit verified_version "$VERIFIED_VERSION"
if [ -n "${version:-}" ] && [ "$version" != "unknown" ]; then
if [ "$version" = "$VERIFIED_VERSION" ]; then
emit version_status match
else
newest="$(printf '%s\n%s\n' "$version" "$VERIFIED_VERSION" | sort -V | tail -1)"
if [ "$newest" = "$version" ]; then
emit version_status newer
note "this packet was verified on $VERIFIED_VERSION and the binary is $version; flags and messages move every release, so check the output against the binary before trusting a step here"
else
emit version_status older
note "this packet was verified on $VERIFIED_VERSION and the binary is $version"
fi
fi
else
emit version_status unknown
fi
# --- platform ----------------------------------------------------------------
kernel="$(uname -s)"
arch="$(uname -m)"
case "$arch" in aarch64|arm64) arch=aarch64 ;; esac
case "$kernel" in
Darwin)
emit platform "darwin-$arch"
emit accel hvf
emit macos_version "$(sw_vers -productVersion)"
hv="$(sysctl -n kern.hv_support 2>/dev/null)"
if [ "$hv" = "1" ]; then emit accel_access ok; else emit accel_access denied; blocked=1; fi
if [ "$arch" != "aarch64" ]; then
emit hardware_verified no
note "Intel Mac is not verified by this packet; the installer accepts it and nothing here was run on one"
else
emit hardware_verified yes
fi
# A VM's agent socket lives under the cache directory. macOS sockaddr_un
# holds 104 bytes including the terminator.
sock="$HOME/Library/Caches/smolvm/vms/0123456789abcdef/agent.sock"
len=${#sock}
emit socket_path_bytes "$len"
if [ "$len" -gt 100 ]; then
emit socket_path_status too_long
blocked=1
note "HOME is too deep: every VM start will fail. From v1.28.0 the error names the socket path and its length; before it, krun_start_enter -22, whose text blames disks and device options. Install under a shorter HOME."
else
emit socket_path_status ok
fi
emit unsupported "cuda"
emit restart_after_stop verified
;;
Linux)
emit platform "linux-$arch"
emit accel kvm
emit socket_path_status n_a
if [ ! -e /dev/kvm ]; then
emit accel_access missing
blocked=1
note "/dev/kvm does not exist; this host has no KVM"
elif [ -r /dev/kvm ] && [ -w /dev/kvm ]; then
emit accel_access ok
else
emit accel_access denied
blocked=1
note "your user cannot open /dev/kvm. The installer warns and continues, so a successful install says nothing about whether a VM will start. Fix: sudo usermod -aG kvm \$USER, then run the next command through sg kvm -c '...' rather than logging out."
fi
emit restart_after_stop verified
;;
*)
emit platform "unsupported-$kernel"
emit accel unknown
emit accel_access unknown
blocked=1
emit restart_after_stop unknown
note "this script covers macOS and Linux. On Windows a stopped machine starts again, on v1.14.6 and v1.22.2, and through v1.22.2 the WHP device budget caps you at four -v mounts, three when a port is published. See references/windows.md."
;;
esac
if [ "$blocked" -eq 0 ]; then emit result ready; else emit result blocked; fi assets/dev.smolfile
# A persistent development machine. `smolvm machine create -s dev.smolfile`.
#
# The two keys worth understanding before you edit this:
#
# init runs ONCE, on the first start, and as root. It does not run again on
# later starts, so anything that has to be true on every boot does not
# belong here.
# user governs `exec` and `shell`, not `init`.
#
# The workload command is not set here. A Smolfile can carry one as `cmd`;
# this packet passes it after `--` on `machine create`, as
# scripts/create-dev-machine.sh does: without one the image's own CMD becomes
# the persistent workload, and for an interpreter image that command exits at
# once and `exec` starts racing the relaunch.
image = "python:3.12-alpine"
net = true
cpus = 2
memory = 2048
# A host mount carries host ownership, so `user` below (made by adduser) could
# not write to it. Read source in, write build output somewhere else.
volumes = ["./src:/app"]
init = [
"sh -c \"id -un > /init-ran-as.txt\"",
"sh -c \"date +%s >> /init-count.txt\"",
"adduser -D app",
]
user = "app"
workdir = "/app" scripts/create-dev-machine.sh
#!/usr/bin/env bash
# Create a persistent dev machine and bring it up for the first time.
#
# usage: create-dev-machine.sh [<name>] [<smolfile>]
# name default smolskill-dev (the prefix cleanup.sh will delete)
# smolfile default ../assets/dev.smolfile
#
# `create` is configuration only: it returns in milliseconds and pulls nothing. It
# does refuse a host mount source that does not exist; the pull, the init commands
# and any of their failures land on the first `start`, so a fast create is not
# evidence that anything works.
set -uo pipefail
here="$(cd "$(dirname "$0")" && pwd)"
NAME="${1:-smolskill-dev}"
SMOLFILE="${2:-$here/../assets/dev.smolfile}"
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
printf 'smolvm not found; set SMOLVM to its path\n' >&2
exit 2
fi
case "$NAME" in
smolskill-*) ;;
*) printf 'name must start with smolskill- so cleanup.sh will delete it\n' >&2; exit 2 ;;
esac
# The Smolfile mounts ./src, which is relative to the working directory.
mkdir -p ./src
# Give the machine a workload that stays up. Without a command, `create` launches
# the image's own ENTRYPOINT/CMD as the persistent workload; for an interpreter
# image such as python:3.12-alpine that command reads EOF and exits at once, the
# container is relaunched, and `exec` then intermittently answers "the container
# `smolvm-<hash>` is not running" instead of running your command. Measured on a
# nested-virt aarch64 host: 1 exec in 20 failed that way with no command, 0 in 20
# with this one, both on a fresh machine and immediately after a restart.
create_out="$("$SMOLVM" machine create --name "$NAME" -s "$SMOLFILE" \
-- sh -c 'while true; do sleep 3600; done' 2>&1)"
create_rc=$?
printf '%s\n' "$create_out" | sed 's/^/ /'
if [ "$create_rc" -ne 0 ]; then printf 'result=failed (create)\n'; exit 1; fi
"$here/cleanup.sh" --record "$NAME"
start_out="$("$SMOLVM" machine start --name "$NAME" 2>&1)"
printf '%s\n' "$start_out" | sed 's/^/ /'
fail=0
# `start` prints the init line only when the Smolfile declares init commands.
if printf '%s' "$start_out" | grep -q 'Running .* init command'; then
printf 'init_ran=yes\n'
elif ! grep -q '^[[:space:]]*init[[:space:]]*=' "$SMOLFILE"; then
printf 'init_ran=none_declared\n'
else
printf 'init_ran=no\n'
fail=1
fi
if printf '%s' "$start_out" | grep -q "Machine '$NAME' running"; then
printf 'machine_running=yes\n'
else
printf 'machine_running=no\n'
fail=1
fi
# `start` returns before the workload container is up. Wait for a VALUE: an empty
# result and a zero exit code both pass while the container is still coming up,
# so a probe that asserts either of those reports ready too early and the next
# three checks read a message as their answer.
wait_for_workload() {
# 120 s: the slowest first start observed while building this packet was
# under 20 s on a nested-virt aarch64 host, so this is six times the worst
# case and still short enough that a real hang is reported, not waited on.
waited=0
while [ "$waited" -lt 120 ]; do
out="$("$SMOLVM" machine exec --name "$1" -- sh -c 'echo WORKLOAD_READY' 2>&1 | tr -d '\r')"
if [ "$out" = "WORKLOAD_READY" ]; then
printf 'workload_ready_after_s=%s\n' "$waited"
return 0
fi
sleep 1
waited=$((waited + 1))
done
printf 'workload_ready_after_s=timeout last_probe=%s\n' "$out"
return 1
}
if wait_for_workload "$NAME"; then
printf 'workload_ready=yes\n'
else
printf 'workload_ready=no\n'
fail=1
fi
# init runs as root even with `user` set. Asserting the value rather than the
# absence of an error is the point: an exec that fails still leaves you guessing.
printf 'init_ran_as=%s\n' "$("$SMOLVM" machine exec --name "$NAME" --user root -- cat /init-ran-as.txt 2>&1 | tr -d '\r')"
printf 'exec_user=%s\n' "$("$SMOLVM" machine exec --name "$NAME" -- id -un 2>&1 | tr -d '\r')"
printf 'workdir=%s\n' "$("$SMOLVM" machine exec --name "$NAME" -- pwd 2>&1 | tr -d '\r')"
if [ "$fail" -eq 0 ]; then printf 'result=up\n'; else printf 'result=failed\n'; fi
exit "$fail" scripts/verify-persistence.sh
#!/usr/bin/env bash
# Prove the machine is worth keeping: that a package installed in one session is
# still there in the next, that init does not run again, and that the parts of
# the filesystem people assume persist actually do.
#
# usage: verify-persistence.sh [<name>] (default smolskill-dev)
#
# Every check asserts a value. Asserting that an import did not crash proves
# nothing here: a system copy of the package satisfies it and tells you nothing
# about whether your install survived.
set -uo pipefail
NAME="${1:-smolskill-dev}"
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
printf 'smolvm not found; set SMOLVM to its path\n' >&2
exit 2
fi
fail=0
check() {
if [ "$2" = "$3" ]; then
printf '%s=ok (%s)\n' "$1" "$2"
else
printf '%s=FAIL expected=%s actual=%s\n' "$1" "$3" "$2"
fail=1
fi
}
exec_in() { "$SMOLVM" machine exec --name "$NAME" -- "$@" 2>&1 | tr -d '\r'; }
# `start` returns before the workload container is up, and an `exec` in that
# window answers "the container `smolvm-<hash>` is not running" on stdout while
# still exiting zero. Wait for a VALUE: an empty result or a zero exit code both
# pass while the container is still coming up, which is how this window gets
# missed. Observed on a nested-virt aarch64 host, where a probe that asserted
# only an empty result let three later checks read that message as their answer.
wait_for_workload() {
# 120 s: six times the slowest start observed while building this packet.
# Long enough to absorb a loaded host, short enough to report a real hang.
waited=0
while [ "$waited" -lt 120 ]; do
probe="$(exec_in sh -c 'echo WORKLOAD_READY')"
if [ "$probe" = "WORKLOAD_READY" ]; then
printf 'workload_ready_after_s=%s\n' "$waited"
return 0
fi
sleep 1
waited=$((waited + 1))
done
printf 'workload_ready_after_s=timeout last_probe=%s\n' "$probe"
return 1
}
if ! wait_for_workload; then
printf 'result=FAILED (workload container never came up)\n'
exit 1
fi
# 1. Install something and record its version, not its presence.
"$SMOLVM" machine exec --name "$NAME" -- pip install --quiet --user requests >/dev/null 2>&1
before="$(exec_in python3 -c 'import requests; print(requests.__version__)')"
printf 'installed_version=%s\n' "$before"
case "$before" in [0-9]*) ;; *) printf 'installed_version=FAIL (not a version)\n'; fail=1 ;; esac
# 2. Seed one file per filesystem so the stop tells them apart.
# shellcheck disable=SC2016 # $HOME must expand inside the guest, not here
"$SMOLVM" machine exec --name "$NAME" -- sh -c 'echo SURVIVES > "$HOME/keep.txt"; echo GONE > /tmp/scratch.txt' >/dev/null 2>&1
"$SMOLVM" machine exec --name "$NAME" --user root -- sh -c 'mkdir -p /storage/keep && echo SURVIVES > /storage/keep/disk.txt' >/dev/null 2>&1
# 3. Stop and come back.
"$SMOLVM" machine stop --name "$NAME" >/dev/null 2>&1
restart_out="$("$SMOLVM" machine start --name "$NAME" 2>&1)"
printf '%s\n' "$restart_out" | sed 's/^/ /'
if ! wait_for_workload; then
fail=1
fi
if printf '%s' "$restart_out" | grep -q 'Init already completed'; then
check init_ran_once yes yes
else
check init_ran_once no yes
fi
after="$(exec_in python3 -c 'import requests; print(requests.__version__)')"
check package_version "$after" "$before"
# shellcheck disable=SC2016 # same: the guest resolves $HOME
check home_file "$(exec_in sh -c 'cat "$HOME/keep.txt" 2>/dev/null')" SURVIVES
check storage_file "$(exec_in cat /storage/keep/disk.txt)" SURVIVES
# /tmp is emptied by a stop. Anything a provisioning step leaves
# there is gone on the next start, while the same step's writes to $HOME or /
# persist. This is the asymmetry that surprises people.
check tmp_file "$(exec_in sh -c 'cat /tmp/scratch.txt 2>/dev/null || echo WIPED')" WIPED
check exec_user "$(exec_in id -un)" app
check workdir "$(exec_in pwd)" /app
if [ "$fail" -eq 0 ]; then printf 'result=persistent\n'; else printf 'result=FAILED\n'; fi
exit "$fail" scripts/cleanup.sh
#!/usr/bin/env bash
# Delete the machines this packet's scripts created, then prove the host is clean.
#
# Only machines recorded in the state file are deleted, with --cascade, so any
# machine branched from one of them goes too, whatever its name. Any other
# machine you or another session created by hand is never deleted. Scripts
# record a name by calling: cleanup.sh --record <name>
#
# usage: cleanup.sh [--record <name>] [--reap] [--purge]
# --record <name> add a machine name to the state file and exit
# --reap kill every VM process under this HOME; run without it first
# --purge also remove the state file once the list is empty
set -uo pipefail
PACKET="dev-env"
PREFIX="smolskill-"
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
STATE_DIR="${SMOLVM_SKILL_STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/smolvm-skills}"
STATE_FILE="$STATE_DIR/$PACKET.machines"
reap=0
purge=0
while [ $# -gt 0 ]; do
case "$1" in
--record)
mkdir -p "$STATE_DIR"
printf '%s\n' "$2" >> "$STATE_FILE"
exit 0
;;
--reap) reap=1 ;;
--purge) purge=1 ;;
*) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
esac
shift
done
if [ -z "$SMOLVM" ]; then
printf 'smolvm not found; set SMOLVM to its path\n' >&2
exit 2
fi
# With SMOLVM_DATA_DIR set, smolvm runs with HOME there on Linux, so its cache
# is under .cache in it.
case "$(uname -s)" in
Darwin) VMS_DIR="$HOME/Library/Caches/smolvm/vms" ;;
*) cache_root="${SMOLVM_DATA_DIR:+$SMOLVM_DATA_DIR/.cache}"
VMS_DIR="${cache_root:-${XDG_CACHE_HOME:-$HOME/.cache}}/smolvm/vms" ;;
esac
VMS_DIR="${SMOLVM_VMS_DIR:-$VMS_DIR}"
SMOLVM_PREFIX="${SMOLVM_PREFIX:-$HOME/.smolvm}"
if [ -n "${SMOLVM:-}" ]; then
case "$(readlink "$SMOLVM" 2>/dev/null || printf '%s' "$SMOLVM")" in
"$SMOLVM_PREFIX"/*) ;;
*) printf 'note=%s is not under %s, so the process scan cannot see its forked VMs (all of its VMs on macOS); set SMOLVM_PREFIX to its directory\n' "$SMOLVM" "$SMOLVM_PREFIX" ;;
esac
fi
# List this HOME's VM processes as "pid marker". The plain run path execs a
# `_boot-vm` child that carries its boot config; the pack-run path forks one that
# carries none. Linux names both `libkrun VM`, or `VM:<hostname>` when HOSTNAME
# is exported; on macOS the executable path and the parent chain scope the
# search. The teardown packet's traps have the why.
list_vm_processes() {
case "$(uname -s)" in
Linux)
for p in /proc/[0-9]*; do
case "$(cat "$p/comm" 2>/dev/null)" in "libkrun VM"|VM:*) ;; *) continue ;; esac
pid="${p#/proc/}"
cfg="$(tr '\0' '\n' < "$p/cmdline" 2>/dev/null | sed -n '3p')"
case "$cfg" in
"$VMS_DIR"/*) printf '%s %s\n' "$pid" "$cfg"; continue ;;
esac
case "$(readlink "$p/exe" 2>/dev/null)" in
"$SMOLVM_PREFIX"/*) printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
esac
done
;;
Darwin)
# shellcheck disable=SC2009 # pgrep -f would match this script.
procs="$(ps -axo pid=,ppid=,command= 2>/dev/null)"
printf '%s\n' "$procs" | while read -r pid ppid rest; do
case "$rest" in "$SMOLVM_PREFIX"/smolvm-bin*) ;; *) continue ;; esac
case "$rest" in
*" _boot-vm "*) printf '%s %s\n' "$pid" "${rest#* _boot-vm }"; continue ;;
esac
# An orphan counts only if it is a run; a fork has its parent's command line.
if [ "$ppid" = 1 ]; then
case "$rest" in
*" machine run "*|*" vm run "*|*" pack run "*)
printf '%s orphaned-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
esac
else
parent="$(printf '%s\n' "$procs" | while read -r q qp qrest; do [ "$q" = "$ppid" ] && { printf '%s' "$qrest"; break; }; done)"
[ "$parent" = "$rest" ] && printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX"
fi
done
;;
esac
}
# 1. Delete recorded machines. Without --force a delete prompts and defaults to
# No; --cascade removes branch children, which otherwise block it.
if [ -s "$STATE_FILE" ]; then
while read -r name; do
[ -n "$name" ] || continue
case "$name" in "$PREFIX"*) ;; *)
printf 'skipping %s: not created by this packet (no %s prefix)\n' "$name" "$PREFIX"
continue ;;
esac
# Only names still listed: a second stop of a missing name leaves a
# directory that reads as a leak. The list is read first because grep -q
# under pipefail can fail the pipeline and skip a listed machine.
listed="$("$SMOLVM" machine list </dev/null 2>/dev/null | awk 'NR>2{print $1}')"
grep -qx -- "$name" <<<"$listed" || continue
"$SMOLVM" machine stop --name "$name" </dev/null >/dev/null 2>&1
"$SMOLVM" machine delete --name "$name" --force --cascade </dev/null 2>&1 | sed 's/^/ /'
done < "$STATE_FILE"
fi
# 2. An ephemeral machine's entry retires after its run returns: poll, 20 s at most.
printf 'waiting=up to 20s for ephemeral entries to retire before asserting\n'
waited=0
listing="$("$SMOLVM" machine list 2>&1)"
while ! grep -q 'No machines found' <<<"$listing" && [ "$waited" -lt 20 ]; do
sleep 1; waited=$((waited + 1))
listing="$("$SMOLVM" machine list 2>&1)"
done
printf 'waited=%ss\n' "$waited"
# 3. Assert the value, not the exit code.
if grep -q 'No machines found' <<<"$listing"; then
printf 'machines=clean\n'
[ "$purge" -eq 1 ] && rm -f "$STATE_FILE"
else
printf 'machines=remaining\n'
printf '%s\n' "$listing" | sed 's/^/ /'
printf 'note=this packet did not create these, so it will not delete them. By name:\n'
printf ' smolvm machine stop --name <NAME> && smolvm machine delete --name <NAME> --force\n'
printf ' add --cascade for a machine that was branched from another\n'
fi
# 4. Report VM processes under this HOME's state, such as a killed wrapper's.
# With --reap every one is killed, including a machine another packet kept.
found=0
while read -r pid cfg; do
[ -n "$pid" ] || continue
found=1
printf 'vm_process=%s config=%s\n' "$pid" "$cfg"
if [ "$reap" -eq 1 ]; then
kill -9 "$pid" 2>/dev/null && printf ' killed %s\n' "$pid"
fi
done <<EOF
$(list_vm_processes)
EOF
if [ "$found" -eq 0 ]; then
printf 'vm_processes=none\n'
elif [ "$reap" -eq 0 ]; then
printf 'rerun with --reap to kill them\n'
fi Persistent dev machine traps
init runs once, not on every start
Observed on v1.14.2 on macOS arm64 and Linux aarch64: init runs on the first start, not
on create, and never again. The second start prints Init already completed, skipping N command(s).
Anything that must be true on every boot does not belong in init. The most common victim is
a bind mount: put mount --bind ... in init and the second boot comes up without it. Re-apply
it in the same command that needs it. The docker-in-machine packet is built around this.
The two help texts read differently. At v1.22.2 smolvm machine create --help describes --init as “Run command on every VM start”, and smolvm machine run --help as “Run command before
the workload … the same as init in a Smolfile”. smolfile.md says init “runs once as root, like a Dockerfile RUN”, and the docs site’s introduction/concepts/smolfile.md (smolmachines.com/docs) says under “When init runs” that it runs once, on the first start. Observed:
once, on the first start. From v1.23.7 machine create --help says the same: “Run command once, on
the machine’s first start”.
init also runs as root, even with user set: /init-ran-as.txt contained root while exec and shell ran as app. There is no machine start --init to force a re-run.
create is not where the time goes, and not where most failures appear
It is pure configuration and returns in milliseconds without touching the registry. The pull, the
init commands and any of their failures all land on the first start. Do not read a fast create as evidence that anything works. It does refuse a host mount source that does not exist;
see below.
create pulls nothing either; the image is pulled on the first start.
exec right after start can answer with a message instead of running
If you see the container `smolvm-<hash>` is not running, the workload container is not up
and the exec did nothing. The hash differs between occurrences, because the container is being
relaunched.
Why it happens. Without a command, machine create launches the image’s own ENTRYPOINT or CMD
as the persistent workload (machine create --help, [COMMAND]). For an interpreter image such
as python:3.12-alpine that command reads EOF from a stdin nobody is holding and exits at once,
so the container dies and is relaunched, and an exec can land in the gap.
Measured on a nested-virt aarch64 host on 2026-09-07, 20 execs each on a fresh machine and again immediately after a restart:
| workload command | failures, fresh | failures, after restart |
|---|---|---|
| none (image CMD) | 1 in 20 | 1 in 20 |
sh -c 'while true; do sleep 3600; done' | 0 in 20 | 0 in 20 |
On v1.18.2, 2026-09-24, the no-command row was 0 in 20 on macOS arm64 and 1 in 20 on Lima aarch64, where it had been 0 in 20 on both on v1.16.1.
The fix is to give the machine a workload that stays up, which is what scripts/create-dev-machine.sh does by passing one after -- on machine create. A Smolfile can
carry it as well, as cmd (with entrypoint), and a command after -- replaces the Smolfile’s cmd.
And this is why the readiness probe has to assert a value. The message goes to stdout, so a
probe that waits for empty output or a zero exit code reports ready while the container is still
flapping, and the next three checks silently read that message as their answer. Both scripts here
wait for the exact string WORKLOAD_READY.
A host mount source that goes missing stops every later start
If a machine created from a Smolfile has started before and now never starts again, check that
every host path in volumes still exists. create refuses a path that is missing when it runs,
with mount source not found: ./src; a later start does not check the path again, and in the run
below every later start failed with:
Error: agent operation failed: start machine: agent operation failed: wait for ready:
agent did not become ready within 30 seconds which names neither the mount nor the missing directory, and reads exactly like host load.
Measured on macOS 26.6.2 arm64 on v1.14.3, 2026-09-08, three create-start-stop-start cycles each with the workload confirmed ready before the stop:
| host mount source | restarts |
|---|---|
./src exists | 3 of 3 |
./src absent | 0 of 3 |
How ./src came to be missing in that run was not recorded. On v1.22.2 on macOS arm64,
2026-10-03, removing ./src after a machine’s first start reproduced it: the next start failed
with the same message, and create with ./src already missing refused with Error: mount source not found: ./src.
scripts/create-dev-machine.sh runs mkdir -p ./src before creating, which is why the packet’s
own flow does not hit this. Anything that writes its own Smolfile has to do the same. A
relative path in volumes resolves against the working directory, so the same Smolfile run from
two directories can behave differently.
An ad-hoc restart loop that omitted the mkdir -p produced 0 of 5 and looked like a release
regression until the two were compared side by side.
machine shell does not start a stopped machine
Before v1.23.7 smolvm machine --help described shell as “Open an interactive shell in a machine
(starts it if stopped)”; from v1.23.7 it says “in a running machine”. It does not start one:
$ smolvm machine stop --name dev
$ smolvm machine shell --name dev
Error: agent operation failed: connect: machine 'dev' is not running.
Use 'smolvm machine start --name dev' first. Verified both through a pipe and under a real pty, so it is not a TTY-detection effect, and again on v1.18.2. Start it explicitly first.
A host volumes mount was not writable by the app user
With volumes = ["./src:/app"] and user = "app" (a user adduser made in the guest), writing to /app failed with Operation not permitted: the mount carries host ownership and app is not its
owner. Read source in through the mount, write build output somewhere else, or run that step as
root. user also takes a numeric uid[:gid], which machine create --help gives as the way to
match the owner of a mounted host directory: on v1.22.2 on macOS arm64, machine run -v "$PWD/src:/app" --user "$(id -u):$(id -g)" wrote /app/w and the file appeared on the host.
/tmp is wiped by a stop
Anything a provisioning step leaves in /tmp is gone on the next start, while the same step’s
writes to $HOME or / persist. What survives, verified by writing one file per filesystem and
restarting:
| location | survives | why |
|---|---|---|
pip --user packages | yes | under $HOME, on the overlay |
$HOME/... files | yes | overlay |
files written at / | yes | overlay |
/tmp | no | emptied at each start, like a tmpfs |
/storage/... | yes | the ext4 disk itself |
Read from inside the guest, the mechanism is an overlay whose upper layer lives on the machine’s ext4 disk:
none on / type overlay (... upperdir=/storage/overlays/persistent-<name>/upper ...)
/dev/vda on /workspace type ext4 (rw,noatime)
/dev/vda on /storage type ext4 (rw,noatime) machine delete prompts and defaults to No
Through v1.16.x a scripted delete without --force printed Delete machine 'dev'? [y/N] Cancelled, exited 0 and left the machine in place while the script carried on. On v1.17.0 and
later it exits 1 with needs confirmation but stdin is not a terminal; pass --force to delete it.
Always pass --force in a script.
A VPN inside the machine takes its gateway and resolver
A machine on virtio-net gets the link 100.96.0.0/30 by default, with the gateway and resolver at .1. Tailscale and other carrier NAT VPNs claim 100.64.0.0/10, which contains it, so once the
VPN is up inside the machine its gateway and resolver route into the VPN and every lookup fails.
The throwaway-machine packet’s traps have the measurement: with the routes Tailscale adds, the default link
gave bad address 'example.com' and --guest-subnet 10.200.0.0/30 with the same routes resolved
and fetched, on v1.18.2 on both hosts.
For a dev machine the flag has three constraints, each observed on v1.18.2:
- It is set at
createonly.machine updatehas no--guest-subnet, so changing it means a new machine. It survives stop and start. - A Smolfile cannot carry it.
[network]acceptsallow_hosts,allow_cidrsandcredentials; pass--guest-subnetonmachine createnext to-s. - It implies
--netand virtio-net, andmachine ls --jsonthen reports"network": falsefor a machine that does reach the network, so do not read that field as the answer.
Do not run two lifecycle commands against the same machine at once
Two overlapping stop and shell invocations left a machine stop unfinished for over two
minutes, against 0.14 s for a single stop on the same machine. It was seen once, with the two
calls overlapping, and was not reproduced as a fault; anything that fans out lifecycle calls should
still serialize them per machine.
Assert the package version, not that the import worked
An import can be satisfied by a system copy and tells you nothing about whether your install
survived the restart. scripts/verify-persistence.sh records the version before the stop and
compares it after.