Dev env: a persistent machine you re-enter

Keeps a persistent smolvm machine with its dependencies already installed and re-enters it cheaply across sessions. Use when a project needs an isolated development environment that survives stop and start; when deciding what belongs in a Smolfile’s init versus what has to run on every boot; when a package installed in a machine has vanished after a restart; or when exec answers “the container smolvm-is not running”. Do not use it for untrusted code, which needs a machine that leaves nothing behind (see the throwaway-machine packet), or for running a Docker daemon inside the machine (see docker-in-machine).

Verified on smolvm v1.34.0 on macOS arm64 and Linux x86_64, 2026-11-09, and on v1.18.2 on Linux aarch64, 2026-09-24; the aarch64 runs used the Smolfile with memory = 1024, for the host reason in “Platform arms”. Done means a second start is fast, skips provisioning, and the packages installed in the first session are still there. The Linux aarch64 runs used the scripts of their date; this version’s preflight and cleanup scripts ran on Linux aarch64 on v1.22.2 on 2026-10-03.

The whole use case turns on one fact: init runs once, not on every start, as smolfile.md says, and as the docs site (smolmachines.com/docs) says under “When init runs”. smolvm machine create --help reads “Run command on every VM start” at v1.22.2, and “Run command once, on the machine’s first start” from v1.23.7; provisioning that relies on the old line is missing on the second boot.

Procedure

1. Preflight.

scripts/preflight.sh

Read-only. restart_after_stop=verified on macOS and Linux. The script covers macOS and Linux only; elsewhere it prints unknown and a note pointing at references/windows.md, where a stopped machine starts again on v1.14.6 and v1.22.2.

2. Declare the machine. assets/dev.smolfile is a working starting point, and the one verify-persistence.sh is written against: it asserts the app user and the /app workdir that file sets. examples/python-app/python.smolfile and examples/node-app/node.smolfile in the repository are plainer examples without them.

image = "python:3.12-alpine"
net = true
cpus = 2
memory = 2048
volumes = ["./src:/app"]
init = [
  "sh -c \"id -un > /init-ran-as.txt\"",
  "sh -c \"date +%s >> /init-count.txt\"",
  "adduser -D app",
]
user = "app"
workdir = "/app"

3. Create and bring it up. Run this from the project directory: the Smolfile mounts ./src, which is relative to the working directory, and the script creates ./src there if it is missing.

scripts/create-dev-machine.sh              # name defaults to smolskill-dev
scripts/create-dev-machine.sh smolskill-myproj ./my.smolfile

It creates the machine with an explicit long-lived workload command, records the name for cleanup, starts it, waits for the workload container to answer with a value, and then reports what actually happened rather than that nothing errored:

init_ran=yes
machine_running=yes
workload_ready_after_s=0
workload_ready=yes
init_ran_as=root
exec_user=app
workdir=/app
result=up

init_ran_as=root with exec_user=app is the correct outcome, not a bug: init provisions the machine and runs as root regardless of user, while exec and shell run as user.

4. Work in it.

smolvm machine exec  --name smolskill-dev -- pip install --user requests
smolvm machine exec  --name smolskill-dev --user root -- sh -c 'mkdir -p /storage/keep'
smolvm machine shell --name smolskill-dev            # interactive, lands in workdir as `user`

5. Prove it is worth keeping.

scripts/verify-persistence.sh

It installs a package and records its version, seeds one file per filesystem, stops, starts, and then asserts each value against what it recorded. A version comparison is the point: an import that does not crash can be satisfied by a system copy and says nothing about your install. It leaves the machine running.

When the setup is done, smolvm machine stop --name smolskill-dev and give the user its name; the next machine start brings it back as it was.

6. Clean up, when the machine is no longer wanted. Not at the end of a setup: the machine is the deliverable, so leave it stopped and tell the user its name.

scripts/cleanup.sh --purge

cleanup.sh waits up to 20 seconds, polling the machine list, and prints waiting=up to 20s first: an ephemeral machine’s entry retires after its run returns.

What init and user actually do, observed on v1.14.2 and again on v1.22.2

Observed, not inferred:

questionobserved
when does init run?on the first start, not on create, and not again
second startprints Init already completed, skipping N command(s)
which user runs init?root, even with user = "app" set
which user runs exec and shell?the Smolfile user
override per commandmachine exec --user root works
is there machine start --init?no

What survives a stop and start

locationsurviveswhy
pip --user packagesyesunder $HOME, on the overlay
$HOME/... filesyesoverlay
files written at /yesoverlay
/tmpnoemptied at each start, like a tmpfs
/storage/...yesthe ext4 disk itself

The root filesystem is an overlay whose upper layer lives on the machine’s ext4 disk, which is why writes to / persist while tmpfs mounts and /tmp do not.

Traps

Full detail in references/traps.md. The ones that cost the most:

  • init runs once (top of this file). Anything that must hold on every boot, a bind mount above all, runs in the command that needs it.
  • Give a machine you exec into a long-lived workload. Without one, an exec right after start can answer the container `smolvm-<hash>` is not running instead of running.
  • machine shell does not start a stopped machine on v1.22.2, while its help says “starts it if stopped”.
  • A machine that runs Tailscale or another carrier NAT VPN needs --guest-subnet at create, which a Smolfile cannot carry: smolvm machine create --name smolskill-dev -s dev.smolfile --guest-subnet 10.200.0.0/30.
  • create is instant and proves nothing, and a host volumes mount was not writable by the app user.

Security defaults, and why they are the defaults

  • user in the Smolfile is what your workload runs as, and it is not root. init running as root is the provisioning step, deliberately separated from the workload’s identity. Keep them separate rather than setting user = "root" to make a mount writable: the mount carries host ownership, and running the workload as root to reach it hands root in the guest everything the mount exposes on the host.
  • A volumes mount is host authority handed to the guest, so mount the narrowest directory that works and prefer :ro for anything the machine only reads. Treat root in the guest as untrusted, as the security model says: every forwarded mount, port and network permission becomes part of the workload’s authority.
  • net = true is outbound access for the whole machine. A dev machine that only needs a package index does not need general egress; --allow-host scopes it.
  • These scripts wrap the public CLI only. They edit no smolvm configuration and nothing under ~/.smolvm, and cleanup deletes only names it recorded under the smolskill- prefix.

Platform arms

  • Linux aarch64: the scripts were run here on v1.18.2 with the Smolfile’s memory at 1024 MiB, and once on v1.22.2 at 1024. At 2048 on v1.18.2 the restart failed with agent did not become ready within 30 seconds: on a small or busy host lower memory before suspecting smolvm, and check the mount source first (the install packet’s traps have the numbers).
  • macOS arm64: the scripts were run here on v1.34.0 as shipped, and every check passed.
  • Linux x86_64: verified on v1.34.0, twice, on an NVIDIA A10 cloud host, with the Smolfile as shipped.
  • Windows x86_64: references/windows.md, re-run on 2026-11-09 against v1.34.0 on Windows 11 Home build 10.0.26200 UBR 9711. Create, three stops and three starts, with a marker read back after each start: a stopped machine starts again and its state survives, as on v1.14.6, where on v1.14.2 it did not. That page also carries the WHP device ceiling a Windows preflight needed through v1.22.2, four -v mounts and three when a port is published; v1.34.0 boots eight with two ports.

Eval prompts, and what they produced

1. “Set me up a Python dev machine I can come back to, and prove the packages survive a restart.” On macOS arm64: the step 3 output above, on v1.22.2.

Then, after the stop and start, verify-persistence.sh:

workload_ready_after_s=0
installed_version=2.34.2
  Init already completed, skipping 3 command(s)
workload_ready_after_s=0
init_ran_once=ok (yes)
package_version=ok (2.34.2)
home_file=ok (SURVIVES)
storage_file=ok (SURVIVES)
tmp_file=ok (WIPED)
exec_user=ok (app)
workdir=ok (/app)
result=persistent

2. “My bind mount is gone after I restarted the machine. It is right there in init.”

init ran once. The second start reports it, on v1.22.2 as in eval 1:

Init already completed, skipping 3 command(s)

There is no machine start --init, so the mount has to be re-applied by the command that needs it. docker-in-machine is the packet built entirely around this.

3. ”smolvm machine exec just told me the container is not running, but the machine is running.”

On v1.14.2, reproduced deliberately by creating the same machine without a workload command, then running 20 execs, on Lima:

 3: the container `smolvm-e51bb94127b751bf` is not running
failures=1/20
 4: the container `smolvm-32edffb50eaf0d20` is not running
failures_after_restart=1/20

and with the explicit long-lived workload scripts/create-dev-machine.sh passes:

failures=0/20
failures_after_restart=0/20

The changing hash is the tell: the workload container is being relaunched between execs.

Re-verified on v1.34.0

Run 2026-11-09 PT against v1.34.0 from the published release, checksum checked, under a fresh isolated HOME on macOS 27.2 arm64 (Apple M4), twice, the second time from a fresh HOME, and once on Lima linux-kvm (Ubuntu 24.04 aarch64, 11 GiB). Every step on both hosts, with the Smolfile as shipped: result=up, init_ran_once=ok (yes), package_version=ok (2.34.2), tmp_file=ok (WIPED) and result=persistent.

On Linux x86_64 (Ubuntu 22.04, KVM, an NVIDIA A10 cloud host with 30 cores and 222 GiB), twice on 2026-11-09, each pass from a fresh HOME, every step with the Smolfile as shipped: result=up, init_ran_once=ok (yes), package_version=ok (2.34.2), tmp_file=ok (WIPED) and result=persistent.

Re-verified on v1.29.0

Run 2026-10-15 PT against v1.29.0 from the published release, checksum checked, under a fresh isolated HOME on macOS 27.0.1 arm64, the scripts twice, the second time from a fresh HOME, and the checks beyond them once. Linux was not run, so the Linux stamp stays on its earlier release.

create-dev-machine.sh gave result=up with init_ran_as=root, exec_user=app and workdir=/app; verify-persistence.sh gave init_ran_once=ok (yes), package_version=ok (2.34.2), home_file=ok (SURVIVES), storage_file=ok (SURVIVES), tmp_file=ok (WIPED) and result=persistent, and the second start printed Init already completed, skipping 3 command(s).

Re-verified on v1.22.2

Run 2026-10-03 PT against v1.22.2 from the published release, checksum checked, under an isolated HOME on macOS 27.0.1 arm64, twice, the second time from a fresh HOME. On Lima linux-kvm (Ubuntu 24.04 aarch64) guests above 2048 MiB timed out on 2026-10-03, so the Linux lines below are a single run and the Linux stamp stays on its earlier release.

macOS: result=up with init_ran_as=root and exec_user=app, then result=persistent, package_version=ok (2.34.2), tmp_file=ok (WIPED). machine create --help describes --init as running “on every VM start”, machine shell on a stopped machine answered is not running. Use 'smolvm machine start ...' first while its help says it starts one, and the exec race gave 0 of 20.

Linux aarch64 at memory = 1024: result=persistent, the exec race 0 of 20. Linux was last verified on v1.18.2, at memory = 1024.

What was not run

  • Windows beyond the restart. The v1.22.2 run covered create, three stops and starts, the marker and init running once; the WHP device ceiling was confirmed by the install packet’s Windows run on 2026-10-03.
  • init after a checkpoint restore. Nowhere, on any platform.
  • The USER interaction still settling. init runs as root on v1.22.2, and whether the image’s own USER still governs it in some paths is open as smolvm#1189. Nothing here tested an image with a USER line.
  • install for the boot this assumes, and teardown for the cleanup script.
  • docker-in-machine for the clearest case of the init-runs-once trap.
  • pack for turning the machine this packet builds into a portable artifact.

Scripts

The files the procedure runs, in the order it runs them. It calls each one by the path in its heading, relative to the directory the procedure is saved in.

scripts/preflight.sh

#!/usr/bin/env bash
# Report whether this host can keep a persistent smolvm dev machine. Read-only:
# starts no VM, writes no smolvm state, changes no group membership.
#
# Output is one key=value per line so a caller can parse it. The last line is
# always result=ready or result=blocked.

set -uo pipefail

VERIFIED_VERSION="1.34.0"

emit() { printf '%s=%s\n' "$1" "$2"; }

blocked=0
note() { printf 'note=%s\n' "$1"; }

# --- the binary --------------------------------------------------------------

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
    emit smolvm_installed no
    emit smolvm_version ""
    blocked=1
else
    emit smolvm_installed yes
    version="$("$SMOLVM" --version 2>/dev/null | awk '{print $NF}')"
    emit smolvm_version "${version:-unknown}"
fi

emit verified_version "$VERIFIED_VERSION"
if [ -n "${version:-}" ] && [ "$version" != "unknown" ]; then
    if [ "$version" = "$VERIFIED_VERSION" ]; then
        emit version_status match
    else
        newest="$(printf '%s\n%s\n' "$version" "$VERIFIED_VERSION" | sort -V | tail -1)"
        if [ "$newest" = "$version" ]; then
            emit version_status newer
            note "this packet was verified on $VERIFIED_VERSION and the binary is $version; flags and messages move every release, so check the output against the binary before trusting a step here"
        else
            emit version_status older
            note "this packet was verified on $VERIFIED_VERSION and the binary is $version"
        fi
    fi
else
    emit version_status unknown
fi

# --- platform ----------------------------------------------------------------

kernel="$(uname -s)"
arch="$(uname -m)"
case "$arch" in aarch64|arm64) arch=aarch64 ;; esac

case "$kernel" in
    Darwin)
        emit platform "darwin-$arch"
        emit accel hvf
        emit macos_version "$(sw_vers -productVersion)"
        hv="$(sysctl -n kern.hv_support 2>/dev/null)"
        if [ "$hv" = "1" ]; then emit accel_access ok; else emit accel_access denied; blocked=1; fi
        if [ "$arch" != "aarch64" ]; then
            emit hardware_verified no
            note "Intel Mac is not verified by this packet; the installer accepts it and nothing here was run on one"
        else
            emit hardware_verified yes
        fi
        # A VM's agent socket lives under the cache directory. macOS sockaddr_un
        # holds 104 bytes including the terminator.
        sock="$HOME/Library/Caches/smolvm/vms/0123456789abcdef/agent.sock"
        len=${#sock}
        emit socket_path_bytes "$len"
        if [ "$len" -gt 100 ]; then
            emit socket_path_status too_long
            blocked=1
            note "HOME is too deep: every VM start will fail. From v1.28.0 the error names the socket path and its length; before it, krun_start_enter -22, whose text blames disks and device options. Install under a shorter HOME."
        else
            emit socket_path_status ok
        fi
        emit unsupported "cuda"
        emit restart_after_stop verified
        ;;
    Linux)
        emit platform "linux-$arch"
        emit accel kvm
        emit socket_path_status n_a
        if [ ! -e /dev/kvm ]; then
            emit accel_access missing
            blocked=1
            note "/dev/kvm does not exist; this host has no KVM"
        elif [ -r /dev/kvm ] && [ -w /dev/kvm ]; then
            emit accel_access ok
        else
            emit accel_access denied
            blocked=1
            note "your user cannot open /dev/kvm. The installer warns and continues, so a successful install says nothing about whether a VM will start. Fix: sudo usermod -aG kvm \$USER, then run the next command through sg kvm -c '...' rather than logging out."
        fi
        emit restart_after_stop verified
        ;;
    *)
        emit platform "unsupported-$kernel"
        emit accel unknown
        emit accel_access unknown
        blocked=1
        emit restart_after_stop unknown
        note "this script covers macOS and Linux. On Windows a stopped machine starts again, on v1.14.6 and v1.22.2, and through v1.22.2 the WHP device budget caps you at four -v mounts, three when a port is published. See references/windows.md."
        ;;
esac

if [ "$blocked" -eq 0 ]; then emit result ready; else emit result blocked; fi

assets/dev.smolfile

# A persistent development machine. `smolvm machine create -s dev.smolfile`.
#
# The two keys worth understanding before you edit this:
#
#   init  runs ONCE, on the first start, and as root. It does not run again on
#         later starts, so anything that has to be true on every boot does not
#         belong here.
#   user  governs `exec` and `shell`, not `init`.
#
# The workload command is not set here. A Smolfile can carry one as `cmd`;
# this packet passes it after `--` on `machine create`, as
# scripts/create-dev-machine.sh does: without one the image's own CMD becomes
# the persistent workload, and for an interpreter image that command exits at
# once and `exec` starts racing the relaunch.
image = "python:3.12-alpine"
net = true
cpus = 2
memory = 2048

# A host mount carries host ownership, so `user` below (made by adduser) could
# not write to it. Read source in, write build output somewhere else.
volumes = ["./src:/app"]

init = [
  "sh -c \"id -un > /init-ran-as.txt\"",
  "sh -c \"date +%s >> /init-count.txt\"",
  "adduser -D app",
]

user = "app"
workdir = "/app"

scripts/create-dev-machine.sh

#!/usr/bin/env bash
# Create a persistent dev machine and bring it up for the first time.
#
# usage: create-dev-machine.sh [<name>] [<smolfile>]
#   name      default smolskill-dev (the prefix cleanup.sh will delete)
#   smolfile  default ../assets/dev.smolfile
#
# `create` is configuration only: it returns in milliseconds and pulls nothing. It
# does refuse a host mount source that does not exist; the pull, the init commands
# and any of their failures land on the first `start`, so a fast create is not
# evidence that anything works.

set -uo pipefail

here="$(cd "$(dirname "$0")" && pwd)"
NAME="${1:-smolskill-dev}"
SMOLFILE="${2:-$here/../assets/dev.smolfile}"

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
    printf 'smolvm not found; set SMOLVM to its path\n' >&2
    exit 2
fi

case "$NAME" in
    smolskill-*) ;;
    *) printf 'name must start with smolskill- so cleanup.sh will delete it\n' >&2; exit 2 ;;
esac

# The Smolfile mounts ./src, which is relative to the working directory.
mkdir -p ./src

# Give the machine a workload that stays up. Without a command, `create` launches
# the image's own ENTRYPOINT/CMD as the persistent workload; for an interpreter
# image such as python:3.12-alpine that command reads EOF and exits at once, the
# container is relaunched, and `exec` then intermittently answers "the container
# `smolvm-<hash>` is not running" instead of running your command. Measured on a
# nested-virt aarch64 host: 1 exec in 20 failed that way with no command, 0 in 20
# with this one, both on a fresh machine and immediately after a restart.
create_out="$("$SMOLVM" machine create --name "$NAME" -s "$SMOLFILE" \
    -- sh -c 'while true; do sleep 3600; done' 2>&1)"
create_rc=$?
printf '%s\n' "$create_out" | sed 's/^/  /'
if [ "$create_rc" -ne 0 ]; then printf 'result=failed (create)\n'; exit 1; fi
"$here/cleanup.sh" --record "$NAME"

start_out="$("$SMOLVM" machine start --name "$NAME" 2>&1)"
printf '%s\n' "$start_out" | sed 's/^/  /'

fail=0
# `start` prints the init line only when the Smolfile declares init commands.
if printf '%s' "$start_out" | grep -q 'Running .* init command'; then
    printf 'init_ran=yes\n'
elif ! grep -q '^[[:space:]]*init[[:space:]]*=' "$SMOLFILE"; then
    printf 'init_ran=none_declared\n'
else
    printf 'init_ran=no\n'
    fail=1
fi
if printf '%s' "$start_out" | grep -q "Machine '$NAME' running"; then
    printf 'machine_running=yes\n'
else
    printf 'machine_running=no\n'
    fail=1
fi

# `start` returns before the workload container is up. Wait for a VALUE: an empty
# result and a zero exit code both pass while the container is still coming up,
# so a probe that asserts either of those reports ready too early and the next
# three checks read a message as their answer.
wait_for_workload() {
    # 120 s: the slowest first start observed while building this packet was
    # under 20 s on a nested-virt aarch64 host, so this is six times the worst
    # case and still short enough that a real hang is reported, not waited on.
    waited=0
    while [ "$waited" -lt 120 ]; do
        out="$("$SMOLVM" machine exec --name "$1" -- sh -c 'echo WORKLOAD_READY' 2>&1 | tr -d '\r')"
        if [ "$out" = "WORKLOAD_READY" ]; then
            printf 'workload_ready_after_s=%s\n' "$waited"
            return 0
        fi
        sleep 1
        waited=$((waited + 1))
    done
    printf 'workload_ready_after_s=timeout last_probe=%s\n' "$out"
    return 1
}

if wait_for_workload "$NAME"; then
    printf 'workload_ready=yes\n'
else
    printf 'workload_ready=no\n'
    fail=1
fi

# init runs as root even with `user` set. Asserting the value rather than the
# absence of an error is the point: an exec that fails still leaves you guessing.
printf 'init_ran_as=%s\n' "$("$SMOLVM" machine exec --name "$NAME" --user root -- cat /init-ran-as.txt 2>&1 | tr -d '\r')"
printf 'exec_user=%s\n'   "$("$SMOLVM" machine exec --name "$NAME" -- id -un 2>&1 | tr -d '\r')"
printf 'workdir=%s\n'     "$("$SMOLVM" machine exec --name "$NAME" -- pwd 2>&1 | tr -d '\r')"

if [ "$fail" -eq 0 ]; then printf 'result=up\n'; else printf 'result=failed\n'; fi
exit "$fail"

scripts/verify-persistence.sh

#!/usr/bin/env bash
# Prove the machine is worth keeping: that a package installed in one session is
# still there in the next, that init does not run again, and that the parts of
# the filesystem people assume persist actually do.
#
# usage: verify-persistence.sh [<name>]     (default smolskill-dev)
#
# Every check asserts a value. Asserting that an import did not crash proves
# nothing here: a system copy of the package satisfies it and tells you nothing
# about whether your install survived.

set -uo pipefail

NAME="${1:-smolskill-dev}"
SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
if [ -z "$SMOLVM" ]; then
    printf 'smolvm not found; set SMOLVM to its path\n' >&2
    exit 2
fi

fail=0
check() {
    if [ "$2" = "$3" ]; then
        printf '%s=ok (%s)\n' "$1" "$2"
    else
        printf '%s=FAIL expected=%s actual=%s\n' "$1" "$3" "$2"
        fail=1
    fi
}

exec_in() { "$SMOLVM" machine exec --name "$NAME" -- "$@" 2>&1 | tr -d '\r'; }

# `start` returns before the workload container is up, and an `exec` in that
# window answers "the container `smolvm-<hash>` is not running" on stdout while
# still exiting zero. Wait for a VALUE: an empty result or a zero exit code both
# pass while the container is still coming up, which is how this window gets
# missed. Observed on a nested-virt aarch64 host, where a probe that asserted
# only an empty result let three later checks read that message as their answer.
wait_for_workload() {
    # 120 s: six times the slowest start observed while building this packet.
    # Long enough to absorb a loaded host, short enough to report a real hang.
    waited=0
    while [ "$waited" -lt 120 ]; do
        probe="$(exec_in sh -c 'echo WORKLOAD_READY')"
        if [ "$probe" = "WORKLOAD_READY" ]; then
            printf 'workload_ready_after_s=%s\n' "$waited"
            return 0
        fi
        sleep 1
        waited=$((waited + 1))
    done
    printf 'workload_ready_after_s=timeout last_probe=%s\n' "$probe"
    return 1
}

if ! wait_for_workload; then
    printf 'result=FAILED (workload container never came up)\n'
    exit 1
fi

# 1. Install something and record its version, not its presence.
"$SMOLVM" machine exec --name "$NAME" -- pip install --quiet --user requests >/dev/null 2>&1
before="$(exec_in python3 -c 'import requests; print(requests.__version__)')"
printf 'installed_version=%s\n' "$before"
case "$before" in [0-9]*) ;; *) printf 'installed_version=FAIL (not a version)\n'; fail=1 ;; esac

# 2. Seed one file per filesystem so the stop tells them apart.
# shellcheck disable=SC2016  # $HOME must expand inside the guest, not here
"$SMOLVM" machine exec --name "$NAME" -- sh -c 'echo SURVIVES > "$HOME/keep.txt"; echo GONE > /tmp/scratch.txt' >/dev/null 2>&1
"$SMOLVM" machine exec --name "$NAME" --user root -- sh -c 'mkdir -p /storage/keep && echo SURVIVES > /storage/keep/disk.txt' >/dev/null 2>&1

# 3. Stop and come back.
"$SMOLVM" machine stop --name "$NAME" >/dev/null 2>&1
restart_out="$("$SMOLVM" machine start --name "$NAME" 2>&1)"
printf '%s\n' "$restart_out" | sed 's/^/  /'

if ! wait_for_workload; then
    fail=1
fi

if printf '%s' "$restart_out" | grep -q 'Init already completed'; then
    check init_ran_once yes yes
else
    check init_ran_once no yes
fi

after="$(exec_in python3 -c 'import requests; print(requests.__version__)')"
check package_version "$after" "$before"

# shellcheck disable=SC2016  # same: the guest resolves $HOME
check home_file    "$(exec_in sh -c 'cat "$HOME/keep.txt" 2>/dev/null')" SURVIVES
check storage_file "$(exec_in cat /storage/keep/disk.txt)" SURVIVES

# /tmp is emptied by a stop. Anything a provisioning step leaves
# there is gone on the next start, while the same step's writes to $HOME or /
# persist. This is the asymmetry that surprises people.
check tmp_file "$(exec_in sh -c 'cat /tmp/scratch.txt 2>/dev/null || echo WIPED')" WIPED

check exec_user "$(exec_in id -un)" app
check workdir   "$(exec_in pwd)" /app

if [ "$fail" -eq 0 ]; then printf 'result=persistent\n'; else printf 'result=FAILED\n'; fi
exit "$fail"

scripts/cleanup.sh

#!/usr/bin/env bash
# Delete the machines this packet's scripts created, then prove the host is clean.
#
# Only machines recorded in the state file are deleted, with --cascade, so any
# machine branched from one of them goes too, whatever its name. Any other
# machine you or another session created by hand is never deleted. Scripts
# record a name by calling: cleanup.sh --record <name>
#
# usage: cleanup.sh [--record <name>] [--reap] [--purge]
#   --record <name>  add a machine name to the state file and exit
#   --reap           kill every VM process under this HOME; run without it first
#   --purge          also remove the state file once the list is empty

set -uo pipefail

PACKET="dev-env"
PREFIX="smolskill-"

SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}"
STATE_DIR="${SMOLVM_SKILL_STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/smolvm-skills}"
STATE_FILE="$STATE_DIR/$PACKET.machines"

reap=0
purge=0
while [ $# -gt 0 ]; do
    case "$1" in
        --record)
            mkdir -p "$STATE_DIR"
            printf '%s\n' "$2" >> "$STATE_FILE"
            exit 0
            ;;
        --reap)  reap=1 ;;
        --purge) purge=1 ;;
        *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;;
    esac
    shift
done

if [ -z "$SMOLVM" ]; then
    printf 'smolvm not found; set SMOLVM to its path\n' >&2
    exit 2
fi

# With SMOLVM_DATA_DIR set, smolvm runs with HOME there on Linux, so its cache
# is under .cache in it.
case "$(uname -s)" in
    Darwin) VMS_DIR="$HOME/Library/Caches/smolvm/vms" ;;
    *)      cache_root="${SMOLVM_DATA_DIR:+$SMOLVM_DATA_DIR/.cache}"
            VMS_DIR="${cache_root:-${XDG_CACHE_HOME:-$HOME/.cache}}/smolvm/vms" ;;
esac
VMS_DIR="${SMOLVM_VMS_DIR:-$VMS_DIR}"
SMOLVM_PREFIX="${SMOLVM_PREFIX:-$HOME/.smolvm}"
if [ -n "${SMOLVM:-}" ]; then
    case "$(readlink "$SMOLVM" 2>/dev/null || printf '%s' "$SMOLVM")" in
        "$SMOLVM_PREFIX"/*) ;;
        *) printf 'note=%s is not under %s, so the process scan cannot see its forked VMs (all of its VMs on macOS); set SMOLVM_PREFIX to its directory\n' "$SMOLVM" "$SMOLVM_PREFIX" ;;
    esac
fi

# List this HOME's VM processes as "pid marker". The plain run path execs a
# `_boot-vm` child that carries its boot config; the pack-run path forks one that
# carries none. Linux names both `libkrun VM`, or `VM:<hostname>` when HOSTNAME
# is exported; on macOS the executable path and the parent chain scope the
# search. The teardown packet's traps have the why.
list_vm_processes() {
    case "$(uname -s)" in
        Linux)
            for p in /proc/[0-9]*; do
                case "$(cat "$p/comm" 2>/dev/null)" in "libkrun VM"|VM:*) ;; *) continue ;; esac
                pid="${p#/proc/}"
                cfg="$(tr '\0' '\n' < "$p/cmdline" 2>/dev/null | sed -n '3p')"
                case "$cfg" in
                    "$VMS_DIR"/*) printf '%s %s\n' "$pid" "$cfg"; continue ;;
                esac
                case "$(readlink "$p/exe" 2>/dev/null)" in
                    "$SMOLVM_PREFIX"/*) printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
                esac
            done
            ;;
        Darwin)
            # shellcheck disable=SC2009  # pgrep -f would match this script.
            procs="$(ps -axo pid=,ppid=,command= 2>/dev/null)"
            printf '%s\n' "$procs" | while read -r pid ppid rest; do
                case "$rest" in "$SMOLVM_PREFIX"/smolvm-bin*) ;; *) continue ;; esac
                case "$rest" in
                    *" _boot-vm "*) printf '%s %s\n' "$pid" "${rest#* _boot-vm }"; continue ;;
                esac
                # An orphan counts only if it is a run; a fork has its parent's command line.
                if [ "$ppid" = 1 ]; then
                    case "$rest" in
                        *" machine run "*|*" vm run "*|*" pack run "*)
                            printf '%s orphaned-under %s\n' "$pid" "$SMOLVM_PREFIX" ;;
                    esac
                else
                    parent="$(printf '%s\n' "$procs" | while read -r q qp qrest; do [ "$q" = "$ppid" ] && { printf '%s' "$qrest"; break; }; done)"
                    [ "$parent" = "$rest" ] && printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX"
                fi
            done
            ;;
    esac
}

# 1. Delete recorded machines. Without --force a delete prompts and defaults to
# No; --cascade removes branch children, which otherwise block it.
if [ -s "$STATE_FILE" ]; then
    while read -r name; do
        [ -n "$name" ] || continue
        case "$name" in "$PREFIX"*) ;; *)
            printf 'skipping %s: not created by this packet (no %s prefix)\n' "$name" "$PREFIX"
            continue ;;
        esac
        # Only names still listed: a second stop of a missing name leaves a
        # directory that reads as a leak. The list is read first because grep -q
        # under pipefail can fail the pipeline and skip a listed machine.
        listed="$("$SMOLVM" machine list </dev/null 2>/dev/null | awk 'NR>2{print $1}')"
        grep -qx -- "$name" <<<"$listed" || continue
        "$SMOLVM" machine stop   --name "$name" </dev/null >/dev/null 2>&1
        "$SMOLVM" machine delete --name "$name" --force --cascade </dev/null 2>&1 | sed 's/^/  /'
    done < "$STATE_FILE"
fi

# 2. An ephemeral machine's entry retires after its run returns: poll, 20 s at most.
printf 'waiting=up to 20s for ephemeral entries to retire before asserting\n'
waited=0
listing="$("$SMOLVM" machine list 2>&1)"
while ! grep -q 'No machines found' <<<"$listing" && [ "$waited" -lt 20 ]; do
    sleep 1; waited=$((waited + 1))
    listing="$("$SMOLVM" machine list 2>&1)"
done
printf 'waited=%ss\n' "$waited"

# 3. Assert the value, not the exit code.
if grep -q 'No machines found' <<<"$listing"; then
    printf 'machines=clean\n'
    [ "$purge" -eq 1 ] && rm -f "$STATE_FILE"
else
    printf 'machines=remaining\n'
    printf '%s\n' "$listing" | sed 's/^/  /'
    printf 'note=this packet did not create these, so it will not delete them. By name:\n'
    printf '  smolvm machine stop --name <NAME> && smolvm machine delete --name <NAME> --force\n'
    printf '  add --cascade for a machine that was branched from another\n'
fi

# 4. Report VM processes under this HOME's state, such as a killed wrapper's.
# With --reap every one is killed, including a machine another packet kept.
found=0
while read -r pid cfg; do
    [ -n "$pid" ] || continue
    found=1
    printf 'vm_process=%s config=%s\n' "$pid" "$cfg"
    if [ "$reap" -eq 1 ]; then
        kill -9 "$pid" 2>/dev/null && printf '  killed %s\n' "$pid"
    fi
done <<EOF
$(list_vm_processes)
EOF

if [ "$found" -eq 0 ]; then
    printf 'vm_processes=none\n'
elif [ "$reap" -eq 0 ]; then
    printf 'rerun with --reap to kill them\n'
fi

Persistent dev machine traps

init runs once, not on every start

Observed on v1.14.2 on macOS arm64 and Linux aarch64: init runs on the first start, not on create, and never again. The second start prints Init already completed, skipping N command(s).

Anything that must be true on every boot does not belong in init. The most common victim is a bind mount: put mount --bind ... in init and the second boot comes up without it. Re-apply it in the same command that needs it. The docker-in-machine packet is built around this.

The two help texts read differently. At v1.22.2 smolvm machine create --help describes --init as “Run command on every VM start”, and smolvm machine run --help as “Run command before the workload … the same as init in a Smolfile”. smolfile.md says init “runs once as root, like a Dockerfile RUN”, and the docs site’s introduction/concepts/smolfile.md (smolmachines.com/docs) says under “When init runs” that it runs once, on the first start. Observed: once, on the first start. From v1.23.7 machine create --help says the same: “Run command once, on the machine’s first start”.

init also runs as root, even with user set: /init-ran-as.txt contained root while exec and shell ran as app. There is no machine start --init to force a re-run.

create is not where the time goes, and not where most failures appear

It is pure configuration and returns in milliseconds without touching the registry. The pull, the init commands and any of their failures all land on the first start. Do not read a fast create as evidence that anything works. It does refuse a host mount source that does not exist; see below.

create pulls nothing either; the image is pulled on the first start.

exec right after start can answer with a message instead of running

If you see the container `smolvm-<hash>` is not running, the workload container is not up and the exec did nothing. The hash differs between occurrences, because the container is being relaunched.

Why it happens. Without a command, machine create launches the image’s own ENTRYPOINT or CMD as the persistent workload (machine create --help, [COMMAND]). For an interpreter image such as python:3.12-alpine that command reads EOF from a stdin nobody is holding and exits at once, so the container dies and is relaunched, and an exec can land in the gap.

Measured on a nested-virt aarch64 host on 2026-09-07, 20 execs each on a fresh machine and again immediately after a restart:

workload commandfailures, freshfailures, after restart
none (image CMD)1 in 201 in 20
sh -c 'while true; do sleep 3600; done'0 in 200 in 20

On v1.18.2, 2026-09-24, the no-command row was 0 in 20 on macOS arm64 and 1 in 20 on Lima aarch64, where it had been 0 in 20 on both on v1.16.1.

The fix is to give the machine a workload that stays up, which is what scripts/create-dev-machine.sh does by passing one after -- on machine create. A Smolfile can carry it as well, as cmd (with entrypoint), and a command after -- replaces the Smolfile’s cmd.

And this is why the readiness probe has to assert a value. The message goes to stdout, so a probe that waits for empty output or a zero exit code reports ready while the container is still flapping, and the next three checks silently read that message as their answer. Both scripts here wait for the exact string WORKLOAD_READY.

A host mount source that goes missing stops every later start

If a machine created from a Smolfile has started before and now never starts again, check that every host path in volumes still exists. create refuses a path that is missing when it runs, with mount source not found: ./src; a later start does not check the path again, and in the run below every later start failed with:

Error: agent operation failed: start machine: agent operation failed: wait for ready:
agent did not become ready within 30 seconds

which names neither the mount nor the missing directory, and reads exactly like host load.

Measured on macOS 26.6.2 arm64 on v1.14.3, 2026-09-08, three create-start-stop-start cycles each with the workload confirmed ready before the stop:

host mount sourcerestarts
./src exists3 of 3
./src absent0 of 3

How ./src came to be missing in that run was not recorded. On v1.22.2 on macOS arm64, 2026-10-03, removing ./src after a machine’s first start reproduced it: the next start failed with the same message, and create with ./src already missing refused with Error: mount source not found: ./src.

scripts/create-dev-machine.sh runs mkdir -p ./src before creating, which is why the packet’s own flow does not hit this. Anything that writes its own Smolfile has to do the same. A relative path in volumes resolves against the working directory, so the same Smolfile run from two directories can behave differently.

An ad-hoc restart loop that omitted the mkdir -p produced 0 of 5 and looked like a release regression until the two were compared side by side.

machine shell does not start a stopped machine

Before v1.23.7 smolvm machine --help described shell as “Open an interactive shell in a machine (starts it if stopped)”; from v1.23.7 it says “in a running machine”. It does not start one:

$ smolvm machine stop --name dev
$ smolvm machine shell --name dev
Error: agent operation failed: connect: machine 'dev' is not running.
       Use 'smolvm machine start --name dev' first.

Verified both through a pipe and under a real pty, so it is not a TTY-detection effect, and again on v1.18.2. Start it explicitly first.

A host volumes mount was not writable by the app user

With volumes = ["./src:/app"] and user = "app" (a user adduser made in the guest), writing to /app failed with Operation not permitted: the mount carries host ownership and app is not its owner. Read source in through the mount, write build output somewhere else, or run that step as root. user also takes a numeric uid[:gid], which machine create --help gives as the way to match the owner of a mounted host directory: on v1.22.2 on macOS arm64, machine run -v "$PWD/src:/app" --user "$(id -u):$(id -g)" wrote /app/w and the file appeared on the host.

/tmp is wiped by a stop

Anything a provisioning step leaves in /tmp is gone on the next start, while the same step’s writes to $HOME or / persist. What survives, verified by writing one file per filesystem and restarting:

locationsurviveswhy
pip --user packagesyesunder $HOME, on the overlay
$HOME/... filesyesoverlay
files written at /yesoverlay
/tmpnoemptied at each start, like a tmpfs
/storage/...yesthe ext4 disk itself

Read from inside the guest, the mechanism is an overlay whose upper layer lives on the machine’s ext4 disk:

none on / type overlay (... upperdir=/storage/overlays/persistent-<name>/upper ...)
/dev/vda on /workspace type ext4 (rw,noatime)
/dev/vda on /storage   type ext4 (rw,noatime)

machine delete prompts and defaults to No

Through v1.16.x a scripted delete without --force printed Delete machine 'dev'? [y/N] Cancelled, exited 0 and left the machine in place while the script carried on. On v1.17.0 and later it exits 1 with needs confirmation but stdin is not a terminal; pass --force to delete it. Always pass --force in a script.

A VPN inside the machine takes its gateway and resolver

A machine on virtio-net gets the link 100.96.0.0/30 by default, with the gateway and resolver at .1. Tailscale and other carrier NAT VPNs claim 100.64.0.0/10, which contains it, so once the VPN is up inside the machine its gateway and resolver route into the VPN and every lookup fails. The throwaway-machine packet’s traps have the measurement: with the routes Tailscale adds, the default link gave bad address 'example.com' and --guest-subnet 10.200.0.0/30 with the same routes resolved and fetched, on v1.18.2 on both hosts.

For a dev machine the flag has three constraints, each observed on v1.18.2:

  • It is set at create only. machine update has no --guest-subnet, so changing it means a new machine. It survives stop and start.
  • A Smolfile cannot carry it. [network] accepts allow_hosts, allow_cidrs and credentials; pass --guest-subnet on machine create next to -s.
  • It implies --net and virtio-net, and machine ls --json then reports "network": false for a machine that does reach the network, so do not read that field as the answer.

Do not run two lifecycle commands against the same machine at once

Two overlapping stop and shell invocations left a machine stop unfinished for over two minutes, against 0.14 s for a single stop on the same machine. It was seen once, with the two calls overlapping, and was not reproduced as a fault; anything that fans out lifecycle calls should still serialize them per machine.

Assert the package version, not that the import worked

An import can be satisfied by a system copy and tells you nothing about whether your install survived the restart. scripts/verify-persistence.sh records the version before the stop and compares it after.