--- title: "Dev env: a persistent machine you re-enter" description: "A persistent machine re-entered across sessions." canonical: "https://smolmachines.com/docs/guides/skills/dev-env" source: "https://github.com/smol-machines/docs/blob/main/guides/skills/dev-env.md" --- # Dev env: a persistent machine you re-enter Keeps a persistent smolvm machine with its dependencies already installed and re-enters it cheaply across sessions. Use when a project needs an isolated development environment that survives stop and start; when deciding what belongs in a Smolfile's init versus what has to run on every boot; when a package installed in a machine has vanished after a restart; or when exec answers "the container smolvm- is not running". Do not use it for untrusted code, which needs a machine that leaves nothing behind (see the throwaway-machine packet), or for running a Docker daemon inside the machine (see docker-in-machine). Verified on **smolvm v1.29.0** on macOS arm64, 2026-10-15, and on **v1.18.2** on Linux aarch64, 2026-09-24; the Linux runs used the Smolfile with `memory = 1024`, for the host reason in "Platform arms". Done means a second `start` is fast, skips provisioning, and the packages installed in the first session are still there. The Linux runs used the scripts of their date; this version's preflight and cleanup scripts ran on Linux aarch64 on v1.22.2 on 2026-10-03. The whole use case turns on one fact: **`init` runs once, not on every start**, as [`smolfile.md`](https://github.com/smol-machines/smolvm/blob/main/docs/smolfile.md) says, and as the docs site (smolmachines.com/docs) says under "When init runs". `smolvm machine create --help` reads "Run command on every VM start" at v1.22.2, and "Run command once, on the machine's first start" from v1.23.7; provisioning that relies on the old line is missing on the second boot. ## Procedure **1. Preflight.** ```bash scripts/preflight.sh ``` Read-only. `restart_after_stop=verified` on macOS and Linux. The script covers macOS and Linux only; elsewhere it prints `unknown` and a note pointing at `references/windows.md`, where a stopped machine starts again on v1.14.6 and v1.22.2. **2. Declare the machine.** `assets/dev.smolfile` is a working starting point, and the one `verify-persistence.sh` is written against: it asserts the `app` user and the `/app` workdir that file sets. `examples/python-app/python.smolfile` and `examples/node-app/node.smolfile` in the repository are plainer examples without them. ```toml image = "python:3.12-alpine" net = true cpus = 2 memory = 2048 volumes = ["./src:/app"] init = [ "sh -c \"id -un > /init-ran-as.txt\"", "sh -c \"date +%s >> /init-count.txt\"", "adduser -D app", ] user = "app" workdir = "/app" ``` **3. Create and bring it up.** Run this from the project directory: the Smolfile mounts `./src`, which is relative to the working directory, and the script creates `./src` there if it is missing. ```bash scripts/create-dev-machine.sh # name defaults to smolskill-dev scripts/create-dev-machine.sh smolskill-myproj ./my.smolfile ``` It creates the machine **with an explicit long-lived workload command**, records the name for cleanup, starts it, waits for the workload container to answer with a value, and then reports what actually happened rather than that nothing errored: ``` init_ran=yes machine_running=yes workload_ready_after_s=0 workload_ready=yes init_ran_as=root exec_user=app workdir=/app result=up ``` `init_ran_as=root` with `exec_user=app` is the correct outcome, not a bug: `init` provisions the machine and runs as root regardless of `user`, while `exec` and `shell` run as `user`. **4. Work in it.** ```bash smolvm machine exec --name smolskill-dev -- pip install --user requests smolvm machine exec --name smolskill-dev --user root -- sh -c 'mkdir -p /storage/keep' smolvm machine shell --name smolskill-dev # interactive, lands in workdir as `user` ``` **5. Prove it is worth keeping.** ```bash scripts/verify-persistence.sh ``` It installs a package and records its **version**, seeds one file per filesystem, stops, starts, and then asserts each value against what it recorded. A version comparison is the point: an import that does not crash can be satisfied by a system copy and says nothing about your install. It leaves the machine running. When the setup is done, `smolvm machine stop --name smolskill-dev` and give the user its name; the next `machine start` brings it back as it was. **6. Clean up, when the machine is no longer wanted.** Not at the end of a setup: the machine is the deliverable, so leave it stopped and tell the user its name. ```bash scripts/cleanup.sh --purge ``` `cleanup.sh` waits up to 20 seconds, polling the machine list, and prints `waiting=up to 20s` first: an ephemeral machine's entry retires after its run returns. ## What `init` and `user` actually do, observed on v1.14.2 and again on v1.22.2 Observed, not inferred: | question | observed | |---|---| | when does `init` run? | on the **first `start`**, not on `create`, and not again | | second start | prints `Init already completed, skipping N command(s)` | | which user runs `init`? | **root**, even with `user = "app"` set | | which user runs `exec` and `shell`? | the Smolfile `user` | | override per command | `machine exec --user root` works | | is there `machine start --init`? | **no** | ## What survives a stop and start | location | survives | why | |---|---|---| | pip `--user` packages | yes | under `$HOME`, on the overlay | | `$HOME/...` files | yes | overlay | | files written at `/` | yes | overlay | | `/tmp` | **no** | `tmpfs` | | `/storage/...` | yes | the ext4 disk itself | The root filesystem is an overlay whose upper layer lives on the machine's ext4 disk, which is why writes to `/` persist while `tmpfs` mounts do not. ## Traps Full detail in `references/traps.md`. The ones that cost the most: - **`init` runs once** (top of this file). Anything that must hold on every boot, a bind mount above all, runs in the command that needs it. - **Give a machine you `exec` into a long-lived workload.** Without one, an `exec` right after `start` can answer ``the container `smolvm-` is not running`` instead of running. - **`machine shell` does not start a stopped machine** on v1.22.2, while its help says "starts it if stopped". - **A machine that runs Tailscale or another carrier NAT VPN needs `--guest-subnet` at create**, which a Smolfile cannot carry: `smolvm machine create --name smolskill-dev -s dev.smolfile --guest-subnet 10.200.0.0/30`. - `create` is instant and proves nothing, and a host `volumes` mount was not writable by the `app` user. ## Security defaults, and why they are the defaults - **`user` in the Smolfile is what your workload runs as, and it is not root.** `init` running as root is the provisioning step, deliberately separated from the workload's identity. Keep them separate rather than setting `user = "root"` to make a mount writable: the mount carries host ownership, and running the workload as root to reach it hands root in the guest everything the mount exposes on the host. - **A `volumes` mount is host authority handed to the guest**, so mount the narrowest directory that works and prefer `:ro` for anything the machine only reads. Treat root in the guest as untrusted, as the [security model](https://github.com/smol-machines/smolvm/blob/main/docs/security-model.md) says: every forwarded mount, port and network permission becomes part of the workload's authority. - **`net = true` is outbound access for the whole machine.** A dev machine that only needs a package index does not need general egress; `--allow-host` scopes it. - **These scripts wrap the public CLI only.** They edit no smolvm configuration and nothing under `~/.smolvm`, and cleanup deletes only names it recorded under the `smolskill-` prefix. ## Platform arms - **Linux aarch64**: the scripts were run here on v1.18.2 with the Smolfile's memory at 1024 MiB, and once on v1.22.2 at 1024. At 2048 on v1.18.2 the restart failed with `agent did not become ready within 30 seconds`: on a small or busy host lower `memory` before suspecting smolvm, and check the mount source first (the `install` packet's traps have the numbers). - **macOS arm64**: the scripts were run here on v1.29.0 as shipped, and every check passed. - **Linux x86_64**: verified on v1.14.2 on an NVIDIA A10 cloud host, not re-run since. - **Windows x86_64**: `references/windows.md`, **re-run on 2026-10-03 against v1.22.2** on Windows 11 Home build 10.0.26200 UBR 9457. Create, three stops and three starts, with a marker read back after each start: **a stopped machine starts again and its state survives**, as on v1.14.6, where on v1.14.2 it did not. That page also carries the WHP device ceiling a Windows preflight needs through v1.22.2: four `-v` mounts, three when a port is published. ## Eval prompts, and what they produced **1. "Set me up a Python dev machine I can come back to, and prove the packages survive a restart."** On macOS arm64: the step 3 output above, on v1.22.2. Then, after the stop and start, `verify-persistence.sh`: ``` workload_ready_after_s=0 installed_version=2.34.2 Init already completed, skipping 3 command(s) workload_ready_after_s=0 init_ran_once=ok (yes) package_version=ok (2.34.2) home_file=ok (SURVIVES) storage_file=ok (SURVIVES) tmp_file=ok (WIPED) exec_user=ok (app) workdir=ok (/app) result=persistent ``` **2. "My bind mount is gone after I restarted the machine. It is right there in `init`."** `init` ran once. The second start reports it, on v1.22.2 as in eval 1: ``` Init already completed, skipping 3 command(s) ``` There is no `machine start --init`, so the mount has to be re-applied by the command that needs it. `docker-in-machine` is the packet built entirely around this. **3. "`smolvm machine exec` just told me the container is not running, but the machine is running."** On v1.14.2, reproduced deliberately by creating the same machine without a workload command, then running 20 execs, on Lima: ``` 3: the container `smolvm-e51bb94127b751bf` is not running failures=1/20 4: the container `smolvm-32edffb50eaf0d20` is not running failures_after_restart=1/20 ``` and with the explicit long-lived workload `scripts/create-dev-machine.sh` passes: ``` failures=0/20 failures_after_restart=0/20 ``` The changing hash is the tell: the workload container is being relaunched between execs. ## Re-verified on v1.29.0 Run 2026-10-15 PT against v1.29.0 from the published release, checksum checked, under a fresh isolated `HOME` on macOS 27.0.1 arm64, the scripts twice, the second time from a fresh `HOME`, and the checks beyond them once. Linux was not run, so the Linux stamp stays on its earlier release. `create-dev-machine.sh` gave `result=up` with `init_ran_as=root`, `exec_user=app` and `workdir=/app`; `verify-persistence.sh` gave `init_ran_once=ok (yes)`, `package_version=ok (2.34.2)`, `home_file=ok (SURVIVES)`, `storage_file=ok (SURVIVES)`, `tmp_file=ok (WIPED)` and `result=persistent`, and the second start printed `Init already completed, skipping 3 command(s)`. ## Re-verified on v1.22.2 Run 2026-10-03 PT against v1.22.2 from the published release, checksum checked, under an isolated `HOME` on macOS 27.0.1 arm64, twice, the second time from a fresh `HOME`. On Lima `linux-kvm` (Ubuntu 24.04 aarch64) guests above 2048 MiB timed out on 2026-10-03, so the Linux lines below are a single run and the Linux stamp stays on its earlier release. macOS: `result=up` with `init_ran_as=root` and `exec_user=app`, then `result=persistent`, `package_version=ok (2.34.2)`, `tmp_file=ok (WIPED)`. `machine create --help` describes `--init` as running "on every VM start", `machine shell` on a stopped machine answered `is not running. Use 'smolvm machine start ...' first` while its help says it starts one, and the exec race gave 0 of 20. Linux aarch64 at `memory = 1024`: `result=persistent`, the exec race 0 of 20. Linux was last verified on v1.18.2, at `memory = 1024`. ## What was not run - **Windows beyond the restart.** The v1.22.2 run covered create, three stops and starts, the marker and `init` running once; the WHP device ceiling was confirmed by the `install` packet's Windows run on 2026-10-03. - **Linux x86_64.** - **`init` after a checkpoint restore.** Nowhere, on any platform. - **The `USER` interaction still settling.** `init` runs as root on v1.22.2, and whether the image's own `USER` still governs it in some paths is open as [smolvm#1189](https://github.com/smol-machines/smolvm/issues/1189). Nothing here tested an image with a `USER` line. ## Related packets - `install` for the boot this assumes, and `teardown` for the cleanup script. - `docker-in-machine` for the clearest case of the `init`-runs-once trap. - `pack` for turning the machine this packet builds into a portable artifact. ## Scripts The files the procedure runs, in the order it runs them. It calls each one by the path in its heading, relative to the directory the procedure is saved in. ### `scripts/preflight.sh` ```bash #!/usr/bin/env bash # Report whether this host can keep a persistent smolvm dev machine. Read-only: # starts no VM, writes no smolvm state, changes no group membership. # # Output is one key=value per line so a caller can parse it. The last line is # always result=ready or result=blocked. set -uo pipefail VERIFIED_VERSION="1.29.0" emit() { printf '%s=%s\n' "$1" "$2"; } blocked=0 note() { printf 'note=%s\n' "$1"; } # --- the binary -------------------------------------------------------------- SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}" if [ -z "$SMOLVM" ]; then emit smolvm_installed no emit smolvm_version "" blocked=1 else emit smolvm_installed yes version="$("$SMOLVM" --version 2>/dev/null | awk '{print $NF}')" emit smolvm_version "${version:-unknown}" fi emit verified_version "$VERIFIED_VERSION" if [ -n "${version:-}" ] && [ "$version" != "unknown" ]; then if [ "$version" = "$VERIFIED_VERSION" ]; then emit version_status match else newest="$(printf '%s\n%s\n' "$version" "$VERIFIED_VERSION" | sort -V | tail -1)" if [ "$newest" = "$version" ]; then emit version_status newer note "this packet was verified on $VERIFIED_VERSION and the binary is $version; flags and messages move every release, so check the output against the binary before trusting a step here" else emit version_status older note "this packet was verified on $VERIFIED_VERSION and the binary is $version" fi fi else emit version_status unknown fi # --- platform ---------------------------------------------------------------- kernel="$(uname -s)" arch="$(uname -m)" case "$arch" in aarch64|arm64) arch=aarch64 ;; esac case "$kernel" in Darwin) emit platform "darwin-$arch" emit accel hvf emit macos_version "$(sw_vers -productVersion)" hv="$(sysctl -n kern.hv_support 2>/dev/null)" if [ "$hv" = "1" ]; then emit accel_access ok; else emit accel_access denied; blocked=1; fi if [ "$arch" != "aarch64" ]; then emit hardware_verified no note "Intel Mac is not verified by this packet; the installer accepts it and nothing here was run on one" else emit hardware_verified yes fi # A VM's agent socket lives under the cache directory. macOS sockaddr_un # holds 104 bytes including the terminator. sock="$HOME/Library/Caches/smolvm/vms/0123456789abcdef/agent.sock" len=${#sock} emit socket_path_bytes "$len" if [ "$len" -gt 100 ]; then emit socket_path_status too_long blocked=1 note "HOME is too deep: every VM start will fail. From v1.28.0 the error names the socket path and its length; before it, krun_start_enter -22, whose text blames disks and device options. Install under a shorter HOME." else emit socket_path_status ok fi emit unsupported "cuda" emit restart_after_stop verified ;; Linux) emit platform "linux-$arch" emit accel kvm emit socket_path_status n_a if [ ! -e /dev/kvm ]; then emit accel_access missing blocked=1 note "/dev/kvm does not exist; this host has no KVM" elif [ -r /dev/kvm ] && [ -w /dev/kvm ]; then emit accel_access ok else emit accel_access denied blocked=1 note "your user cannot open /dev/kvm. The installer warns and continues, so a successful install says nothing about whether a VM will start. Fix: sudo usermod -aG kvm \$USER, then run the next command through sg kvm -c '...' rather than logging out." fi emit restart_after_stop verified ;; *) emit platform "unsupported-$kernel" emit accel unknown emit accel_access unknown blocked=1 emit restart_after_stop unknown note "this script covers macOS and Linux. On Windows a stopped machine starts again, on v1.14.6 and v1.22.2, and through v1.22.2 the WHP device budget caps you at four -v mounts, three when a port is published. See references/windows.md." ;; esac if [ "$blocked" -eq 0 ]; then emit result ready; else emit result blocked; fi ``` ### `assets/dev.smolfile` ```toml # A persistent development machine. `smolvm machine create -s dev.smolfile`. # # The two keys worth understanding before you edit this: # # init runs ONCE, on the first start, and as root. It does not run again on # later starts, so anything that has to be true on every boot does not # belong here. # user governs `exec` and `shell`, not `init`. # # The workload command is not set here. A Smolfile can carry one as `cmd`; # this packet passes it after `--` on `machine create`, as # scripts/create-dev-machine.sh does: without one the image's own CMD becomes # the persistent workload, and for an interpreter image that command exits at # once and `exec` starts racing the relaunch. image = "python:3.12-alpine" net = true cpus = 2 memory = 2048 # A host mount carries host ownership, so `user` below (made by adduser) could # not write to it. Read source in, write build output somewhere else. volumes = ["./src:/app"] init = [ "sh -c \"id -un > /init-ran-as.txt\"", "sh -c \"date +%s >> /init-count.txt\"", "adduser -D app", ] user = "app" workdir = "/app" ``` ### `scripts/create-dev-machine.sh` ```bash #!/usr/bin/env bash # Create a persistent dev machine and bring it up for the first time. # # usage: create-dev-machine.sh [] [] # name default smolskill-dev (the prefix cleanup.sh will delete) # smolfile default ../assets/dev.smolfile # # `create` is configuration only: it returns in milliseconds and pulls nothing. It # does refuse a host mount source that does not exist; the pull, the init commands # and any of their failures land on the first `start`, so a fast create is not # evidence that anything works. set -uo pipefail here="$(cd "$(dirname "$0")" && pwd)" NAME="${1:-smolskill-dev}" SMOLFILE="${2:-$here/../assets/dev.smolfile}" SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}" if [ -z "$SMOLVM" ]; then printf 'smolvm not found; set SMOLVM to its path\n' >&2 exit 2 fi case "$NAME" in smolskill-*) ;; *) printf 'name must start with smolskill- so cleanup.sh will delete it\n' >&2; exit 2 ;; esac # The Smolfile mounts ./src, which is relative to the working directory. mkdir -p ./src # Give the machine a workload that stays up. Without a command, `create` launches # the image's own ENTRYPOINT/CMD as the persistent workload; for an interpreter # image such as python:3.12-alpine that command reads EOF and exits at once, the # container is relaunched, and `exec` then intermittently answers "the container # `smolvm-` is not running" instead of running your command. Measured on a # nested-virt aarch64 host: 1 exec in 20 failed that way with no command, 0 in 20 # with this one, both on a fresh machine and immediately after a restart. create_out="$("$SMOLVM" machine create --name "$NAME" -s "$SMOLFILE" \ -- sh -c 'while true; do sleep 3600; done' 2>&1)" create_rc=$? printf '%s\n' "$create_out" | sed 's/^/ /' if [ "$create_rc" -ne 0 ]; then printf 'result=failed (create)\n'; exit 1; fi "$here/cleanup.sh" --record "$NAME" start_out="$("$SMOLVM" machine start --name "$NAME" 2>&1)" printf '%s\n' "$start_out" | sed 's/^/ /' fail=0 # `start` prints the init line only when the Smolfile declares init commands. if printf '%s' "$start_out" | grep -q 'Running .* init command'; then printf 'init_ran=yes\n' elif ! grep -q '^[[:space:]]*init[[:space:]]*=' "$SMOLFILE"; then printf 'init_ran=none_declared\n' else printf 'init_ran=no\n' fail=1 fi if printf '%s' "$start_out" | grep -q "Machine '$NAME' running"; then printf 'machine_running=yes\n' else printf 'machine_running=no\n' fail=1 fi # `start` returns before the workload container is up. Wait for a VALUE: an empty # result and a zero exit code both pass while the container is still coming up, # so a probe that asserts either of those reports ready too early and the next # three checks read a message as their answer. wait_for_workload() { # 120 s: the slowest first start observed while building this packet was # under 20 s on a nested-virt aarch64 host, so this is six times the worst # case and still short enough that a real hang is reported, not waited on. waited=0 while [ "$waited" -lt 120 ]; do out="$("$SMOLVM" machine exec --name "$1" -- sh -c 'echo WORKLOAD_READY' 2>&1 | tr -d '\r')" if [ "$out" = "WORKLOAD_READY" ]; then printf 'workload_ready_after_s=%s\n' "$waited" return 0 fi sleep 1 waited=$((waited + 1)) done printf 'workload_ready_after_s=timeout last_probe=%s\n' "$out" return 1 } if wait_for_workload "$NAME"; then printf 'workload_ready=yes\n' else printf 'workload_ready=no\n' fail=1 fi # init runs as root even with `user` set. Asserting the value rather than the # absence of an error is the point: an exec that fails still leaves you guessing. printf 'init_ran_as=%s\n' "$("$SMOLVM" machine exec --name "$NAME" --user root -- cat /init-ran-as.txt 2>&1 | tr -d '\r')" printf 'exec_user=%s\n' "$("$SMOLVM" machine exec --name "$NAME" -- id -un 2>&1 | tr -d '\r')" printf 'workdir=%s\n' "$("$SMOLVM" machine exec --name "$NAME" -- pwd 2>&1 | tr -d '\r')" if [ "$fail" -eq 0 ]; then printf 'result=up\n'; else printf 'result=failed\n'; fi exit "$fail" ``` ### `scripts/verify-persistence.sh` ```bash #!/usr/bin/env bash # Prove the machine is worth keeping: that a package installed in one session is # still there in the next, that init does not run again, and that the parts of # the filesystem people assume persist actually do. # # usage: verify-persistence.sh [] (default smolskill-dev) # # Every check asserts a value. Asserting that an import did not crash proves # nothing here: a system copy of the package satisfies it and tells you nothing # about whether your install survived. set -uo pipefail NAME="${1:-smolskill-dev}" SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}" if [ -z "$SMOLVM" ]; then printf 'smolvm not found; set SMOLVM to its path\n' >&2 exit 2 fi fail=0 check() { if [ "$2" = "$3" ]; then printf '%s=ok (%s)\n' "$1" "$2" else printf '%s=FAIL expected=%s actual=%s\n' "$1" "$3" "$2" fail=1 fi } exec_in() { "$SMOLVM" machine exec --name "$NAME" -- "$@" 2>&1 | tr -d '\r'; } # `start` returns before the workload container is up, and an `exec` in that # window answers "the container `smolvm-` is not running" on stdout while # still exiting zero. Wait for a VALUE: an empty result or a zero exit code both # pass while the container is still coming up, which is how this window gets # missed. Observed on a nested-virt aarch64 host, where a probe that asserted # only an empty result let three later checks read that message as their answer. wait_for_workload() { # 120 s: six times the slowest start observed while building this packet. # Long enough to absorb a loaded host, short enough to report a real hang. waited=0 while [ "$waited" -lt 120 ]; do probe="$(exec_in sh -c 'echo WORKLOAD_READY')" if [ "$probe" = "WORKLOAD_READY" ]; then printf 'workload_ready_after_s=%s\n' "$waited" return 0 fi sleep 1 waited=$((waited + 1)) done printf 'workload_ready_after_s=timeout last_probe=%s\n' "$probe" return 1 } if ! wait_for_workload; then printf 'result=FAILED (workload container never came up)\n' exit 1 fi # 1. Install something and record its version, not its presence. "$SMOLVM" machine exec --name "$NAME" -- pip install --quiet --user requests >/dev/null 2>&1 before="$(exec_in python3 -c 'import requests; print(requests.__version__)')" printf 'installed_version=%s\n' "$before" case "$before" in [0-9]*) ;; *) printf 'installed_version=FAIL (not a version)\n'; fail=1 ;; esac # 2. Seed one file per filesystem so the stop tells them apart. # shellcheck disable=SC2016 # $HOME must expand inside the guest, not here "$SMOLVM" machine exec --name "$NAME" -- sh -c 'echo SURVIVES > "$HOME/keep.txt"; echo GONE > /tmp/scratch.txt' >/dev/null 2>&1 "$SMOLVM" machine exec --name "$NAME" --user root -- sh -c 'mkdir -p /storage/keep && echo SURVIVES > /storage/keep/disk.txt' >/dev/null 2>&1 # 3. Stop and come back. "$SMOLVM" machine stop --name "$NAME" >/dev/null 2>&1 restart_out="$("$SMOLVM" machine start --name "$NAME" 2>&1)" printf '%s\n' "$restart_out" | sed 's/^/ /' if ! wait_for_workload; then fail=1 fi if printf '%s' "$restart_out" | grep -q 'Init already completed'; then check init_ran_once yes yes else check init_ran_once no yes fi after="$(exec_in python3 -c 'import requests; print(requests.__version__)')" check package_version "$after" "$before" # shellcheck disable=SC2016 # same: the guest resolves $HOME check home_file "$(exec_in sh -c 'cat "$HOME/keep.txt" 2>/dev/null')" SURVIVES check storage_file "$(exec_in cat /storage/keep/disk.txt)" SURVIVES # /tmp is tmpfs and is wiped by a stop. Anything a provisioning step leaves # there is gone on the next start, while the same step's writes to $HOME or / # persist. This is the asymmetry that surprises people. check tmp_file "$(exec_in sh -c 'cat /tmp/scratch.txt 2>/dev/null || echo WIPED')" WIPED check exec_user "$(exec_in id -un)" app check workdir "$(exec_in pwd)" /app if [ "$fail" -eq 0 ]; then printf 'result=persistent\n'; else printf 'result=FAILED\n'; fi exit "$fail" ``` ### `scripts/cleanup.sh` ```bash #!/usr/bin/env bash # Delete the machines this packet's scripts created, then prove the host is clean. # # Only machines recorded in the state file are deleted, with --cascade, so any # machine branched from one of them goes too, whatever its name. Any other # machine you or another session created by hand is never deleted. Scripts # record a name by calling: cleanup.sh --record # # usage: cleanup.sh [--record ] [--reap] [--purge] # --record add a machine name to the state file and exit # --reap kill every VM process under this HOME; run without it first # --purge also remove the state file once the list is empty set -uo pipefail PACKET="dev-env" PREFIX="smolskill-" SMOLVM="${SMOLVM:-$(command -v smolvm 2>/dev/null)}" STATE_DIR="${SMOLVM_SKILL_STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/smolvm-skills}" STATE_FILE="$STATE_DIR/$PACKET.machines" reap=0 purge=0 while [ $# -gt 0 ]; do case "$1" in --record) mkdir -p "$STATE_DIR" printf '%s\n' "$2" >> "$STATE_FILE" exit 0 ;; --reap) reap=1 ;; --purge) purge=1 ;; *) printf 'unknown argument: %s\n' "$1" >&2; exit 2 ;; esac shift done if [ -z "$SMOLVM" ]; then printf 'smolvm not found; set SMOLVM to its path\n' >&2 exit 2 fi # With SMOLVM_DATA_DIR set, smolvm runs with HOME there on Linux, so its cache # is under .cache in it. case "$(uname -s)" in Darwin) VMS_DIR="$HOME/Library/Caches/smolvm/vms" ;; *) cache_root="${SMOLVM_DATA_DIR:+$SMOLVM_DATA_DIR/.cache}" VMS_DIR="${cache_root:-${XDG_CACHE_HOME:-$HOME/.cache}}/smolvm/vms" ;; esac VMS_DIR="${SMOLVM_VMS_DIR:-$VMS_DIR}" SMOLVM_PREFIX="${SMOLVM_PREFIX:-$HOME/.smolvm}" if [ -n "${SMOLVM:-}" ]; then case "$(readlink "$SMOLVM" 2>/dev/null || printf '%s' "$SMOLVM")" in "$SMOLVM_PREFIX"/*) ;; *) printf 'note=%s is not under %s, so the process scan cannot see its forked VMs (all of its VMs on macOS); set SMOLVM_PREFIX to its directory\n' "$SMOLVM" "$SMOLVM_PREFIX" ;; esac fi # List this HOME's VM processes as "pid marker". The plain run path execs a # `_boot-vm` child that carries its boot config; the pack-run path forks one that # carries none. Linux names both `libkrun VM`, or `VM:` when HOSTNAME # is exported; on macOS the executable path and the parent chain scope the # search. The teardown packet's traps have the why. list_vm_processes() { case "$(uname -s)" in Linux) for p in /proc/[0-9]*; do case "$(cat "$p/comm" 2>/dev/null)" in "libkrun VM"|VM:*) ;; *) continue ;; esac pid="${p#/proc/}" cfg="$(tr '\0' '\n' < "$p/cmdline" 2>/dev/null | sed -n '3p')" case "$cfg" in "$VMS_DIR"/*) printf '%s %s\n' "$pid" "$cfg"; continue ;; esac case "$(readlink "$p/exe" 2>/dev/null)" in "$SMOLVM_PREFIX"/*) printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX" ;; esac done ;; Darwin) # shellcheck disable=SC2009 # pgrep -f would match this script. procs="$(ps -axo pid=,ppid=,command= 2>/dev/null)" printf '%s\n' "$procs" | while read -r pid ppid rest; do case "$rest" in "$SMOLVM_PREFIX"/smolvm-bin*) ;; *) continue ;; esac case "$rest" in *" _boot-vm "*) printf '%s %s\n' "$pid" "${rest#* _boot-vm }"; continue ;; esac # An orphan counts only if it is a run; a fork has its parent's command line. if [ "$ppid" = 1 ]; then case "$rest" in *" machine run "*|*" vm run "*|*" pack run "*) printf '%s orphaned-under %s\n' "$pid" "$SMOLVM_PREFIX" ;; esac else parent="$(printf '%s\n' "$procs" | while read -r q qp qrest; do [ "$q" = "$ppid" ] && { printf '%s' "$qrest"; break; }; done)" [ "$parent" = "$rest" ] && printf '%s forked-under %s\n' "$pid" "$SMOLVM_PREFIX" fi done ;; esac } # 1. Delete recorded machines. Without --force a delete prompts and defaults to # No; --cascade removes branch children, which otherwise block it. if [ -s "$STATE_FILE" ]; then while read -r name; do [ -n "$name" ] || continue case "$name" in "$PREFIX"*) ;; *) printf 'skipping %s: not created by this packet (no %s prefix)\n' "$name" "$PREFIX" continue ;; esac # Only names still listed: a second stop of a missing name leaves a # directory that reads as a leak. The list is read first because grep -q # under pipefail can fail the pipeline and skip a listed machine. listed="$("$SMOLVM" machine list /dev/null | awk 'NR>2{print $1}')" grep -qx -- "$name" <<<"$listed" || continue "$SMOLVM" machine stop --name "$name" /dev/null 2>&1 "$SMOLVM" machine delete --name "$name" --force --cascade &1 | sed 's/^/ /' done < "$STATE_FILE" fi # 2. An ephemeral machine's entry retires after its run returns: poll, 20 s at most. printf 'waiting=up to 20s for ephemeral entries to retire before asserting\n' waited=0 listing="$("$SMOLVM" machine list 2>&1)" while ! grep -q 'No machines found' <<<"$listing" && [ "$waited" -lt 20 ]; do sleep 1; waited=$((waited + 1)) listing="$("$SMOLVM" machine list 2>&1)" done printf 'waited=%ss\n' "$waited" # 3. Assert the value, not the exit code. if grep -q 'No machines found' <<<"$listing"; then printf 'machines=clean\n' [ "$purge" -eq 1 ] && rm -f "$STATE_FILE" else printf 'machines=remaining\n' printf '%s\n' "$listing" | sed 's/^/ /' printf 'note=this packet did not create these, so it will not delete them. By name:\n' printf ' smolvm machine stop --name && smolvm machine delete --name --force\n' printf ' add --cascade for a machine that was branched from another\n' fi # 4. Report VM processes under this HOME's state, such as a killed wrapper's. # With --reap every one is killed, including a machine another packet kept. found=0 while read -r pid cfg; do [ -n "$pid" ] || continue found=1 printf 'vm_process=%s config=%s\n' "$pid" "$cfg" if [ "$reap" -eq 1 ]; then kill -9 "$pid" 2>/dev/null && printf ' killed %s\n' "$pid" fi done <` is not running``**, the workload container is not up and the exec did nothing. The hash differs between occurrences, because the container is being relaunched. **Why it happens.** Without a command, `machine create` launches the image's own ENTRYPOINT or CMD as the persistent workload (`machine create --help`, `[COMMAND]`). For an interpreter image such as `python:3.12-alpine` that command reads EOF from a stdin nobody is holding and exits at once, so the container dies and is relaunched, and an `exec` can land in the gap. Measured on a nested-virt aarch64 host on 2026-09-07, 20 execs each on a fresh machine and again immediately after a restart: | workload command | failures, fresh | failures, after restart | |---|---|---| | none (image CMD) | 1 in 20 | 1 in 20 | | `sh -c 'while true; do sleep 3600; done'` | 0 in 20 | 0 in 20 | **On v1.18.2**, 2026-09-24, the no-command row was 0 in 20 on macOS arm64 and 1 in 20 on Lima aarch64, where it had been 0 in 20 on both on v1.16.1. **The fix is to give the machine a workload that stays up**, which is what `scripts/create-dev-machine.sh` does by passing one after `--` on `machine create`. A Smolfile can carry it as well, as `cmd` (with `entrypoint`), and a command after `--` replaces the Smolfile's `cmd`. **And this is why the readiness probe has to assert a value.** The message goes to stdout, so a probe that waits for empty output or a zero exit code reports ready while the container is still flapping, and the next three checks silently read that message as their answer. Both scripts here wait for the exact string `WORKLOAD_READY`. ### A host mount source that goes missing stops every later start **If a machine created from a Smolfile has started before and now never starts again**, check that every host path in `volumes` still exists. `create` refuses a path that is missing when it runs, with `mount source not found: ./src`; a later `start` does not check the path again, and in the run below every later `start` failed with: ``` Error: agent operation failed: start machine: agent operation failed: wait for ready: agent did not become ready within 30 seconds ``` which names neither the mount nor the missing directory, and reads exactly like host load. Measured on macOS 26.6.2 arm64 on v1.14.3, 2026-09-08, three create-start-stop-start cycles each with the workload confirmed ready before the stop: | host mount source | restarts | |---|---| | `./src` exists | **3 of 3** | | `./src` absent | **0 of 3** | How `./src` came to be missing in that run was not recorded. On v1.22.2 on macOS arm64, 2026-10-03, removing `./src` after a machine's first start reproduced it: the next `start` failed with the same message, and `create` with `./src` already missing refused with `Error: mount source not found: ./src`. `scripts/create-dev-machine.sh` runs `mkdir -p ./src` before creating, which is why the packet's own flow does not hit this. **Anything that writes its own Smolfile has to do the same.** A relative path in `volumes` resolves against the working directory, so the same Smolfile run from two directories can behave differently. An ad-hoc restart loop that omitted the `mkdir -p` produced 0 of 5 and looked like a release regression until the two were compared side by side. ### `machine shell` does not start a stopped machine Before v1.23.7 `smolvm machine --help` described `shell` as "Open an interactive shell in a machine (starts it if stopped)"; from v1.23.7 it says "in a running machine". It does not start one: ``` $ smolvm machine stop --name dev $ smolvm machine shell --name dev Error: agent operation failed: connect: machine 'dev' is not running. Use 'smolvm machine start --name dev' first. ``` Verified both through a pipe and under a real pty, so it is not a TTY-detection effect, and again on v1.18.2. Start it explicitly first. ### A host `volumes` mount was not writable by the `app` user With `volumes = ["./src:/app"]` and `user = "app"` (a user `adduser` made in the guest), writing to `/app` failed with `Operation not permitted`: the mount carries host ownership and `app` is not its owner. Read source in through the mount, write build output somewhere else, or run that step as root. `user` also takes a numeric `uid[:gid]`, which `machine create --help` gives as the way to match the owner of a mounted host directory: on v1.22.2 on macOS arm64, `machine run -v "$PWD/src:/app" --user "$(id -u):$(id -g)"` wrote `/app/w` and the file appeared on the host. ### `/tmp` is tmpfs and is wiped by a stop Anything a provisioning step leaves in `/tmp` is gone on the next start, while the same step's writes to `$HOME` or `/` persist. What survives, verified by writing one file per filesystem and restarting: | location | survives | why | |---|---|---| | pip `--user` packages | yes | under `$HOME`, on the overlay | | `$HOME/...` files | yes | overlay | | files written at `/` | yes | overlay | | `/tmp` | **no** | `tmpfs` | | `/storage/...` | yes | the ext4 disk itself | Read from inside the guest, the mechanism is an overlay whose upper layer lives on the machine's ext4 disk: ``` none on / type overlay (... upperdir=/storage/overlays/persistent-/upper ...) /dev/vda on /workspace type ext4 (rw,noatime) /dev/vda on /storage type ext4 (rw,noatime) ``` ### `machine delete` prompts and defaults to No Through v1.16.x a scripted delete without `--force` printed `Delete machine 'dev'? [y/N] Cancelled`, exited 0 and left the machine in place while the script carried on. On v1.17.0 and later it exits 1 with `needs confirmation but stdin is not a terminal; pass --force to delete it`. Always pass `--force` in a script. ### A VPN inside the machine takes its gateway and resolver A machine on virtio-net gets the link `100.96.0.0/30` by default, with the gateway and resolver at `.1`. Tailscale and other carrier NAT VPNs claim `100.64.0.0/10`, which contains it, so once the VPN is up inside the machine its gateway and resolver route into the VPN and every lookup fails. The `throwaway-machine` packet's traps have the measurement: with the routes Tailscale adds, the default link gave `bad address 'example.com'` and `--guest-subnet 10.200.0.0/30` with the same routes resolved and fetched, on v1.18.2 on both hosts. For a dev machine the flag has three constraints, each observed on v1.18.2: - **It is set at `create` only.** `machine update` has no `--guest-subnet`, so changing it means a new machine. It survives stop and start. - **A Smolfile cannot carry it.** `[network]` accepts `allow_hosts`, `allow_cidrs` and `credentials`; pass `--guest-subnet` on `machine create` next to `-s`. - **It implies `--net` and virtio-net**, and `machine ls --json` then reports `"network": false` for a machine that does reach the network, so do not read that field as the answer. ### Do not run two lifecycle commands against the same machine at once Two overlapping `stop` and `shell` invocations left a `machine stop` unfinished for over two minutes, against 0.14 s for a single `stop` on the same machine. It was seen once, with the two calls overlapping, and was not reproduced as a fault; anything that fans out lifecycle calls should still serialize them per machine. ### Assert the package version, not that the import worked An import can be satisfied by a system copy and tells you nothing about whether your install survived the restart. `scripts/verify-persistence.sh` records the version before the stop and compares it after.