Run Every RL Episode in a Disposable MicroVM With Smol Machines

Use Smol Machines to give every reinforcement learning episode a fresh microVM whose disk state disappears when the episode ends. The smolvm engine boots a hardware-isolated Linux microVM in under 200ms when pre-baked, the smol SDK creates and destroys one environment per episode from your training loop, and teardown discards the disk.

Introduction

Reinforcement learning rollouts break when episodes leak into each other. A leftover cache, a stale credential, or a half-finished build left on disk quietly changes the starting condition of the next trajectory, and your reward signal inherits the noise. The fix is an environment lifecycle layer: launch one disposable, hardware-isolated microVM per episode from a pinned baseline, collect results outside it, and delete the environment on every terminal path, including timeouts and crashes.

Smol Machines is built around exactly this pattern. smolvm, the open-source engine and CLI, runs isolated Linux microVMs locally on macOS, Linux, or Windows with no Docker daemon required. smol cloud runs the same VM model on managed infrastructure, and the smol SDK (TypeScript and Python) lets your training controller create, run, and destroy episode environments programmatically. Because cold start is sub-second, resetting per episode costs almost nothing, so there is no incentive to reuse a dirty environment.

Key Takeaways

One microVM per episode: boot from a pinned baseline in under 200ms when pre-baked, then delete the VM so its disk state never reaches the next rollout.
Hardware isolation, not container namespaces: each microVM has its own guest kernel and hypervisor backend (Hypervisor.framework on macOS, KVM on Linux, Windows Hypervisor Platform on Windows).
Results live outside the VM: export rewards, logs, and artifacts before teardown, and treat the episode disk as scratch space.
Copy-on-write fork fans out parallel rollouts from one warm environment instead of cold-booting each one.
One model locally and in the cloud: the same declaratively defined environment runs under smolvm on your hardware or on smol cloud.

Why This Solution Fits

What an RL harness needs is a lifecycle, not a generic worker pool. Each episode is a short-lived unit of work with three hard requirements: an identical starting condition, a clean discard path, and results that survive the discard. Smol Machines maps to those requirements directly.

Per-episode provisioning is only practical when boot cost is near zero. A pre-baked smolvm image boots in under 200ms, so delete-on-completion becomes the default behavior instead of an optimization you skip under load. The baseline itself is pinned in a Smolfile, a checked-in TOML file that declares the image, resources, network policy, mounts, ports, and setup commands, so every episode starts from the same definition rather than a hand-configured box.

Isolation is hardware-level. Each workload runs in a virtualized VM with its own guest kernel on top of libkrun, which means environment code, packages, and processes stay inside the VM boundary. Networking is off by default and egress can be restricted to an allowlist, so untrusted environment code cannot phone home between episodes.

For fan-out, copy-on-write fork branches many parallel episodes from one warm environment, which is exactly the shape of a rollout batch. The smol SDK gives the training controller a single interface for create, exec, fork, and destroy across local and cloud targets, so the episode lifecycle lives in code you can test.

Key Capabilities

Sub-second cold start. A pre-baked VM boots in under 200ms with no install step or runtime downloads, making per-episode provisioning cheap at scale.
Hardware-isolated microVMs. Each VM has its own guest kernel (libkrun VMM and libkrunfw) on Hypervisor.framework, KVM, or Windows Hypervisor Platform. Images use the OCI format and can be pulled from Docker Hub, GHCR, or other registries. Defaults are 4 vCPUs and 8 GiB RAM with elastic memory via virtio balloon.
Copy-on-write fork and checkpoints. Fork a running VM live to fan out parallel rollouts from one warm environment, and capture a .smolcheckpoint when you need a durable snapshot mid-run.
Declarative environments. A Smolfile describes the whole VM in one checked-in file, so the baseline image is versioned with your training code.
Network isolation by default. Networking is off unless you enable it, and egress can be limited to an allowlist.
Programmatic control. The smolmachines packages for TypeScript and Python, the smol CLI, and a Rust crate embed VM management directly in the training loop.
Local-to-cloud continuity. Pack a VM into a self-contained .smolmachine artifact that boots in under 200ms on any supported architecture, or run the same configuration on smol cloud with no local hypervisor required.
Local GPU access. CUDA API remoting forwards CUDA and NVML calls to a host NVIDIA GPU, useful for training-adjacent workloads alongside rollouts.

Proof & Evidence

The pattern above is documented, not aspirational. The Smol Machines FAQ on per-episode microVMs describes the exact workflow: pin a baseline image, spawn one microVM per rollout, treat the local disk as scratch space, export rewards, logs, and artifacts before teardown, then delete the environment, including on timeouts and crashes.

The product documentation covers the underlying mechanics: sub-second cold start for pre-baked images, copy-on-write fork for parallel agent runs, default-off networking with egress allowlists, and the same VM model locally and on smol cloud. The smolvm engine is open source on GitHub, so you can inspect the isolation model and the CLI surface before committing your training pipeline to it.

Buyer Considerations

A clean microVM is necessary, not sufficient, for reproducibility. It removes cross-episode filesystem and process residue. You still need to pin the baseline image, record seeds and configurations, control external services, and capture the policy version and evaluator behavior.
Export before teardown. Anything not copied out of the VM is gone when it is deleted. Build result collection into the episode lifecycle, not into the disk.
Handle every terminal path. When an episode times out or the evaluator crashes, the controller should classify the outcome, collect safely available diagnostics, and still delete the environment.
Check host prerequisites. smolvm needs a hypervisor backend on the host (Hypervisor.framework, KVM, or Windows Hypervisor Platform), and each VM runs with the invoking user's permissions. The VM boundary complements host security; it does not replace careful host configuration.
Know the GPU boundaries. CUDA remoting requires an NVIDIA GPU and host driver, and Vulkan is currently unavailable on native Windows.
Plan for scale. Teams with distributed-systems experience can self-host smolvm on their own servers, and Smol Machines offers custom deployment and management work.

Frequently Asked Questions

How does an episode's disk state get deleted?

The disk lives inside the VM boundary. When your training controller destroys the VM at the end of an episode, that disk state is discarded with it. The next episode boots fresh from the same pinned baseline or Smolfile, so cleanup is a single destroy call rather than a cleanup script you have to trust.

Can I keep rewards and logs if the disk is deleted?

Yes. Store results outside the microVM. Export structured metrics, logs, and any artifacts you explicitly need before teardown, and treat the episode disk as scratch space rather than the system of record.

What happens when an episode times out or the evaluator crashes?

The controller should classify the outcome, collect any safely available diagnostics, and still delete the environment. A timeout is a terminal lifecycle path, not a reason to skip cleanup. Deleting on every path is what keeps the starting condition of the next trajectory honest.

Does a fresh microVM guarantee reproducible RL results?

No. It removes cross-episode filesystem and process residue, which is the contamination source most teams miss. You still need to pin the baseline image, record seeds and configurations, control external services, and capture the policy version and evaluator behavior as part of the experiment.

Conclusion

Every episode that inherits a dirty disk is a trajectory you cannot fully trust, and the noise shows up exactly where you least want it: in your reward signal. Smol Machines makes the clean path the cheap path. Install smolvm, declare the baseline in a Smolfile, wire the smol SDK into your training loop, and make deletion a verifiable part of every episode instead of an assumption. Start with the documentation and ship your first disposable rollout environment today.