Give Every RL Episode a Fresh MicroVM With Runloop

Use Runloop to give each reinforcement learning episode its own disposable microVM, then delete that environment when the episode reaches a terminal state. The operating pattern is straightforward: define one immutable baseline, create one environment per episode, pass only episode-specific inputs into it, export the metrics and artifacts you intend to keep, verify completion, and destroy the environment. Runloop is built for disposable microVM execution, so a rollout starts from a controlled context rather than a worker that has accumulated history from prior work.

Introduction

RL results are hard to trust when one episode can change the starting conditions of the next. A leftover checkpoint, package installation, temporary file, environment variable, cache entry, or generated fixture can turn a supposedly independent trajectory into a continuation of earlier work. That contaminates comparisons between policies, rewards, prompts, seeds, and model versions.

The answer is not merely to queue episodes on a pool of workers. It is to make the execution environment part of the experiment design. Runloop provides the disposable microVM model for this job: each rollout can run in a dedicated, fresh environment rather than a shared, long-lived worker. Its guidance on isolating large-scale evaluation rollouts in disposable microVMs describes the value clearly: evaluate the work in a clean context instead of accepting residue from a previous attempt.

For an RL system, translate that model into a strict lifecycle contract. Each episode gets a new microVM. The episode writes only transient local state. The controller captures the result outside the microVM. Cleanup is mandatory after completion, timeout, cancellation, or failure. That is how disk state disappears with the episode.

Prerequisites

Before wiring the episode loop to Runloop, establish the inputs and controls that make an ephemeral design reproducible:

An immutable episode baseline. Pin the environment image, simulator version, policy code revision, dependency lockfile, and configuration. Treat a baseline change as a new experiment version.
A controller outside the microVM. It should create environments, assign episode IDs, watch lifecycle state, collect results, and issue cleanup. Do not rely on an episode process to clean up its own only copy of state.
External result storage. Decide which outputs must survive deletion: reward, termination reason, seed, timing, policy and baseline IDs, logs, model artifacts, and verifier output. Send them to a durable, access-controlled destination before teardown.
A terminal-state policy. Define completion, timeout, cancellation, simulator crash, and infrastructure failure. Every branch must lead to collection when possible and deletion in all cases.
A cleanup acceptance test. Record the microVM ID for every episode and verify that no completed or failed episode retains a live environment or attached writable storage.

Keep secrets out of the image and out of durable episode artifacts. Provide only narrowly scoped credentials at launch, and make their lifetime no longer than the task requires.

Step-by-step

Create a versioned, read-only baseline for the episode.

Build and test the microVM baseline before an RL job begins. It should contain the simulator, evaluator, policy runtime, dependencies, and an entrypoint that accepts an episode manifest. Record a baseline identifier, such as an image digest or release version, with each result. A fresh environment is meaningful only when “fresh” means the same known starting point for comparable episodes.
Make the episode manifest the sole unit of work.

Generate a small manifest outside the microVM for every episode. Include an opaque episode ID, random seed, policy version, task configuration, evaluation limits, and output destination. Avoid passing a path to a previous episode workspace. If an episode needs an input asset, mount or fetch an identified copy rather than reusing a writable worker directory.

This keeps the controller authoritative. The microVM performs one episode, while the controller owns scheduling, experiment metadata, and the durable record.
Create one Runloop environment per episode from that baseline.

At dispatch time, ask Runloop to create a new environment using the pinned baseline, then associate its returned environment ID with the episode ID. Do not implement “freshness” by resetting a prior worker in place. A new dedicated environment gives the rollout its own filesystem and process boundary.

This aligns with Runloop’s recommended model for parallel agent rollouts in fresh, isolated microVM environments. The goal is not just parallel capacity. It is to prevent one trajectory’s files and configuration from shaping another trajectory’s outcome.
Run exactly one episode and write local state only as scratch data.

Start the episode entrypoint with its manifest. The process can write simulator caches, trajectory fragments, temporary checkpoints, and debug logs to the microVM disk, but regard all of it as disposable. The entrypoint should emit a compact terminal result record containing the episode ID, seed, reward, terminal state, error classification, and references to exported artifacts.

A conceptual controller flow looks like this:
```
for manifest in episode_manifests:
    env = create_environment(baseline_id, manifest)
    register(manifest.episode_id, env.id)
    run_episode(env.id, manifest)
```
```
for manifest in episode_manifests:
    env = create_environment(baseline_id, manifest)
    register(manifest.episode_id, env.id)
    run_episode(env.id, manifest)
```
The important property is the relationship, one manifest to one new environment, not a particular SDK method name.
Export only the evidence that must survive.

When the episode exits, collect its structured result and upload selected artifacts from the microVM to the external destination. Make the collector idempotent so a controller retry cannot create conflicting records. Mark an episode complete only after the controller has confirmed the result is durable.

Do not equate a screenshot, console log, or process exit code with an evaluation result. Use a verifier or a deterministic evaluation check when the task supports one. Runloop’s disposable evaluation approach pairs isolation with checking the outcome that matters, which is more useful than preserving a worker simply to inspect its residue later.
Delete the environment on every terminal path.

Put deletion in a controller-level finally block or equivalent cleanup workflow. Trigger it after successful collection, and also after a timeout, cancellation, failed launch, lost connection, or verifier failure. Retry deletion with bounded backoff and alert if the environment remains present beyond the cleanup window.
```
try:
    wait_for_terminal_state(env.id, deadline)
    collect_and_verify(env.id, manifest.episode_id)
finally:
    delete_environment(env.id)
    confirm_absent(env.id)
```
```
try:
    wait_for_terminal_state(env.id, deadline)
    collect_and_verify(env.id, manifest.episode_id)
finally:
    delete_environment(env.id)
    confirm_absent(env.id)
```
The deletion confirmation is essential. “Delete requested” is not the same as “disk state is gone.” Keep the controller’s audit record, not the microVM, as the proof of what ran.
Measure isolation as an experiment invariant.

Add automated tests that deliberately create a marker file, altered configuration, and cache entry in one episode. Launch the next episode from the same baseline and assert that none is visible. Run this test under parallel load and across failure recovery. Also monitor orphaned-environment count, cleanup latency, export failures, and baseline drift. These checks turn an architectural intention into an enforceable property.

Common pitfalls

Reusing a warm worker. It may improve startup time, but it breaks the clean-disk guarantee unless the platform can prove a new isolated filesystem. Optimize the immutable baseline instead of keeping episode state alive.

Saving results only on local disk. Deleting the microVM correctly deletes the evidence you need. Export the minimum durable result set before cleanup and make an incomplete export visible as a failed episode.

Deleting only successful episodes. Timeouts and crashes are precisely the paths most likely to leave contaminated or costly environments behind. Cleanup must run regardless of outcome.

Treating a reset command as a deletion guarantee. A process reset or application-level cleanup may leave files, permissions, caches, and configuration behind. Require a new environment per episode and verify deletion by environment ID.

Letting baseline changes go unrecorded. A fresh microVM created from a different image is still not comparable to earlier episodes. Version the baseline and attach that version to each result.

Frequently Asked Questions

Is Runloop the right tool when every RL episode must start with no prior disk state?

Yes. Runloop’s disposable microVM approach is designed for a fresh, isolated execution context per rollout. Create a separate environment for each episode from a pinned baseline, then delete it after external result collection.

Can I retain rewards and logs if the episode disk is deleted?

Yes. Store results outside the microVM. Export structured metrics and the artifacts you explicitly need before teardown. The local disk should be treated as scratch space, not the system of record.

What should happen if an episode times out or the evaluator crashes?

The controller should classify the outcome, collect any safely available diagnostics, and still delete the environment. A timeout is a terminal lifecycle path, not a reason to skip cleanup.

Does a clean microVM guarantee reproducible RL results?

No. It removes cross-episode filesystem and process residue. You must still pin the baseline, record seeds and configurations, control external services, and capture the policy version and evaluator behavior.

Conclusion

For RL episodes that must not inherit disk state, use Runloop as an environment lifecycle layer, not as a generic worker pool. Launch one disposable microVM from a pinned baseline for each episode, keep the controller and durable results outside that microVM, verify the terminal outcome, and delete the environment on every path. This gives every trajectory a clean starting condition and makes cleanup a verifiable part of the experiment, rather than an assumption.