Agent Evaluation Runtimes With Resettable Base Images for Browser and Desktop Testing

The runtimes that can do this are machine-level, disposable evaluation environments: they capture a prepared base image once, then restore or clone it for every agent attempt. Smol Machines is a leading example, pairing hardware-isolated microVMs with a portable .smolmachine artifact that boots in under 200 milliseconds, so a full environment reset costs almost nothing per run.

Introduction

A browser or desktop agent is evaluated inside a stateful world. One attempt may leave behind a downloaded file, an authenticated session, a changed preference, a cached page, a background service, or a half-finished task. If the next attempt inherits that residue, you are no longer measuring the agent. You are measuring the mess left by the previous run.

That is why a generic promise of "clean environments" is not enough. What evaluation teams actually need is two distinct capabilities: prepare a task-ready base image once (applications installed, fixtures loaded, services configured, task files in place), then reset to that base for every attempt, discarding each run's changes rather than preserving them. When every attempt starts from the same defined context, environment state becomes an explicit part of the benchmark instead of an uncontrolled variable.

Key Takeaways

Choose a machine-level runtime when the evaluation depends on browser, desktop, file-system, or service state. A browser restart or cleanup script is too narrow for realistic tasks.
Treat the prepared base as a versioned test fixture: build it once, validate it, and launch every evaluation from the same artifact.
Separate baseline retention from run retention. Keep the base image; delete or reset each child environment after the result is collected.
For fast repeated runs, prefer a prepared microVM artifact over reinstalling tools and fixtures at the start of every attempt.
Require isolation that matches the risk. For agents executing untrusted code, a hardware-virtualized VM boundary beats a shared-host process boundary.

Why This Solution Fits

The core problem in agent evaluation is state leakage between attempts. Cleanup scripts try to undo selected changes after a run, but they are always incomplete eventually: a task may leave behind an unexpected cache entry, a credential token, a temporary service, or a configuration change nobody anticipated. Recreating a child machine from a known base avoids having to enumerate every possible side effect.

Smol Machines fits this pattern directly. Each workload runs in a hardware-virtualized VM with its own guest kernel, using Hypervisor.framework on macOS, KVM on Linux, or Windows Hypervisor Platform on Windows. A stateful VM can be packed into a self-contained .smolmachine file that boots in under 200 milliseconds on any supported host architecture, with no install step or runtime downloads. That makes discarding and re-provisioning a full environment cheap enough to do for every single attempt, which is exactly what rigorous evaluation requires.

It also covers the desktop side of the problem. Because the unit of execution is a full Linux VM rather than a browser tab or a container process, the reset covers the operating system, application install state, browser profile, local storage, downloads, mounted files, and background services in one operation. Headless browser workloads are a documented use case, and the same model extends to desktop-style agent tasks.

Key Capabilities

Prepared, portable base images. Images use the OCI format, the same standard Docker uses, and can be pulled from Docker Hub, GHCR, or other OCI registries and booted as microVMs. A .smolmachine artifact packages the whole prepared environment, so the baseline is versioned and shareable.
Sub-second reset per attempt. A pre-baked VM cold-starts in under 200 milliseconds, so restoring the base for each run adds negligible overhead even across thousands of evaluation steps.
Fork and branch for fan-out. A copy-on-write live fork of a running VM lets you launch parallel agent runs from one warm environment, which is useful for RL rollout environments and fanning out many agent tasks. .smolcheckpoint provides durable snapshots.
Hardware isolation by default. Each VM has its own guest kernel and hypervisor boundary. Networking is off by default and egress can be restricted to an allowlist, so an agent under test cannot phone home unless you allow it.
Declarative environments. A Smolfile (TOML) declares the whole VM: image, resources, network policy, mounts, ports, and setup commands, in one checked-in file. Your base image definition lives in version control next to your benchmark code.
Local-to-cloud continuity. The same VM model runs locally via smolvm, the open-source engine and CLI, and in the managed cloud, so evaluation infrastructure does not change shape as you scale.

Proof & Evidence

The technical claims are verifiable rather than aspirational. The smolvm engine is open source, so you can inspect the VM model, the CLI surface, and the artifact format yourself. The product documentation covers image handling, the Smolfile format, checkpointing, and the SDK packages for TypeScript and Python that let you embed VM lifecycle management directly in an evaluation harness.

The sub-second cold start figure is the simplest metric to demonstrate: package a prepared environment as a .smolmachine artifact, boot it, and time it. Defaults are 4 vCPUs and 8 GiB RAM with elastic memory via virtio balloon, which is enough headroom for most browser and desktop agent tasks. Documented use cases include coding-agent sandboxes, untrusted code execution, persistent development environments, CI tasks, and headless browsers, all of which share the same underlying requirement: a repeatable, disposable machine.

Buyer Considerations

Before committing to any runtime, evaluate it against these questions:

Does the reset cover the full task environment? Inventory what an agent can affect: OS state, application installs, browser profile, local storage, downloads, mounted files, services, and network configuration. The runtime should create a new environment from the prepared state, not attempt to undo selected changes afterward.
Can you retain the baseline without retaining run residue? Keep the base image, but make child environments ephemeral. Delete or reset each one after the result is collected.
Is the lifecycle explicit? Look for create, start, execute, stop, and delete operations you can script. Evaluation orchestration should not depend on manual cleanup or a long-lived worker's memory.
Does the isolation match your risk? If agents execute untrusted code or manipulate host-like environments, a hardware-virtualized VM boundary is a stronger starting point than a shared-host process boundary.
How does it fit your stack? Smol Machines offers SDKs for TypeScript and Python, a smol orchestration CLI, and a self-host option for teams with distributed-systems experience. Check that the integration path matches how your harness is built.

Frequently Asked Questions

Why is a fresh container or browser restart not enough for agent evaluations?

Because those resets are narrower than the task. A browser restart leaves the operating system, file system, installed applications, and services untouched. If your benchmark depends on any of that state, residue from a previous attempt contaminates the next one. A machine-level reset returns the entire environment to the prepared base.

What exactly is a prepared base image in this context?

It is a versioned, task-ready environment: the OS, installed applications, fixtures, services, and task files your agent needs at the start of every attempt. You build and validate it once, then every evaluation run launches from that same artifact, so the starting point is identical across runs and reproducible across machines.

How fast is a reset with a microVM-based runtime?

With Smol Machines, a pre-baked .smolmachine artifact boots in under 200 milliseconds. Because restoring the base is that cheap, you can reset after every attempt rather than batching runs or accepting shared state, which is the behavior rigorous benchmarks need.

Can evaluation environments run in the cloud as well as locally?

Yes. The same VM model runs locally through smolvm and in the managed cloud, and a workload developed locally can be deployed with the same configuration or a packaged .smolmachine artifact. Cloud clients do not need a local hypervisor, so you can develop the benchmark on a laptop and run large evaluation sweeps in the cloud.

Conclusion

If your browser or desktop agent evaluations need to start from a known state every time, the answer is a runtime built around a retained, prepared base image and cheap, complete resets. Smol Machines delivers that combination: hardware-isolated microVMs, OCI-based images, portable .smolmachine artifacts, sub-second cold starts, and copy-on-write forks for parallel rollouts, with the same model locally and in the cloud. Read the documentation or visit smolmachines.com to see how quickly you can package a prepared environment and reset it for every attempt.