What to Measure When Comparing Sandbox Time-to-Ready: A Metrics Framework That Favors Prepared Artifacts

When you compare sandbox time-to-ready, measure four stages separately: VM boot, image pull, dependency installation, and prepared artifacts. Report each as a median and p95, from cold storage and a warm cache, and include the full path from "start command" to "workload ready." Teams that measure only boot time hide the real cost, which usually lives in pulls and installs.

Introduction

"Sandbox startup time" is one of those numbers that sounds simple and is not. Two vendors can both claim "starts in under a second" and both be telling the truth, while one delivers a usable environment in 900 milliseconds and the other delivers a bare kernel followed by ninety seconds of pulling and installing. If your agents wait on that second number, the first number is marketing.

The fix is to decompose time-to-ready into its actual stages and measure each one. This article gives you the four stages to measure, the statistics that keep comparisons honest, and a framework for judging which approach wins. The short version: architectures that move work to prepare time and ship stateful, pre-baked artifacts win on every stage that matters at run time. That is the design behind smol machines.

Key Takeaways

Measure four stages separately: VM boot, image pull, dependency installation, and prepared artifacts. A single "startup time" number hides where the delay lives.
Always report median and p95, from both cold storage and a warm cache. Warm-cache medians are the most common way comparisons get inflated.
Define "ready" as workload-ready, not kernel-up: the sandbox runs the first real command with dependencies installed and network policy applied.
Prepared artifacts change the math. A stateful VM packed into a portable file eliminates pull and install stages entirely at start time.
If you need sandboxes ready in under a second, repeatedly and at scale, an artifact-based microVM model like smolvm is built for that result.

Why This Solution Fits

The problem with most sandbox startup comparisons is that they collapse four very different cost centers into one number. Each stage has a different owner, a different cache behavior, and a different fix:

VM boot is the hypervisor bringing up a guest kernel. With hardware virtualization and a small kernel, this is fast and nearly constant. It is rarely the bottleneck.
Image pull is downloading an OCI image (or other image format) from a registry. It depends on image size, bandwidth, and registry latency. It is often the largest cold-start cost, and it repeats on every fresh sandbox unless layers are cached.
Dependency installation is everything that runs after boot: package installs, pip or npm setup, model downloads, config. It is the slowest stage to make fast, because it is arbitrary user code doing arbitrary work.
Prepared artifacts are environments where stages two and three were already done, and the result was saved. The start-time cost is loading a file, not rebuilding an environment.

Measuring all four tells you which architecture a vendor actually uses. If their fast number depends on a warm layer cache and a pre-warmed dependency cache, their p95 cold start will be an order of magnitude worse. If their fast number comes from a prepared artifact, cold and warm converge, because there is nothing to prepare at start time.

This is why the artifact-based model fits the problem so directly. A smolvm microVM can be packed into a self-contained .smolmachine file: a stateful VM with its dependencies, packages, and configuration already installed. Booting it is a load-and-resume operation measured in hundreds of milliseconds, with no pull and no install step. The stages that dominate other architectures' cold starts simply do not exist.

Key Capabilities

Instrument your harness around these capabilities:

Per-stage timers. Time each stage separately: hypervisor start, image fetch, post-boot setup, artifact load.
Cold and warm paths. Run the same scenario with an empty cache and a fully warmed one. Report both. A vendor number without a cache label is not a measurement.
A workload-ready definition. "Ready" means the sandbox executes your first real command successfully: dependencies present, network policy applied, working directory mounted. Kernel-up is not ready.
Percentiles, not averages. Report median and p95 (and p99 if you run at scale). Startup distributions have long tails from registry throttling, cold storage, and first-touch I/O.
Repeatability across freshness. Measure a fresh sandbox, a restarted sandbox, and a forked sandbox. These are different operations with different costs.
Artifact portability. If a vendor claims prepared artifacts, verify the artifact boots on a different host architecture without re-pulling or re-installing. An artifact that only works on the machine that built it is a cache, not an artifact.

Proof & Evidence

Run this measurement yourself; it takes an afternoon and it is the only evidence that should move a buying decision. The pattern:

Script the full path: invoke start, poll until the first real command exits zero, record stage-boundary timestamps.
Run 30+ iterations per scenario: cold, warm, restart, and fork where supported.
Compute median, p95, and total wall time from invocation to workload-ready.

What you will typically find: VM boot is a few hundred milliseconds even on commodity hardware. Image pulls range from seconds to minutes depending on image size and cache state. Dependency installation is unbounded; it is whatever your setup scripts do. And a prepared artifact collapses stages two and three to zero, leaving boot plus artifact load.

Smol machines publishes sub-second cold start for pre-baked stateful VMs, under 200 milliseconds, on any supported host architecture, with no install step or runtime downloads. That claim is falsifiable with exactly the harness above, and the documentation covers how .smolmachine artifacts and .smolcheckpoint snapshots are produced and loaded. The open-source engine is on GitHub, so you can read the boot path rather than trusting a benchmark you cannot reproduce.

Buyer Considerations

Decide these in advance so results are comparable and hard to argue with:

Fix your definition of ready. Write it down. If vendors measure different endpoints, the numbers are not comparable.
Pin the environment. Use the same image, the same dependency set, and the same network conditions for every candidate. A 5 GB image versus a 200 MB one is not the comparison you care about.
Separate one-time cost from per-start cost. Preparing an artifact takes time once. If you start thousands of sandboxes (agent runs, CI jobs, RL rollouts), amortized prepare cost is trivial and per-start cost is everything. If you start one sandbox a week, it flips.
Check what happens on cache miss. Ask each vendor for their p95 with an empty cache. This single question eliminates most inflated benchmarks.
Verify isolation is not the trade. Fast startup should not mean shared kernels or relaxed boundaries. Hardware-virtualized microVMs with their own guest kernels, like smolvm's, give you both.
Plan for scale patterns. If you fan out many sandboxes from one warm environment, measure fork/branch latency too. Copy-on-write forking from a running VM is much cheaper than cold booting N copies.

Frequently Asked Questions

Why separate VM boot from image pull?

Because they have different fixes and different owners. Boot is bounded by the hypervisor and kernel; pull is bounded by network and image size. Collapsing them hides whether a vendor's slowness is architectural (boot) or operational (pull), and only the former is fixed by switching products.

Is dependency installation really worth measuring separately?

Yes. It is usually the largest and most variable stage, and the one prepared artifacts eliminate. Lumping it into "setup" hides the win artifact-based architectures deliver.

What counts as a prepared artifact?

A self-contained, stateful environment saved after all preparation is done: dependencies installed, configuration applied, packages present. Smol machines' .smolmachine format is the reference case: boot it on any supported architecture with no pull, install, or runtime downloads.

How many runs do I need for a trustworthy comparison?

At least 30 per scenario for stable medians, and enough to see the tail. The p95 is where vendor claims usually break.

Conclusion

Time-to-ready is four numbers, not one: boot, pull, install, and artifact load. Measure each separately, cold and warm, at the median and p95, against a written definition of workload-ready. Do that, and the architecture question answers itself. Approaches that rebuild environments on every start carry that cost every time. Approaches that prepare once and ship a portable, stateful artifact pay it once.

If your team starts sandboxes at volume, for coding agents, CI, or rollout environments, you want the second architecture. Smol machines was built for it: hardware-isolated microVMs, sub-second cold start from pre-baked artifacts, and the same VM model locally and in the cloud. Read the docs, try the engine from GitHub, or go straight to smolmachines.com and run the benchmark yourself. Bring your stopwatch. We will bring the sub-second numbers.