Running agent evals on a laptop, one VM per trial

Harbor is the evaluation framework from the team behind Terminal-Bench, and the official harness for Terminal-Bench 2.0. Every trial gives an agent a task inside an isolated environment, then runs a verifier that scores what the agent left behind. Harbor doesn't run those environments itself. It hands them to a provider: Docker, one of a dozen cloud sandbox services, Apple's container tool, or smol machines.

The smol provider differs from the rest in one way: it embeds the VM. Harbor imports the smolmachines Python package, and each trial gets its own Linux VM, started from inside Harbor's own process. There is no daemon to install, no service to start and no account. We wanted to know what that is worth in practice, so we ran the same eval through Harbor's smol and Apple container environments on one MacBook. We wrote Harbor's smol environment, so weigh the numbers with that in mind. The commands to reproduce them are at the end.

from smol import Machine, MachineConfig, ConnectOptions

machine = Machine.create(
    MachineConfig(image="python:3.12-slim"),
    ConnectOptions(target="local"),   # a VM on this computer, no daemon
)
result = machine.exec(["python3", "-m", "unittest", "-v"])

The setup

We used an M4 Max MacBook Pro with 36 GB of memory, Harbor 0.24.0, smolmachines 1.29.1 and Apple's container 0.6.0 with its recommended kernel. The task is small on purpose: fix a slugify function in Python, checked by a unittest verifier, in the python:3.12-slim image. Harbor's oracle agent applies the reference solution, so no model is involved and the timings measure the environment, not an LLM. Every run below had the image cached already, and every trial passed.

The task allows internet access, because Harbor's Apple container environment rejects tasks that turn the network off. smol runs the same task with the network off as well, at the same speed: 0.85 seconds of median setup against 0.84.

One trial: 1.3 seconds against 21

We ran 16 trials, 8 at a time, on each environment.

The median trial took 1.33 seconds on smol and 21.4 seconds on Apple container. Getting a running environment took 0.84 seconds against 10.6. The oracle's step, which uploads a script and runs it, took 0.05 seconds against 1.35, and the verifier 0.26 against 1.98. All 16 trials finished in 5.3 seconds on smol and 46.9 seconds on Apple container.

Where the time goes

Setup is mostly the time to get a running Linux with the task's image in it. smol prepares an image once, and later machines of that image start from the prepared copy in under a second.

The per-command gap comes from the path each command takes. Harbor's smol environment calls the SDK in its own process, and the SDK talks to the VM's helper directly. Harbor's Apple container environment drives Apple's tool through its command line, so every command starts a new container process, which asks Apple's container API server, which reaches the VM.

The oracle runs one step per trial, so here that cost is small. A coding agent runs dozens to hundreds of commands per trial, and each one pays it. At the per-step times we measured, 200 agent commands add about 10 seconds on smol and about 4.5 minutes on Apple container.

To be fair to Apple's tool, this compares Harbor's two integrations as they ship, not the raw boot speed of two VMs. Apple's container is built to be driven from a terminal, and a command line is the only way Harbor can drive it. An SDK that lives in the caller's process is the difference we set out to measure.

How many trials fit on one laptop

Next we ran 64 trials on smol, changing only how many run at once.

One at a time, the 64 trials took 47.5 seconds. With 32 at once they took 8.3. Each trial gets slower as more share the machine, from a median of 0.68 seconds alone to 2.21 seconds with 32 at once, so the total levels off past 16.

Memory stayed small. The task asks for 1 GiB per machine, but each VM held between 75 and 117 MiB of host memory while it ran, because a guest's memory is only backed as the guest touches it. At 32 trials at once, 33 VMs held 2.5 GiB in total. The extra VM is Harbor's warm base machine, which the next section explains.

Branching from a warm machine

Harbor's smol environment prepares one machine per task and branches each trial from it, instead of starting every trial from the image. On this task that made no difference: a branch was ready in a median of 0.93 seconds and a fresh machine in 0.85, because python:3.12-slim starts fast anyway. Branching is meant for tasks with expensive setup, such as installing a coding agent and a repository's dependencies, where that work then happens once instead of once per trial. This benchmark doesn't measure that case.

Try it

uv tool install 'harbor[smol]'

# 16 trials of one task, 8 at a time, each in its own VM
harbor run -p ./my-task -a oracle -e smol -k 16 -n 8

Local VMs need an Apple Silicon Mac or Linux with KVM. The Harbor docs list the other environments, and the smol SDK docs cover the Python, Node, Rust and Go packages Harbor builds on.