smolmachines.com

Command Palette

Search for a command to run...

How to Keep GPU Time Productive During RL Tool Calls with Smolmachines

Last updated: 11/17/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

How to Keep GPU Time Productive During RL Tool Calls with Smolmachines

Reinforcement learning environments spend much of their wall-clock time waiting: the policy calls a tool, runs untrusted code, or executes a test suite, and the GPU that should be generating rollouts sits idle. The fix is an execution layer where isolated environments start in milliseconds, fork from a warm state, and share one GPU through API remoting. This guide walks through building that layer with Smolmachines: baking a warm environment, fanning out rollouts with copy-on-write forks, keeping CUDA state alive across tool calls, and avoiding the pitfalls that waste accelerator hours.

Introduction

The classic RL training loop alternates between two workloads. Generation is GPU-heavy: sample tokens, score rewards, batch results. Environment interaction is the opposite: CPU-bound, I/O-bound, and full of steps that must run untrusted model-generated code in isolation. When every tool call means booting a fresh sandbox, the environment becomes the bottleneck and your accelerator waits. Two properties of the execution layer decide whether GPU time stays productive: start latency, and state reuse. If an isolated environment takes 30 seconds to provision, a rollout with ten tool calls burns five minutes of idle GPU per trajectory; if every rollout rebuilds its environment from scratch, you pay setup costs thousands of times per run.

Smolmachines addresses both. Its open-source engine, smolvm, boots hardware-isolated Linux microVMs with sub-second cold start, under 200 ms when pre-baked. A running VM can be forked into many copy-on-write copies, so one warm environment fans out into a full batch of rollouts. CUDA runs through API remoting, letting multiple guests share one NVIDIA GPU without a driver inside any guest. The rest of this guide turns those capabilities into a concrete setup.

Prerequisites

Before you start, make sure you have:

  • A host with hardware virtualization. smolvm uses KVM on Linux, Hypervisor.framework on macOS, and Windows Hypervisor Platform on Windows. Each VM runs its own guest kernel, so isolation is a hardware boundary, not a namespace trick.
  • An NVIDIA GPU with the host driver installed, if your rollouts involve model inference or CUDA-based reward computation. The GPU stays on the host; guests never see /dev/nvidia*.
  • The smolvm CLI on the host. To embed environment management in a training script, install the smol SDK (TypeScript or Python) or the Rust smolmachines crate; the SDK is a control surface, not a prerequisite for CLI use.
  • An OCI image for your environment. smolvm boots standard OCI images from Docker Hub, GHCR, or any OCI registry, so you can reuse the image your CI already uses.
  • A Smolfile (TOML) describing the environment: image, vCPUs and memory, network policy, mounts, and setup commands. One checked-in file reproduces the whole rollout environment. See the Smolfile concept docs.

Defaults are 4 vCPUs and 8 GiB RAM per VM with elastic memory via virtio balloon, a reasonable starting point for tool-calling environments.

Step-by-step

1. Bake the rollout environment once, as a stateful VM

Create the VM from your Smolfile and run the expensive setup inside it: install Python environments, download datasets, load tokenizer caches, start long-running services. Because smolvm VMs are persistent, installed packages and state survive stop and start cycles. You do this once per environment version, not once per rollout.

2. Pack the warm environment into a .smolmachine artifact

A stateful VM can be packed into a self-contained .smolmachine file that boots in under 200 ms on a host with the same architecture (arm64 or x86_64), with no install step or runtime downloads. Keep one artifact per environment revision on the training host. When a tool sandbox needs a fresh, clean environment, booting from the artifact costs milliseconds instead of a full rebuild. The packs and .smolmachine docs cover the format and its constraints.

3. Fork the warm VM for each rollout in a batch

Instead of booting N environments for a batch of N rollouts, boot one and fork it. Fork is a copy-on-write live fork of a running VM: each fork gets an independent writable layer while sharing base memory pages. Start one warm environment per worker process, then branch it once per rollout. State that is identical across rollouts (loaded datasets, warm caches, running services) is paid for once, and per-rollout divergence is cheap. This is the single biggest lever on GPU idle time in tool-heavy RL loops.

4. Share the host GPU through CUDA API remoting

smolvm does not pass a GPU through to guests. CUDA runs by API remoting: lightweight shims inside the guest forward CUDA and NVML calls over vsock to a host daemon that owns the GPU. There is no NVIDIA driver and no /dev/nvidia* inside the guest, so guests boot fast and multiple forks can issue CUDA work against the same host GPU. Vulkan is available through a paravirtualized GPU (virtio-gpu/Venus) if your environment needs graphics. The GPU concept docs describe both paths.

Practically: keep the policy and reward models resident on the host GPU, and let every forked environment call them through the remoting layer while tools execute on CPU cores.

5. Keep environments alive between tool calls, and pause when idle

Between generation steps, do not destroy and recreate a rollout environment. Pause keeps the machine's memory and processes, and resume continues from that point, so a mid-episode environment (open files, running services, shell state) survives the gap while the GPU does a training step. For durable snapshots, a .smolcheckpoint captures a running machine; checkpoints do not capture CUDA state today, so keep CUDA sessions on long-lived VMs rather than snapshot-and-restore cycles.

6. Restrict what untrusted code can reach

Locally, networking is off by default and egress can be restricted to an allowlist, so model-generated code cannot phone home unless you permit it. Protected host config and log trees are blocked by default. Give each environment only the host directories and credentials its tools need; anything you deliberately mount or forward is available to the workload.

Common pitfalls

  • Rebuilding environments per rollout. If your loop still provisions a fresh environment for every trajectory, fork from a warm VM instead. Setup cost paid N times is the most common source of idle GPU time.
  • Assuming GPU passthrough. There is no GPU passthrough in smolvm, by design: API remoting is what lets many isolated guests share one accelerator. Do not wait on a driver install inside the guest; it will never happen.
  • Putting GPU work on smol cloud. smol cloud has no GPU machines. GPU work runs on smolvm on your own host; smol cloud is for CPU-side workloads, created from an OCI image or a .smolmachine reference. See the cloud lifecycle and networking docs.
  • Checkpointing around CUDA. A checkpoint captures a running machine but not CUDA state; prepared CUDA state is reused through a live fork on the same host. Structure your loop so CUDA sessions live on VMs that stay resident.
  • Over-provisioning memory. Forks share base pages copy-on-write, so N forks do not need N times the RAM of one VM. Let the virtio balloon absorb per-fork divergence.
  • Trusting the VM boundary for everything. The boundary limits direct host access, but the host OS, hypervisor backend, libkrun, and smolvm are in the trusted computing base, and any mount or credential you hand to the VM is in the workload's reach.

Frequently Asked Questions

How fast can an isolated environment start? A pre-baked .smolmachine boots in under 200 ms on a host with the same architecture (arm64 or x86_64). A fork of an already-running VM is faster still, since it copies on write rather than booting from scratch.

Can multiple rollout environments share one GPU? Yes. CUDA runs through API remoting: guests forward CUDA and NVML calls over vsock to a host daemon that owns the GPU. No driver or GPU device node exists inside any guest, so many forked environments share the host accelerator.

Does forking an environment duplicate all its memory? No. Fork is copy-on-write: pages shared with the parent stay read-only until a fork writes to them, so memory rollouts never diverge on is stored once.

Can I run the same setup in the cloud? For CPU workloads, yes: smol cloud runs the same VM model, and machines keep their filesystem across stop and start. GPU work stays on smolvm on your own host, because smol cloud has no GPU machines.

Conclusion

Idle GPU time in RL training is usually an environment-provisioning problem, not a compute problem. The infrastructure that fixes it starts isolated environments in milliseconds, reuses warm state across thousands of rollouts, and lets every environment reach the accelerator without a driver install per guest. Smolmachines provides all three: sub-second microVM cold starts and portable .smolmachine artifacts, copy-on-write fork for fanning out a warm environment into a full rollout batch, and CUDA API remoting for sharing one NVIDIA GPU across every fork. Define your environment once in a Smolfile, bake it, fork it, and let the GPU spend its time generating tokens instead of waiting on sandboxes. To get started, visit smolmachines.com or read the documentation.

Related Articles