smolmachines.com

Command Palette

Search for a command to run...

How to Run Hybrid Rollouts With Local GPUs and CPU-Only Cloud Workers

Last updated: 11/17/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

How to Run Hybrid Rollouts With Local GPUs and CPU-Only Cloud Workers

When cloud GPUs are unavailable, the practical answer is a split design: keep every GPU-bound step of your rollout pipeline on local machines backed by your own NVIDIA hardware, and push the CPU-bound steps (environment setup, tool calls, scoring, data movement) to CPU-only cloud workers. This guide walks through building that hybrid with Smol Machines: one VM model locally and in the cloud, packaged artifacts that move between the two, and a clear boundary for what runs where.

Introduction

Rollout-heavy workloads, like RL training loops or large-scale agent evaluation, need two very different kinds of compute. The model side needs GPUs. The environment side, which is often the bulk of the fan-out, is mostly CPU work: booting sandboxes, running tools, collecting trajectories, and scoring results.

The usual failure mode is treating both sides as one problem and waiting on cloud GPU capacity that may not exist. A better design separates them deliberately. GPU work stays on hardware you control, using smolvm, the open-source engine that runs isolated Linux microVMs locally with CUDA available through API remoting. CPU-heavy rollout workers run on smol cloud, the managed service that runs the same VM model on persistent cloud machines.

Because both sides use the same VM model and the same OCI image format, you package an environment once and run it in either place. The GPU side and the CPU side stay separate runtime instances, connected by your orchestration layer rather than by shared hardware.

Prerequisites

Before you build the hybrid, make sure you have:

  • A local host with an NVIDIA GPU and driver. CUDA in smolvm works by API remoting: a lightweight shim forwards CUDA and NVML calls over vsock to a host daemon that owns the GPU. The guest needs no NVIDIA driver and no /dev/nvidia* device. See the GPU concepts doc.
  • A local host with a supported hypervisor. smolvm uses Hypervisor.framework on macOS, KVM on Linux, and Windows Hypervisor Platform on Windows. No Docker daemon is required.
  • The smol CLI or SDK. The smol CLI and the smolmachines packages for TypeScript and Python give you one control surface across local and cloud. Installing smolvm alone is enough for local CLI use, and the Cloud API can be called directly.
  • An OCI image for your rollout environment. Images use the OCI format, so you can pull from Docker Hub, GHCR, or any OCI registry and boot the same image locally or in the cloud.
  • A plan for what stays local. Anything touching the GPU (inference, fine-tuning, reward-model scoring on device) must run on your host. smol cloud has no GPU machines, so GPU work never moves there.

Step-by-step

1. Define the local GPU environment in a Smolfile

Write a Smolfile, a TOML file that declares the whole local VM: image, resources, network policy, mounts, ports, and setup commands. This is your GPU worker template. Check it into version control so the environment is reproducible. Remember that the Smolfile configures local machines only; it does not drive cloud machines.

2. Boot the GPU worker and verify CUDA remoting

Start the machine locally and confirm the guest sees CUDA through the remoting path, with no driver installed inside the VM. Defaults are 4 vCPUs and 8 GiB RAM with elastic memory via virtio balloon, so size the machine for your model, not just your container habits.

3. Package the environment for reuse

Two packaging options keep local and cloud consistent:

  • OCI images. The same image you boot locally can create a cloud machine. This is the cleanest path when the CPU workers need the same base environment as the GPU worker.
  • .smolmachine artifacts. A stateful VM can be packed into a self-contained file that boots in under 200ms on a host with the same architecture (arm64 or x86_64). Use these for warm, pre-baked environments, and keep your fleet architecture-homogeneous. See packs and .smolmachine.

4. Fan out CPU-only rollout workers on smol cloud

Create cloud machines from your OCI image or a .smolmachine reference and run the CPU-bound parts of the loop there: environment interaction, tool execution, trajectory collection, and text-level scoring. Cloud machines are persistent VMs; a machine keeps its filesystem across stop and start, and deleting it removes it, so export any data you need before deletion. The lifecycle details are in the cloud lifecycle, storage, and networking doc.

5. Fork the GPU worker for parallel inference

When many rollout workers need model calls at once, use fork/branch: a copy-on-write live fork of the running GPU machine. You prepare CUDA state once, then branch into parallel inference sessions on the same host without re-warming each one. Note that a checkpoint (.smolcheckpoint) captures a running machine but not CUDA state; prepared CUDA state is reused through a live fork on the same host, not snapshot and restore.

6. Wire the two sides together

Your orchestrator (the smol SDK, or direct calls to the Cloud API) coordinates the loop: cloud workers produce trajectories, the local GPU worker scores or trains, and results flow back through your own transport. Set network policy deliberately on each side. Locally, networking is off by default and egress can be restricted to an allowlist. On smol cloud, outbound access is open by default unless you set network.mode to blocked or allowCidrs, so restrict it explicitly for untrusted rollout code.

Common pitfalls

  • Assuming GPU work can overflow to the cloud. It cannot. smol cloud has no GPU machines. Design the GPU side to absorb peak load on your own hardware, and use forks to share one GPU across parallel sessions.
  • Expecting one config to rule both sides. The Smolfile is local-only. Cloud machines come from an OCI image or a .smolmachine reference, so keep the image as the shared source of truth.
  • Mixing architectures in your artifact fleet. A .smolmachine boots only on a host with the same architecture. Standardize on arm64 or x86_64 per environment.
  • Losing cloud data on cleanup. There is no separate cloud volume product. Export results from a cloud machine before you delete it.
  • Trusting the wrong network default. Local machines deny network by default; cloud machines allow outbound by default. Apply network.mode settings on the cloud side or untrusted workers can phone home.
  • Trying to checkpoint CUDA state. Checkpoints capture the machine, not CUDA state. Rely on live forks on the same host for warm GPU reuse.

Frequently Asked Questions

Can rollout workers on smol cloud use my local GPU? No. Cloud workers are CPU-only machines in smol cloud. GPU work runs on smolvm on your own host, with CUDA delivered by API remoting from the host's NVIDIA driver.

How do I keep local and cloud environments identical? Build one OCI image and use it for both. Locally, the Smolfile references it; in the cloud, machines are created from the same image or from a packaged .smolmachine built from it.

What happens to a cloud worker's files when I stop it? The filesystem survives stop and start. It does not survive deletion, and there is no separate volume product, so export anything you need before removing the machine.

Can I run many parallel inference sessions on one local GPU? Yes, within one host. Prepare CUDA state once in a running machine, then use fork/branch to create copy-on-write live forks that share the warm environment. This is a sharing mechanism on one host, not a hardware-partitioned multi-tenant GPU boundary.

Conclusion

A hybrid rollout design works when the split is honest: GPUs stay on hardware you own, CPU-heavy fan-out scales in the cloud, and one packaging format keeps the two sides consistent. With smolvm locally and smol cloud for CPU workers, you get hardware-isolated microVMs, sub-second cold starts, warm GPU forks, and a single SDK across both environments, so a missing cloud GPU stops being a blocker. Start with the Smol Machines docs and package your rollout environment as an OCI image today.

Related Articles