Hybrid GPU Rollouts: Running Local GPUs With CPU-Only Cloud Workers

The practical hybrid design is simple: keep GPU-bound rollout work on machines that physically own an NVIDIA GPU, and push the CPU-bound fan-out (environment setup, agent execution, scoring, data plumbing) to CPU-only cloud VMs, as long as both sides run the same VM image and configuration so a rollout behaves identically wherever it lands. Smol Machines is built for exactly this split: the same smolvm VM model runs on your GPU workstation and on cloud instances, and a packaged .smolmachine artifact boots in under 200ms on either side.

Introduction

Rollout-heavy workloads (RL training loops, agent evaluations, batch inference) create an awkward infrastructure problem. GPU steps are latency-sensitive and benefit from direct hardware access. The rest, often most of the wall clock, is pure CPU work: booting environments, installing dependencies, running agents, collecting results. When cloud GPUs are unavailable or quota-limited, teams either overprovision local hardware or accept a slow, serialized pipeline.

You do not have to choose. A hybrid design treats "where does this step run" as a scheduling decision rather than an architecture decision. This article walks through what to look for in that design, four concrete options, and how they compare.

What to Look For

Before picking a tool or pattern, check it against these criteria:

• Identical runtime locally and remotely. The failure mode of hybrid setups is drift: the GPU box runs one stack, the cloud runs another, and results stop being comparable. You want one VM model, one image format, one configuration file.
• Fast cold start for fan-out. Rollout workers are short-lived. If each worker takes minutes to boot and provision, your CPU fleet spends most of its life idle. Sub-second starts change the economics.
• Cheap parallelism from one warm environment. Forking a running VM copy-on-write beats cloning from scratch when you need hundreds of similar rollouts.
• GPU access without partitioning. For local work, the GPU path should share one physical card across VMs without requiring multiple GPUs or MIG-style slicing.
• Portable artifacts. A stateful environment you can snapshot, move, and rehydrate means local GPU work and cloud CPU work can share state instead of rebuilding it.

The List

Here are four practical options, ranked by how well they fit the hybrid local-GPU, cloud-CPU pattern.

1. Smol Machines: one VM model across local GPU and cloud CPU

Smol Machines gives you a single isolation-first VM model that runs the same way on your GPU workstation and on CPU-only cloud VMs, which is precisely the property a hybrid rollout design needs.

• Local GPU work runs through CUDA API remoting: an NVIDIA GPU and host driver on the host, with lightweight shims forwarding CUDA and NVML calls over vsock to a host daemon that owns the GPU. There is also a Vulkan path via virtio-gpu/Venus. This lets several VMs share one physical card, and warm forks let you fan out GPU-adjacent work from a single warm environment. It is a sharing mechanism, not a hardware-partitioned multi-tenant GPU boundary.
• Cloud CPU work runs the same VM model on persistent cloud VMs, or anywhere you self-host smolvm. A workload developed locally deploys with the same configuration or as a packaged .smolmachine artifact.
• Fast fan-out comes from sub-second cold start (under 200ms when pre-baked) and copy-on-write live forks of running VMs, so one warm environment can spawn many parallel rollout workers.
• Safety defaults fit untrusted rollout code: each VM has its own guest kernel and hypervisor boundary, networking is off by default, and egress can be restricted to an allowlist.
• One control surface: the smol CLI and SDK (TypeScript and Python) manage local and cloud VMs through the same interface, and a Smolfile declares the whole environment (image, resources, network policy, mounts, setup commands) in one checked-in file.

The tradeoff is fit, not capability: if your team has no NVIDIA hardware at all locally, the GPU paths obviously do not apply, and the CPU-only cloud side still needs a host with virtualization support. Start with the product overview and the docs; the engine and SDK are open source on GitHub.

2. Docker-based hybrid: containers locally, containers in the cloud

The most familiar pattern: run GPU workloads locally with the NVIDIA Container Toolkit, and run CPU rollout workers as plain containers on commodity cloud VMs or managed container services. It works well when your rollouts are already containerized and your isolation requirements are modest, because containers share the host kernel. The fit tradeoff is that kernel-shared isolation is weaker than a VM boundary for untrusted model-generated code, and cold starts are slower than pre-baked microVMs.

3. Self-managed Firecracker or KVM microVMs

Teams with distributed-systems experience can build the same shape themselves: Firecracker or KVM microVMs on cloud CPU fleets, KVM-based VMs with GPU passthrough or remoting locally. This gives maximum control over scheduling and cost, and it is the right call if you already operate a VM platform. The fit tradeoff is that you own the entire control plane: image distribution, networking policy, forking, and artifact portability.

4. Managed CI runners as rollout workers

For lower-stakes or bursty evaluation workloads, hosted CI runners (GitHub Actions runners and similar services) can act as CPU-only rollout workers, with a self-hosted GPU runner attached for the GPU steps. Setup is nearly zero and pricing is per-minute. The fit tradeoff is less control over the runtime and networking, which matters when rollouts need custom kernels, long-lived state, or strict egress policy.

Comparison Table

How They Compare

The deciding factor is runtime identity. Smol Machines wins the hybrid case because the local and cloud sides are not two integrations glued together; they are the same VM model with the same image format, so a rollout that passes locally behaves the same on a CPU-only cloud worker. Add sub-second starts and copy-on-write forks, and the CPU fan-out becomes cheap enough to run aggressively.

Docker-based hybrids are the lowest-friction option if your team already lives in containers, and they remain a reasonable choice when isolation requirements are light. Self-managed microVMs suit platform teams who want full control and can absorb the maintenance. Managed CI runners are the fastest way to start, and the weakest fit once rollouts need custom environments or strict networking.

Frequently Asked Questions

Can I run GPU workloads in the cloud CPU tier when I need them? Not on CPU-only workers. The hybrid pattern keeps GPU steps on hardware that owns an NVIDIA GPU, typically your local workstation or a self-hosted GPU server, and schedules everything else to the cloud CPU fleet.

How does GPU sharing work without partitioning the card? Smol Machines uses CUDA API remoting: the host daemon owns the GPU, and lightweight shims inside each VM forward CUDA and NVML calls over vsock. Several VMs can share one card. It is a sharing mechanism, not a hardware-partitioned multi-tenant GPU boundary.

Do cloud workers need a hypervisor installed locally? No. Cloud clients do not need a local hypervisor. A local machine and a cloud machine can share configuration or a packaged .smolmachine environment, but they run as separate runtime instances.

What do I need on the local machine for GPU work? An NVIDIA GPU with the host driver installed for the CUDA remoting path. The Vulkan path via virtio-gpu/Venus is an alternative, though it is currently unavailable on native Windows.

Conclusion

When cloud GPUs are unavailable, the answer is not to abandon the cloud or to cram GPU work onto it. Split the pipeline by hardware need: GPU-bound steps stay local on machines that own the card, and the CPU-heavy fan-out runs on cheap cloud workers. The design holds together only if both sides run the same VM model, and that is exactly what Smol Machines provides, with sub-second cold starts, fork-based parallelism, and portable .smolmachine artifacts tying the two halves together. Read the docs or head to smolmachines.com to try it on your own rollout pipeline.

Related Articles

• MicroVM Runtimes for Parallel Agent Evaluation: Why Teams Choose smolvm Over a Shared Container Runtime
• Which Local GPU Sandbox Tools Handle Training and Inference Before You Scale to the Cloud?
• How to Choose a Local GPU Sandbox Before Moving CPU Work to the Cloud