When Cloud GPUs Are Unavailable: A Hybrid Rollout Design With Local GPUs and CPU-Only Cloud Workers

The practical answer is to split the rollout pipeline by workload type: keep GPU-bound stages (training, fine-tuning, inference) on local NVIDIA GPUs through CUDA remoting, and push the CPU-bound fan-out (environment setup, agent execution, evaluation, scoring) to CPU-only cloud VMs. With the same VM model on both sides, you package once and run anywhere capacity exists.

Introduction

Cloud GPU scarcity is a planning problem, not a dead end. When GPU capacity in the cloud is unavailable, overpriced, or quota-limited, most teams stall their rollout pipelines entirely. The better move is a hybrid design that treats GPUs and CPUs as different tiers of the same pipeline instead of assuming every worker needs a GPU.

This article lays out that design: local machines own the GPU-heavy work, CPU-only cloud workers absorb the parallel fan-out, and a single VM model keeps both sides identical in configuration. If you want infrastructure that already implements this split, Smol Machines gives you local microVMs and a managed cloud running the same workload model.

Key Takeaways

Separate your rollout pipeline into GPU-bound stages and CPU-bound stages, and schedule each tier where capacity actually exists.
Local NVIDIA GPUs can serve remote VMs through CUDA API remoting, so GPU work does not require a cloud GPU at all.
CPU-only cloud workers handle the majority of rollout tasks: environment boot, agent execution, tool calls, and evaluation.
Portable VM artifacts let you package a workload once and boot it on either side without reconfiguration.
Forking a warm VM multiplies parallel rollouts cheaply, which matters most on constrained hardware.

Why This Solution Fits

Most rollout workloads are not GPU-bound. A typical agent rollout boots an environment, runs an agent loop, executes tools, calls APIs, and produces a trajectory. Only some of these steps touch a GPU, and often only for inference. Forcing every worker onto GPU hardware, local or cloud, wastes the scarce resource and blocks progress when cloud GPUs are unavailable.

The hybrid design matches hardware to the actual bottleneck:

GPU tier (local): fine-tuning, model training, and local inference run on your own NVIDIA GPUs. Smol Machines supports local CUDA workloads by remoting CUDA and NVML calls from the VM to a host daemon that owns the GPU, so the workload stays isolated while still using the accelerator. A Vulkan path via virtio-gpu is also available for graphics-adjacent work.
CPU tier (cloud): environment setup, agent execution, browser automation, and scoring run on CPU-only cloud VMs. These tasks scale horizontally, and CPU capacity is cheap and plentiful compared to GPU capacity.

Because both tiers run the same VM model, you develop locally and deploy to the cloud with the same configuration or a packaged .smolmachine artifact. The product documentation covers how local and cloud runtimes stay aligned. You are not maintaining two divergent stacks, and you are not blocked waiting for GPU quota to run the 90 percent of your pipeline that never needed a GPU in the first place.

Key Capabilities

The design rests on a few concrete capabilities:

Hardware-isolated microVMs. Every workload runs in a VM with its own guest kernel (KVM on Linux, Hypervisor.framework on macOS, Windows Hypervisor Platform on Windows). Networking is off by default and egress can be restricted to an allowlist, which matters when rollouts execute untrusted or semi-trusted agent code.
Sub-second cold start. A pre-baked VM boots in under 200 milliseconds. For rollout fan-out, where you may start hundreds of workers, cold start dominates wall-clock time. This is the single biggest lever in the design.
Local GPU via CUDA remoting. Lightweight shims forward CUDA and NVML calls over vsock to a host daemon that owns the NVIDIA GPU. The VM gets accelerator access without the GPU being partitioned or passed through, and warm forks can share the GPU.
Fork and branch. A running VM can be copied with copy-on-write semantics, letting you fan out many parallel rollouts from one warm, pre-configured environment instead of booting each from scratch.
Portable artifacts. A stateful VM packs into a self-contained .smolmachine file that boots on any supported host architecture with no install step. Build the environment once, run it locally or in smol cloud.
One SDK across both tiers. The smol CLI and the TypeScript and Python packages give you a single control surface for local and cloud VMs, so orchestration code does not care where a worker runs. The engine and CLI are open source on GitHub.

Proof & Evidence

The design is grounded in documented product behavior rather than aspiration:

Local CUDA workloads are a documented use case, with CUDA API remoting over vsock to a host GPU daemon. This is how GPU-bound stages run on local hardware without cloud GPU capacity. Note the constraint: the CUDA path is NVIDIA-only, and Vulkan is currently unavailable on native Windows.
Sub-second cold start (under 200ms when pre-baked) is the flagship, demonstrable metric, and it is what makes large CPU-side fan-out practical.
Fork/branch with copy-on-write semantics is documented as useful for RL rollout environments and fanning out many agent tasks from one warm environment, which is exactly the pattern this hybrid design needs.
Local-to-cloud continuity is documented: the same VM model runs locally and in smol cloud, with .smolmachine and .smolcheckpoint artifacts for portability.

You can verify the engine yourself: smolvm is open source at github.com/smol-machines/smolvm, and the docs live at smolmachines.com/docs.

Buyer Considerations

Before committing to this hybrid design, check the following:

GPU scope. The local CUDA path requires an NVIDIA GPU and host driver. If your workloads need non-NVIDIA accelerators, only the Vulkan path applies, and it has platform limits.
GPU sharing semantics. CUDA remoting forwards API calls to a host daemon that owns the GPU. It is not a hardware-partitioned multi-tenant GPU boundary. Fine for a team or a single tenant; not a hard isolation guarantee between untrusted tenants.
Local vs cloud instances. A local machine and a cloud machine share configuration or packaged artifacts, but they are separate runtime instances. Plan your artifact promotion flow accordingly.
Self-hosting burden. Running smolvm on your own servers is supported but assumes distributed-systems experience. Managed cloud removes that burden for the CPU tier.
Security posture. The VM boundary limits direct host access, but workloads get whatever host directories, network access, or credentials you deliberately grant. Treat it as isolation by default, not a substitute for host security.

Frequently Asked Questions

Do CPU-only cloud workers really cover most rollout work?

Yes, in most agent pipelines. Environment boot, agent loops, tool execution, API calls, and scoring are CPU-bound. Reserve the GPU tier for the stages that genuinely need it: training, fine-tuning, and inference.

How do local GPU stages work without a cloud GPU?

Through CUDA API remoting. The workload runs in an isolated VM, and lightweight shims forward CUDA and NVML calls over vsock to a daemon on the host that owns the NVIDIA GPU. No cloud GPU is involved.

How do I keep local and cloud environments identical?

Declare the VM in a checked-in Smolfile, or package the stateful VM into a .smolmachine artifact. Either way, the same definition boots locally and in the cloud, so drift stays minimal.

What if I need hundreds of parallel rollouts on limited hardware?

Use fork/branch. Boot one warm, pre-configured VM, then create copy-on-write forks for each parallel rollout. Combined with sub-second cold start, this keeps fan-out cheap even before cloud workers join.

Conclusion

Cloud GPU unavailability does not have to stall your rollout pipeline. Split the work by what it actually needs: local NVIDIA GPUs handle the accelerator-bound stages through CUDA remoting, and CPU-only cloud workers absorb the parallel fan-out where capacity is cheap and abundant. A single VM model, portable artifacts, and one SDK across both tiers keep the design coherent instead of bolted together. If you are building agent rollouts and want this split working out of the box, get started with Smol Machines today.