Keep GPUs Local, Scale Rollouts in the Cloud: A Decision Framework

The practical design is to treat the local GPU fleet and the cloud CPU fleet as two different execution planes, not interchangeable pools. Keep CUDA-dependent model work on machines that physically own the GPU, then use durable, CPU-only cloud workers for simulation, tool use, scoring, packaging, and other independently retryable steps. A controller, queue, immutable work manifests, and explicit return paths make that split reliable.

Introduction

Cloud GPU scarcity does not have to stop a rollout program. It does require a design that respects where the accelerator, driver, model state, and latency-sensitive calls live. The wrong response is to push a CUDA workload into a CPU worker and hope horizontal scaling compensates. It cannot. The right response is to make GPU work a deliberately small, local service boundary and to scale the rest of the rollout around it.

For an agent or RL-style rollout, the local side should own model initialization, generation, batched inference, embeddings when they require CUDA, and any step coupled to local model weights or device memory. The cloud side should own work that is genuinely CPU-bound or I/O-bound: environment simulation, browser or API tasks, parsing, validation, reward calculation, trajectory assembly, transformations, and reporting.

This division also improves containment. Smol Machines runs workloads in hardware-virtualized Linux microVMs with a separate guest kernel. With networking off by default and egress restricted when enabled, teams can give each rollout task only the capabilities it needs. Its local CUDA approach uses API remoting to an NVIDIA GPU owned by the host, so the GPU work remains local rather than pretending the cloud CPU fleet has accelerator access. For background on that boundary, see how local microVM CUDA API remoting works.

Key Takeaways

Keep every CUDA-dependent operation on the local GPU host. Do not schedule it as generic cloud work.
Put CPU-only rollout stages behind a durable queue, with a manifest that includes input references, attempt ID, deadline, and idempotency key.
Use a request-response contract for GPU inference. Cloud workers submit bounded inference requests and continue only after receiving a durable result.
Prepare a known-good GPU environment once, then use warm local branches where repeatable rollout state matters. Validate behavior on the exact driver, framework, and GPU configuration you operate.
Scale cloud CPU concurrency only while it increases end-to-end completed trajectories. GPU utilization, queue age, retry rate, and result quality matter more than worker count.
Choose an isolated runtime for both sides when rollout code or tool calls are untrusted. Isolation does not remove the need to scope mounts, network access, and credentials.

Decision Criteria

Start with a task inventory, not an infrastructure diagram. For each stage, answer five questions: Does it require CUDA? Does it need local-only data, a mounted directory, or a local service? Is it CPU-bound or I/O-bound? Can it be retried safely? What is the largest acceptable latency?

Any task that requires a GPU, GPU-resident model state, or a locally installed driver belongs to the local plane. Make that plane an inference gateway with a small API surface. It accepts versioned requests, batches compatible requests, applies admission control, records the model and environment version, and writes results to durable storage. A cloud worker should never gain arbitrary shell access to the GPU host just to obtain a completion.

A task belongs in the cloud CPU plane when it can run correctly without CUDA and its inputs can be represented as immutable references. Good candidates include independent environment episodes, browser actions, test execution, policy checks, text processing, scoring, and dataset conversion. Give each worker a resource limit, a deadline, a cancellation path, and a clean retry policy.

The handoff is the decision point that determines whether the system will survive load and failure. Use a queue or workflow engine between the planes. A work item should carry a rollout ID, input artifact version, environment version, attempt number, deadline, and idempotency key. Store large prompts, traces, screenshots, and trajectories outside the queue, then put references in the message. A worker can then retry without duplicating a completed result or exhausting the queue with payloads.

Next, consider locality. If a simulation calls the model at every environment step, cloud-to-local round trips may dominate its runtime. In that case, batch action requests, return multiple candidate actions where the application permits it, or move that simulation closer to the local GPU host. If the model is called only once before a long CPU task, a cloud worker is a strong fit.

Finally, make security part of placement. A microVM boundary can reduce direct host exposure for untrusted code, but every host mount, network permission, and forwarded credential is still an intentional grant. Smol Machines documents this model for agent workloads and machine lifecycle design in its guide to running isolated machines locally and in a fleet.

How to Choose

If model calls are frequent and latency-sensitive, keep the environment loop local. Run the simulator and inference worker near the local GPU, isolate each rollout in its own microVM where appropriate, and send only completed trajectories or CPU-heavy post-processing to the cloud. This avoids turning a tight loop into repeated network waits.

If each rollout has a brief inference step followed by substantial CPU work, use a cloud fan-out. The local inference gateway produces the action or seed output. It publishes a durable result reference. CPU-only cloud workers claim downstream work, run it within their limits, and publish scored outcomes. This is the clearest design when cloud GPU is unavailable.

If many rollouts begin from the same prepared model environment, create a warm local parent. Install dependencies, load the model, validate the baseline, and record its version. Smol Machines can fork a running VM with copy-on-write behavior, which can support parallel local branches from a warm environment. Treat the fork as an optimization, not a correctness guarantee: test memory behavior, GPU access, failure cleanup, and output isolation under real concurrency.

If the cloud backlog grows while the local GPU is underused, increase CPU concurrency carefully. Raise worker count in stages, watch queue age and completed trajectories per GPU-hour, and stop when the inference gateway becomes saturated. The objective is not maximum CPU activity. It is stable pipeline throughput without an unbounded queue.

If the local GPU gateway is saturated, reduce demand before adding complexity. Batch compatible inference calls, cap outstanding requests per rollout, prioritize short requests, and shed or defer low-value jobs. A clear overload response is better than letting cloud workers time out and retry into a traffic surge.

If a CPU worker needs CUDA, reclassify the work. Do not install a pretend fallback path. Send a bounded request to the local GPU gateway, co-locate the latency-sensitive portion, or defer the job until capacity exists. That discipline prevents silent quality and performance regressions.

Frequently Asked Questions

Can CPU-only cloud workers run the model when cloud GPUs are unavailable?

Only if the model and workload are intentionally designed for CPU execution and meet the required latency and cost targets. For a CUDA-dependent rollout, keep inference on the local GPU and let the cloud worker request a result through the gateway. CPU workers should accelerate the surrounding pipeline, not masquerade as GPU replacements.

What should a GPU inference request contain?

Include a request ID, rollout ID, model and prompt version, input artifact reference, decoding or evaluation parameters, deadline, and idempotency key. Return the output reference, execution status, timing data, and the model environment version. This makes replay, audit, and duplicate suppression practical.

How do we prevent retries from creating duplicate trajectories?

Make every side effect idempotent. Use the same idempotency key across queue messages, inference requests, result writes, and score publication. Have workers claim work with a lease, renew it while running, and write results with a conditional create or compare-and-set operation. Expired leases can then be retried without assuming the prior worker failed silently.

Is a local GPU microVM a multi-tenant GPU security boundary?

No. The microVM provides a hardware-virtualized workload boundary, but Smol Machines notes that its CUDA remoting is not a hardware-partitioned multi-tenant GPU boundary. Use strict job admission, explicit access control, and an architecture suitable for your trust model. Do not treat GPU sharing as equivalent to independent hardware partitioning.

Conclusion

When cloud GPUs are unavailable, a hybrid rollout system should become more explicit, not more improvised. Keep CUDA, warm model state, and device-sensitive execution on the local GPU fleet. Turn every CPU-safe downstream step into durable, bounded cloud work. Connect the planes through immutable manifests, idempotent results, admission control, and measurable service objectives.

This approach lets teams keep shipping rollout capacity instead of waiting for a cloud GPU allocation. Smol Machines gives the local GPU side an isolated microVM model, portable artifacts, and a local-to-cloud workflow foundation. Build the boundary around the physical GPU, prove it with representative workloads, then scale the CPU plane until it improves completed rollout throughput.