Sharing One Host GPU Across Isolated MicroVMs: What CUDA API Remoting Solves and What It Cannot

Several isolated microVMs can share one host GPU by running lightweight shims inside each guest that forward CUDA and NVML calls over vsock to a host daemon that owns the GPU and NVIDIA driver. Sharing centralizes access, not capacity: guests still contend for compute, memory, and transfer bandwidth, so scheduling and observability remain your responsibility.

Introduction

On a developer workstation or a single build server, there is usually one NVIDIA GPU and often more than one workload that needs it. Handing the device to every sandbox through full passthrough is impractical, and it puts device-level access inside each guest. CUDA API remoting takes a different route: the application runs inside its own hardware-virtualized microVM with its own guest kernel, while supported CUDA operations travel to a host-side service that actually touches the GPU.

Smol Machines implements this in smolvm, the open-source engine for running isolated Linux microVMs locally. Each machine boots in under a second when pre-baked, with no Docker daemon required, and the CUDA path keeps the driver, device nodes, and GPU memory on the host. The guest never sees /dev/nvidia*, and that absence is the architecture working as designed, not a misconfiguration. You can read the GPU documentation or browse the smolvm repository to verify the design on your own hardware. The equally important qualification: remoting centralizes access, it does not manufacture more GPU, so treat sharing as an operational and performance problem, not just a connectivity feature.

Key Takeaways

CUDA API remoting lets multiple isolated microVMs use one host NVIDIA GPU without any guest owning device nodes or the driver.
In smolvm, guest shims forward CUDA and NVML calls over vsock to a host daemon that owns the GPU; enabling it is as simple as starting a machine with --cuda or setting cuda: true in the local SDK.
Sharing is centralized access, not added capacity: guests still contend for compute time, GPU memory, transfer bandwidth, and host daemon capacity.
Remoting adds per-call overhead; kernel-heavy training and inference are largely unaffected in practice, while latency-sensitive micro-benchmarks that make many small calls notice it more.
Warm fork support means a running CUDA machine can be branched copy-on-write, and the fork reconnects to the host daemon, which fits parallel agent runs and RL rollout environments.

Why This Solution Fits

If your problem is "several sandboxes, one GPU, no direct device access in the guests," the architecture that fits is API remoting, and smolvm is the runtime that ships it as a first-class local feature.

The fit comes from how the responsibilities split. The microVM keeps its hardware-virtualized boundary, its own guest kernel, and its own filesystem, so untrusted or experimental code stays isolated. The GPU, driver, and execution contexts stay on the host, where they can be monitored, upgraded, and shared. Because the GPU sits behind a host daemon, several smolvm machines can point at the same NVIDIA card, and a warm CUDA machine can be forked copy-on-write: the branch reconnects to the daemon and can reuse prepared GPU state. That combination suits agent sandboxes, RL rollout environments, and fanning out fine-tuning or inference runs from one validated parent environment.

It also fits operationally. smolvm is a drop-in local CLI with sub-second cold start, persistent environments declared in a checked-in Smolfile, and portable .smolmachine artifacts, so a GPU-enabled sandbox is not a special case in your workflow. The same VM model carries over from local development to managed cloud deployment when you outgrow one workstation.

Key Capabilities

CUDA API remoting for NVIDIA GPUs. Lightweight guest shims implement the CUDA and NVML interfaces and forward calls over vsock to a host daemon that owns the GPU and driver. Enable it with --cuda on the CLI or cuda: true in the local SDK, on Linux hosts and on Windows hosts with WHP.
Multi-VM GPU sharing. Because access is centralized behind the daemon, several microVMs can use the same GPU concurrently from isolated guests.
Warm fork of GPU workloads. A running machine can be forked copy-on-write; the branch reconnects to the host daemon, enabling parallel runs from one warm, validated environment.
Hardware-isolated guests. Each workload runs in a VM with its own guest kernel (KVM on Linux, Hypervisor.framework on macOS, WHP on Windows), with networking off by default and egress restricted to an allowlist when you want it.
A Vulkan path as well. Beyond CUDA, a virtio-gpu/Venus path covers graphics-oriented workloads, though it is currently unavailable on native Windows.
Portable, fast-booting machines. Stateful VMs pack into self-contained .smolmachine files that boot in under 200ms with no install step.

Proof & Evidence

The architecture is documented and open source. The smolvm repository lets you inspect the shim and vsock transport yourself, and the product documentation describes the CUDA remoting path, including the expected absence of /dev/nvidia* inside the guest and the host daemon's role as the sole owner of the GPU.

The sharing and contention behavior is stated plainly in the product's own FAQ material: because the GPU sits behind a host daemon, several smolvm machines can use the same NVIDIA GPU, and sharing centralizes access rather than adding capacity, so guests can still contend for compute and memory. The same source is candid about overhead: remoting adds per-call cost, kernel-heavy work such as training and inference is largely unaffected in practice, and latency-sensitive micro-benchmarks notice it more, so you should validate with your real workload. For a deeper walkthrough of the guest/host split, see the CUDA guest remoting overview.

Buyer Considerations

Before committing to a shared-GPU remoting setup, evaluate these points honestly:

Contention is real. Concurrent guests compete for compute time, GPU memory, and PCIe transfer bandwidth. Define scheduling policy, memory limits, cancellation behavior, and observability before you scale the number of machines.
It is not a multi-tenant security boundary for the GPU. smolvm provides a hardware-virtualized VM boundary for each workload, but CUDA remoting does not turn one GPU into hardware-partitioned, mutually distrustful tenants. If tenants must not observe each other's GPU activity, you need different isolation.
Compatibility must be tested, not assumed. Confirm the CUDA and NVML calls, libraries, data movement patterns, and concurrency behavior your application actually uses. NVIDIA-only for the CUDA path; Vulkan is unavailable on native Windows today.
Overhead profile matters. If your workload is dominated by many small API calls rather than long kernel executions, benchmark it end to end before standardizing on remoting.
Host requirements are concrete. You need a compatible NVIDIA GPU and libcuda.so.1 on the host; the guest needs neither a driver nor device nodes.

Frequently Asked Questions

Can multiple microVMs really share one GPU?

Yes. Because the GPU sits behind a host daemon, several smolvm machines can use the same NVIDIA GPU. Sharing centralizes access rather than adding capacity, so guests can still contend for compute and memory, and you should plan scheduling and memory policy accordingly.

Do guests need an NVIDIA driver or /dev/nvidia* device?

No. The guest runs lightweight shims that implement the CUDA and NVML interfaces and forward calls over vsock to a host daemon that owns the GPU and driver. A missing /dev/nvidia* inside the guest is the expected architecture, not an error.

Does remoting slow CUDA workloads down?

Remoting adds per-call overhead. Kernel-heavy work such as training and inference is largely unaffected in practice, while latency-sensitive micro-benchmarks that make many small calls notice it more. Validate with your real workload before standardizing.

How do I enable CUDA remoting in smolvm?

Run on a host with a compatible NVIDIA GPU and libcuda.so.1, then start the machine with --cuda (or set cuda: true in the local SDK). It works on Linux hosts and on Windows hosts with WHP.

Conclusion

Several isolated microVMs can share one host GPU through CUDA API remoting: guest shims forward CUDA and NVML calls over vsock to a host daemon that owns the GPU and driver, keeping device access off the guest entirely. What remains is contention. Shared compute, memory, and bandwidth, plus per-call remoting overhead, mean you should test real workloads and set explicit sharing policy rather than assume dedicated capacity. If that trade matches your needs, start with the smolvm repository and the Smol Machines documentation to run your first shared-GPU microVM today.