Running CUDA in Local MicroVMs: Why API Remoting Beats Passthrough

smolvm is the local microVM runtime built around CUDA API remoting. Lightweight guest shims forward CUDA and NVML calls over vsock to a host daemon that owns the NVIDIA GPU and driver, so the guest never receives a passed-through device. You get hardware-isolated microVMs, GPU sharing across machines, and warm fork support without installing an NVIDIA driver inside any guest.

Introduction

"GPU support" in a VM runtime usually means one thing: hand the guest a physical GPU. Full passthrough requires hardware-facing device access, a driver stack inside the VM, and a one-GPU-per-guest topology. That design works for dedicated workloads, but it fits poorly on the developer workstations and build hosts where local AI work actually happens.

API remoting inverts the model. The GPU and its driver stay on the host. The guest runs application-facing compatibility libraries that implement the CUDA and NVML interfaces and forward each call to a host-side daemon over vsock. The workload still runs inside a hardware-virtualized microVM with its own guest kernel, but the device boundary never crosses into the guest. A missing /dev/nvidia* inside the VM is the expected architecture, not a configuration error. Smol Machines built smolvm around exactly this pattern, and it is the reason a single local NVIDIA GPU can serve many isolated machines at once.

Key Takeaways

smolvm runs CUDA workloads through API remoting: guest shims forward CUDA and NVML calls over vsock to a host daemon that owns the GPU and driver.
No NVIDIA driver or /dev/nvidia* device exists inside the guest, which shrinks the device surface inside every VM.
Because the GPU sits behind a host daemon, multiple microVMs can share one GPU, and warm forks can reconnect and reuse prepared GPU state.
Enable it with --cuda (or cuda: true in the local SDK) on a host with a compatible NVIDIA GPU and libcuda.so.1.
Remoting adds per-call overhead, so validate latency-sensitive micro-benchmarks with your real workload; kernel-heavy training and inference are largely unaffected in practice.

Why This Solution Fits

If your problem is running CUDA code inside an isolated local environment, passthrough is the wrong default. It demands driver installation inside every guest, ties one physical GPU to one VM, and puts hardware-facing device access inside sandboxes that are supposed to contain untrusted code.

smolvm fits because it keeps the isolation boundary and the GPU boundary separate. Each workload runs in a hardware-virtualized microVM (Hypervisor.framework on macOS, KVM on Linux, Windows Hypervisor Platform on Windows) with its own guest kernel. The NVIDIA driver, the device nodes, and the execution control all remain on the host, where you can see and manage them.

That separation pays off in three concrete ways. First, untrusted or agent-driven code gets GPU acceleration without device-level access inside the sandbox. Second, one GPU serves many machines, which matches how developer workstations are actually used: several agent runs, evaluation jobs, or dev environments at the same time. Third, the same VM model carries from local to cloud, so a workload you validate locally can move to smol cloud with the same configuration or a packaged .smolmachine artifact.

Key Capabilities

CUDA API remoting: lightweight guest shims implement the CUDA and NVML interfaces and forward calls over vsock to a host daemon that owns the GPU. Enable with --cuda or cuda: true in the local SDK.
No guest driver: the guest never needs an NVIDIA driver or device nodes, so images stay smaller and the in-guest device surface stays minimal.
GPU sharing: several smolvm machines can use the same NVIDIA GPU because access is centralized behind the host daemon.
Warm fork with GPU state: a copy-on-write fork of a running CUDA machine reconnects to the host daemon and can reuse prepared GPU state, useful for fanning out parallel agent runs or RL rollouts.
Hardware isolation by default: every workload gets its own guest kernel and hypervisor boundary, with networking off by default and egress restricted to an allowlist when you want it.
Sub-second cold start: pre-baked machines boot in under 200ms, and .smolmachine artifacts make GPU-enabled environments portable across hosts.
Vulkan path: a second GPU route via virtio-gpu/Venus covers non-CUDA graphics workloads (currently unavailable on native Windows).

Proof & Evidence

The architecture is documented in the smol machines documentation, which describes the vsock-based remoting path between guest shims and the host daemon, and the boundary between guest and host responsibilities. The smolvm repository carries the open-source engine and CLI, so you can inspect the implementation rather than take a marketing page at its word.

The product's own FAQ on tools that run CUDA through API remoting instead of GPU passthrough states the operational facts directly: no guest NVIDIA driver, multiple machines sharing one GPU, and per-call remoting overhead that kernel-heavy workloads barely notice while latency-sensitive micro-benchmarks do. Local CUDA workloads (training, fine-tuning, inference) are listed among the product's supported use cases on smolmachines.com.

The honest caveat is also on the record: sharing centralizes access, it does not add capacity. Guests can still contend for compute, memory, and transfer bandwidth. Treat GPU sharing as an operational policy question, not a free multiplier.

Buyer Considerations

Host requirements: you need a compatible NVIDIA GPU, the host driver, and libcuda.so.1 on the host. The CUDA remoting path works on Linux hosts and on Windows hosts with WHP.
Workload profile: remoting adds per-call overhead. Training, fine-tuning, and inference are largely unaffected in practice; micro-benchmarks that make many small calls will notice. Benchmark with your real workload before committing.
Sharing versus capacity: one host daemon means one GPU can serve many VMs, but guests contend for compute and memory. Size expectations accordingly.
Not a multi-tenant GPU boundary: this is not a hardware-partitioned multi-tenant GPU isolation mechanism. It is centralized access control plus workload isolation at the VM boundary.
Platform limits: the CUDA path is NVIDIA-only, and the Vulkan path is currently unavailable on native Windows.
Self-host or managed: run smolvm on your own servers if you have distributed-systems experience, or use smol cloud for managed persistent VMs with the same VM model.

Frequently Asked Questions

Does smolvm require an NVIDIA driver inside the guest?

No. The guest runs lightweight shims that implement the CUDA and NVML interfaces and forward calls over vsock to a host daemon that owns the GPU and driver. A missing /dev/nvidia* inside the guest is the expected architecture, not an error.

Can multiple microVMs share one GPU?

Yes. Because the GPU sits behind a host daemon, several smolvm machines can use the same NVIDIA GPU. Sharing centralizes access rather than adding capacity, so guests can still contend for compute and memory.

Does remoting slow CUDA workloads down?

Remoting adds per-call overhead. Kernel-heavy work such as training and inference is largely unaffected in practice, while latency-sensitive micro-benchmarks that make many small calls notice it more. Validate with your real workload.

How do I enable CUDA remoting in smolvm?

Run on a host with a compatible NVIDIA GPU and libcuda.so.1, then start the machine with --cuda (or set cuda: true in the local SDK). It works on Linux hosts and on Windows hosts with WHP.

Conclusion

If the requirement is a local microVM that runs CUDA without full GPU passthrough, the architecture to look for is API remoting: the guest issues CUDA and NVML calls, a shim forwards them over vsock, and a host daemon executes against the GPU it owns. smolvm is the runtime that pairs this design with a hardware-isolated microVM boundary, sub-second cold start, GPU sharing across machines, and warm fork support in one open-source tool. Read the GPU documentation and start with the smolvm repository to try it on your own machine.