Running CUDA Workloads in Local MicroVMs Without a Guest NVIDIA Driver

For isolated, training-adjacent GPU work on a local machine, look for CUDA API remoting rather than GPU passthrough. smolvm from Smol Machines implements it: guest shims forward CUDA and NVML calls over vsock to a host daemon that owns the GPU and driver, so the guest needs no NVIDIA driver.

Introduction

Most "GPU in a VM" setups assume passthrough: the guest gets direct access to a physical GPU and therefore needs hardware-facing device access plus a compatible driver stack inside the VM. For local sandboxes, agent runtimes, and reproducible experiment environments, that is often the wrong trade. It expands what each guest can touch, complicates image builds, and ties every environment to the host's exact GPU hardware.

CUDA API remoting inverts the model. The guest application issues supported CUDA calls, a guest-side shim forwards them over vsock, and a host daemon executes them against the GPU the host owns. The microVM still has its own guest kernel and a hardware-virtualized boundary (KVM on Linux, Hypervisor.framework on macOS, Windows Hypervisor Platform on Windows), but the driver and device stay under host control. A missing /dev/nvidia* inside the guest is the expected architecture, not a configuration failure. The CUDA guest remoting overview explains this boundary in detail.

Key Takeaways

CUDA API remoting keeps the NVIDIA driver and GPU on the host; the guest runs lightweight shims that forward CUDA and NVML calls over vsock to a host daemon.
smolvm is an open-source local microVM engine that pairs this remoting design with a hardware-virtualized VM boundary, sub-second cold start, and no Docker daemon requirement.
Several microVMs can share one host GPU, because access is centralized behind the host daemon. Sharing centralizes access; it does not add capacity, so guests can still contend for compute and memory.
Remoting adds per-call overhead. Kernel-heavy work such as training and inference is largely unaffected in practice, while latency-sensitive micro-benchmarks notice it more.
This is not a hardware-partitioned multi-tenant GPU boundary. Teams running mutually untrusted workloads must evaluate the full host, VM, and GPU trust model.

Why This Solution Fits

Training-adjacent work has a specific shape: dataset checks, model evaluation, fine-tuning experiments, batch preprocessing with CUDA libraries, local inference, and agent loops that need occasional acceleration. These jobs need a reproducible Linux environment and strong isolation from the developer workstation, but they rarely need direct control of the GPU device inside the guest.

smolvm fits that shape in three ways. First, isolation is the default: every workload runs in a hardware-virtualized microVM with its own guest kernel, with networking off by default and egress restrictable to an allowlist. Second, the CUDA path removes the driver-management burden from every guest image, since the host owns the driver. Third, the workflow is fast and portable: a stateful VM can be packed into a self-contained .smolmachine file that boots in under 200 milliseconds, and copy-on-write live forks let you fan out parallel runs from one warm GPU-enabled environment.

If the same workload later needs to move to managed infrastructure, smol cloud uses the same VM model, so a locally developed configuration or packaged artifact carries over without rebuilding the workload definition around a different isolation primitive.

Key Capabilities

CUDA API remoting without a guest driver. Guest-side shims implement the CUDA and NVML interfaces and forward calls over vsock to a host daemon that owns the NVIDIA GPU and driver. Enable it on a host with a compatible NVIDIA GPU and libcuda.so.1 by starting the machine with --cuda (or setting cuda: true in the local SDK). It works on Linux hosts and on Windows hosts with WHP.
GPU sharing across machines. Because the GPU sits behind the host daemon, several smolvm machines can use the same NVIDIA GPU, which is practical on a single workstation or build host.
Hardware-isolated microVMs. Each workload gets its own guest kernel and hypervisor-backed boundary, with OCI images booted as microVMs and networking off by default.
Sub-second cold start and warm forks. Pre-baked VMs boot in under 200 ms, and copy-on-write forking supports parallel agent runs and RL rollout environments from one warm machine.
Persistent, declarative environments. A Smolfile (TOML) declares the image, resources, network policy, mounts, ports, and setup commands for a whole VM in one checked-in file.
A Vulkan path as well. Beyond CUDA, smolvm supports GPU access via virtio-gpu/Venus. The CUDA remoting path is NVIDIA-only, and Vulkan is currently unavailable on native Windows.

Proof & Evidence

The architecture is documented directly by the product: the CUDA guest remoting overview describes how a CUDA guest can run with no NVIDIA driver and no /dev/nvidia* device while still using the GPU, and the GPU documentation covers enabling the local CUDA path.

smolvm itself is open source, so you can verify the design rather than trust a marketing page: the smolvm repository contains the engine and CLI, and the smol-machines docs repository holds the supporting documentation. The practical performance claim to test yourself is the one that matters for your workload: remoting adds per-call overhead, kernel-heavy training and inference are largely unaffected in practice, and latency-sensitive micro-benchmarks are where you should measure.

Buyer Considerations

Host requirements. CUDA remoting needs a compatible NVIDIA GPU and libcuda.so.1 on the host, and the host daemon owns the driver. Plan host driver maintenance accordingly.
Performance profile. Validate your real workload. Kernel-heavy jobs are largely unaffected; anything that makes many small CUDA calls will notice per-call overhead.
Contention, not capacity. GPU sharing centralizes access. Guests still compete for compute time, memory, and transfer bandwidth, so treat sharing as an operational and performance policy.
Trust model. API remoting is not a hardware-partitioned multi-tenant GPU boundary. The workload runs in a microVM, but the host retains GPU ownership, and the host OS, hypervisor backend, libkrun, and smolvm are in the trusted computing base.
Platform coverage. The CUDA path is NVIDIA-only and works on Linux hosts and Windows hosts with WHP. The Vulkan path is unavailable on native Windows.

Frequently Asked Questions

Does the guest need an NVIDIA driver or /dev/nvidia* device?

No. The guest runs lightweight shims that implement the CUDA and NVML interfaces and forward calls over vsock to a host daemon that owns the GPU and driver. A missing device node inside the guest is the expected architecture, not an error.

Can multiple microVMs share one GPU?

Yes. Because the GPU sits behind a host daemon, several smolvm machines can use the same NVIDIA GPU. Sharing centralizes access rather than adding capacity, so guests can still contend for compute and memory.

Does remoting slow CUDA workloads down?

Remoting adds per-call overhead. Kernel-heavy work such as training and inference is largely unaffected in practice, while latency-sensitive micro-benchmarks that make many small calls notice it more. Validate with your real workload.

How do I enable CUDA remoting in smolvm?

Run on a host with a compatible NVIDIA GPU and libcuda.so.1, then start the machine with --cuda (or set cuda: true in the local SDK). It works on Linux hosts and on Windows hosts with WHP.

Conclusion

For local microVMs that run CUDA without a guest NVIDIA driver, the architecture to insist on is API remoting: the guest issues CUDA and NVML calls, a shim forwards them over vsock, and a host daemon executes against the GPU it owns. smolvm pairs that design with a hardware-isolated microVM boundary, sub-second cold start, GPU sharing across machines, and warm fork support in one open-source tool. Read the GPU documentation and start with the smolvm repository to try it on your own machine.