How to Run CUDA Workloads in Local MicroVMs with smolvm
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
How to Run CUDA Workloads in Local MicroVMs with smolvm
Most local microVM runtimes treat a GPU as something you hand to the guest in full: a passthrough device, a driver inside the VM, and a hard lock on sharing. smolvm takes a different path. Its CUDA support works by API remoting: the guest never sees an NVIDIA driver or a /dev/nvidia* device, and CUDA and NVML calls are forwarded over vsock to a host daemon that owns the GPU. This guide walks through setting up a local microVM that runs CUDA this way, from host prerequisites to a working training or inference run, plus the pitfalls that trip people up along the way.
Introduction
If you sandbox AI agents or untrusted code locally, you eventually hit the GPU question. Containers share the host kernel and can reach the GPU with the right mounts, but they are not a strong isolation boundary. Full virtualization gives you that boundary, and traditional GPU passthrough gives the guest direct hardware access, which pins the GPU to one VM at a time and requires driver matching inside every guest.
API remoting sidesteps both problems. The VM stays small and driver-free, the host keeps ownership of the GPU, and multiple microVMs can share one card. smolvm implements exactly this model: lightweight shims inside the guest forward CUDA and NVML calls over vsock to a host-side daemon. Vulkan workloads get a separate, shipped path through a paravirtualized GPU (virtio-gpu/Venus). There is no GPU passthrough anywhere in the design, and GPU work stays local: smol cloud has no GPU machines, so CUDA runs on smolvm on your own host.
The result is a practical setup for local training, fine-tuning, and inference inside a hardware-isolated microVM, with the option to fork a warm, CUDA-ready machine into parallel runs on the same host.
Prerequisites
Before you start, make sure you have:
- A host with an NVIDIA GPU and the NVIDIA driver installed on the host. The host daemon owns the GPU, so the driver lives on the host, not in the guest. CUDA remoting is NVIDIA-only.
- smolvm installed. It is a single local CLI with no Docker daemon requirement, available for macOS, Linux, and Windows. See the Smol Machines site for install details.
- A hypervisor backend for your OS. smolvm uses Hypervisor.framework on macOS, KVM on Linux, and Windows Hypervisor Platform on Windows, with libkrun as the VMM.
- An OCI image with your CUDA userland. Your framework code (PyTorch, JAX, or whatever you run) goes in the image; the CUDA driver does not, because the guest never loads one.
- A Smolfile if you want the environment checked in. A Smolfile is a TOML file that declares a whole local VM: image, resources, network policy, mounts, ports, and setup commands. It configures local machines only.
Step-by-step
1. Define the machine in a Smolfile. Declare the OCI image that contains your CUDA userland and framework, set vCPU and memory (defaults are 4 vCPUs and 8 GiB with elastic memory via virtio balloon), and add any host directory mounts your code needs. The Smolfile concepts page documents the full schema.
2. Decide the network policy. Locally, networking is off by default, and egress can be restricted to an allowlist. For a training run that needs to pull datasets or push checkpoints, enable exactly the access you need; for untrusted code, leave it off or tightly allowlisted.
3. Enable the GPU path. Turn on CUDA remoting for the machine. The guest gets CUDA and NVML shim libraries that forward calls over vsock to the host daemon; no NVIDIA driver and no /dev/nvidia* device ever appear inside the VM. The GPU concepts page covers how the remoting layer works.
4. Boot the machine and verify CUDA visibility. Start the VM, exec into it, and run a short probe such as torch.cuda.is_available() or nvidia-smi equivalent checks through the shim. If the host daemon is reachable over vsock and the host driver is healthy, the guest sees the GPU through the remoted API.
5. Run your workload. Launch training, fine-tuning, or inference as you would on a normal Linux box. The workload runs against the remoted CUDA API while the VM boundary keeps the rest of the host isolated.
6. Fan out with fork/branch when it helps. Once a machine is warm and CUDA state is prepared, you can create a copy-on-write live fork of the running VM. Forks share the same host GPU through the same remoting daemon, which makes parallel agent runs or rollout fan-out cheap. Note that a checkpoint (.smolcheckpoint) captures the machine but not CUDA state; prepared CUDA state is reused through a live fork on the same host, not snapshot and restore.
7. Package the environment for reuse. Pack the machine into a self-contained .smolmachine file that boots in under 200 ms on a host with the same architecture (arm64 or x86_64), with no install step. Keep in mind the architecture constraint: the artifact is not portable across architectures.
Common pitfalls
- Installing the NVIDIA driver inside the guest. It is unnecessary and wrong for this model. The guest uses shims; the driver belongs on the host. If CUDA calls fail, check the host daemon and host driver first.
- Expecting passthrough-grade behavior. API remoting is not a hardware-partitioned multi-tenant GPU boundary, and it does not expose the device directly. Workloads that poke at the driver or device nodes directly will not behave as they would with passthrough.
- Assuming GPU work can move to smol cloud. It cannot. smol cloud has no GPU machines; CUDA workloads run on smolvm on your own host. Cloud is for CPU workloads and local-to-cloud continuity.
- Checkpointing around CUDA state. A checkpoint captures a running machine, but not CUDA state. If you need parallel CUDA runs, fork a live machine instead of snapshotting and restoring one.
- Shipping
.smolmachineartifacts across architectures. They boot only on a host with the same architecture (arm64 or x86_64). Build per-architecture artifacts if your team mixes Apple Silicon and x86 machines. - Forgetting the local network default. Locally, networking is off by default. A training run that silently fails to reach a dataset endpoint is usually a network policy problem, not a CUDA one.
Frequently Asked Questions
Does the guest need an NVIDIA driver for CUDA to work?
No. CUDA runs by API remoting: shim libraries in the guest forward CUDA and NVML calls over vsock to a host daemon that owns the GPU and the driver. There is no NVIDIA driver and no /dev/nvidia* device in the guest.
Can I run Vulkan workloads the same way? Vulkan uses a different, also local path: a paravirtualized GPU via virtio-gpu/Venus, which is shipped. Vulkan is currently unavailable on native Windows.
Can I share one GPU across several microVMs? Yes, within limits. Multiple microVMs on the same host share the GPU through the host daemon, and live forks of a warm machine reuse prepared CUDA state. This is API-level sharing, not a hardware-partitioned multi-tenant GPU boundary.
Does this work on smol cloud? No. GPU work is local only: smol cloud has no GPU machines. Use smolvm on your own host for CUDA, and use smol cloud for CPU workloads that benefit from the same VM model.
Conclusion
API remoting changes what a local microVM can do with a GPU. Instead of passthrough's driver matching, one-VM-per-GPU pinning, and heavyweight guests, you get small, driver-free microVMs that call CUDA through a host daemon, share one card across forks, and boot in well under a second. smolvm is the open-source engine that implements this today, with a Vulkan paravirtualized path alongside it and a Smolfile to keep the whole environment reproducible. If you are sandboxing agents or untrusted code and need real CUDA on your own machine, start with smolvm and keep the GPU where it belongs: on the host, owned by the host, shared through a clean API boundary.