GPU access by API remoting: how a driverless microVM runs CUDA, and how it compares

A microVM has a real kernel and a real isolation boundary, which is exactly what you want for untrusted or agent-driven code. It also, by default, has no GPU. The recent work in smol machines gives a guest a GPU without passing one through: the guest carries no NVIDIA driver and no device nodes — only a set of drop-in shim libraries that forward every CUDA call over vsock to a host process that owns the real GPU. This post is about that approach, API remoting, and how it stacks up against the other ways to put a GPU behind an isolation boundary.

Four ways to give a workload a GPU

Before the mechanism, the landscape — because API remoting only makes sense against what it replaces. There are, broadly, four ways to let an isolated workload touch an NVIDIA GPU.

The first two keep the GPU driver inside the guest and hand the guest real hardware — great performance, but the guest is now a full driver host, and passthrough binds an entire GPU to one VM. Container device injection shares a GPU cheaply but drops the VM boundary, which is the one thing a microVM exists to keep. API remoting is the odd one out: the GPU and its driver never enter the guest at all.

What API remoting is

The idea is old — rCUDA and similar systems have remoted CUDA across a network for years. An API-remoting system intercepts the CUDA API in the guest and forwards each call to a server that owns the GPU, which runs it and returns the result. The guest holds no driver, opens no /dev/nvidia*, and can be a few megabytes of musl userspace; the GPU stays where it is and is shared by whoever connects.

smol machines applies this host-locally. The server is not across a datacenter network — it is the host the microVM is already running on, reached over vsock, so the transport is a memory copy between guest and host rather than a NIC round trip.

How smol machines does it

The guest gets three drop-in libraries, each implementing a CUDA C ABI by marshalling every call over the same vsock RPC:

smol machines places these where the loader looks, so an unmodified CUDA binary runs unchanged — it links libcuda.so.1 like always, except the library is a proxy. The NVML shim is the quietly important one: frameworks decide whether a GPU exists by asking NVML, so without it a remoted GPU is invisible to the very tools you want to run.

On the host, a server owns the GPU. Each guest connection is served against a backend chosen automatically: the real driver (a GpuBackend that loads libcuda.so.1) when a GPU is present, or a CPU-emulation backend otherwise — so the whole transport is testable on a machine with no NVIDIA GPU. For sharing and for fork, smolvm runs a single daemon that holds one GPU context and serves every CUDA VM's connection through it; it starts lazily on the first CUDA VM, persists across VMs, and releases the context on an idle timeout when nothing is connected.

The trade-offs, honestly

API remoting is not free, and it is not the right answer for every workload:

• Per-call overhead. Every CUDA call becomes an RPC. vsock keeps that to a host-local memory copy rather than a network hop, but it is not the near-zero cost of passthrough. Workloads dominated by many tiny, latency-sensitive calls feel it most; ones that launch large kernels and large transfers feel it least.
• The CUDA ABI is a surface smol machines has to cover. Remoting means implementing the CUDA API, and the API is large and moving. A call smol machines has not implemented yet fails where passthrough would have simply worked — the cost of not shipping the driver is having to implement its interface.
• It needs a host GPU and driver, and it is NVIDIA-only. The host still runs the real driver on real hardware. Remoting moves where the driver lives; it does not remove it.

Against that: the guest carries no driver and no device nodes, one physical GPU is shared across many VMs through the daemon, and — unlike container device injection — the workload stays behind a real VM boundary. For running untrusted or agent-generated GPU code, that boundary is the whole point, and it is the one passthrough-into-a-container can't give you.

Where it pulls ahead: fork

The reason to remote the API rather than pass the hardware through shows up when you fork. A passed-through GPU belongs to one VM and cannot be cloned; a forked VM has nowhere to attach it. Remoting decouples the workload from the device, so a fork clone simply opens its own connection to the same host daemon.

Because the daemon holds a single shared GPU context, a clone can reconnect and reuse its golden's device memory rather than reloading it. The recent work carries this through the hard cases: captured CUDA graphs are handed off to clones so graph-capturing workloads fork-resume instead of faulting on the parent's handles, the retained primary context is reported so cuBLAS survives CUDA-graph capture, and buffered ops are carried across the clone's reconnect. The practical result is that a warm, model-loaded server VM — vLLM with torch.compile and cudagraphs, its production configuration — can be forked, and each clone keeps serving on the same physical GPU. Passthrough has no equivalent.

Turning it on

One flag remotes the guest's CUDA calls to the host GPU:

# The guest has no GPU and no NVIDIA driver — only the shim libraries.
# --cuda points its libcuda.so.1 at the host GPU over vsock.
smol machine run --cuda -I nvidia/cuda:12.4.0-runtime-ubuntu22.04 -- \
    python -c "import torch; print(torch.cuda.get_device_name(0))"

# a persistent, forkable serving VM — fork it per request, each keeps the GPU
smol machine create --name serve --cuda --forkable
smol machine start  --name serve

The same applies to a packed .smolmachine (the CUDA setting is preserved through pack create --from-vm, so a packed GPU workload runs self-contained) and to the embedded engine the SDKs drive (a cuda resource flag). The guest picks its transport from an environment variable, which is what makes the shims testable off a real GPU:

SMOLVM_CUDA_RPC (guest, chosen by the shim):
  unset / vsock   AF_VSOCK to host CID 2, port 7000   (production, in-guest)
  tcp:HOST:PORT   TCP                                  (host-side testing)
  unix:/path      AF_UNIX                              (host-side testing)

When to reach for which

The honest summary is that these are different tools. If you own the host, trust the workload, and want maximum throughput on a whole GPU, passthrough is hard to beat. If you are partitioning datacenter GPUs across tenants and have the licensing, vGPU and MIG are built for it. If you just need containers to see a GPU and isolation is not the concern, the container toolkit is the least friction.

API remoting wins on a specific axis: a small, driverless, VM-isolated guest that shares a host GPU and can be forked. That is the combination that matters for running many short, untrusted, GPU-touching workloads — which is exactly the shape of agent and sandbox workloads — and it is the combination none of the others offer at once.

CUDA-over-vsock is in the smol machines runtime — read the GPU guide or take the runtime from GitHub.