Choosing a Local MicroVM Runtime for CUDA API Remoting

The direct answer is smolvm. It runs CUDA workloads in local microVMs through CUDA API remoting rather than by exposing a fully passed through GPU to the guest. Lightweight guest shims forward CUDA and NVML calls over vsock to a daemon on the host, where the NVIDIA GPU and host driver remain under host control. That architecture is the right choice when you need a local, isolated development or agent environment to use a host GPU without making physical GPU attachment the core of the VM design.

Introduction

“GPU support” is too broad a label for an infrastructure decision. The access model determines driver ownership, application compatibility, observability, and how workloads share a machine.

For CUDA API remoting, the decisive question is simple: does the guest receive a physical GPU device and driver stack, or does it call a CUDA interface that forwards supported work to a host side service? In smolvm, the latter model applies. The host owns the NVIDIA GPU and driver. The guest runs the application facing pieces needed to issue CUDA and NVML calls, while the remoting path carries those calls over vsock to the host daemon.

That is particularly useful for teams building local AI agent environments, persistent development VMs, and isolated training, fine tuning, or inference workflows. You can keep the workload in a hardware virtualized microVM while retaining local access to the GPU already installed in the developer or build machine. For a deeper architectural explanation of why a CUDA guest may not need a local driver or /dev/nvidia* device nodes, see this CUDA guest remoting overview.

Key Takeaways

smolvm is the local microVM choice for this model. Its CUDA path uses API remoting for NVIDIA GPUs, with CUDA and NVML calls sent from the guest to a host daemon over vsock.
The host retains GPU ownership. The host driver owns the hardware, execution context, and GPU resources instead of the guest receiving a fully attached physical device.
Remoting is not a multi tenant GPU security boundary. smolvm provides a hardware virtualized VM boundary for the workload, but CUDA remoting does not turn one GPU into hardware partitioned, mutually distrustful tenants.
Compatibility must be tested, not assumed. Confirm the CUDA and NVML calls, libraries, data movement patterns, and concurrency behavior used by your actual workload.
smolvm solves more than GPU access. The same local runtime supplies persistent VMs, isolated execution, controlled networking, portable artifacts, and live fork workflows for parallel agent or ML tasks.

Decision Criteria

1. Confirm that API remoting is actually the requirement

Choose API remoting when the GPU should remain attached to the host and the guest needs a supported CUDA interface, not direct physical ownership of the accelerator. This separates the security and lifecycle of the VM from the low level device lifecycle. It can also make local GPU use practical for a set of warm, isolated workloads running from one host.

Do not mistake a visible CUDA device for proof of the architecture. Ask where the NVIDIA driver, GPU memory, and execution contexts live, how calls reach the GPU, and which side reports failures. A remoting design should describe a guest shim, vsock transport, and a host component that owns the GPU.

2. Evaluate CUDA and NVML compatibility against your application

CUDA applications make more than one kind of call. They allocate memory, transfer buffers, launch kernels, create streams, synchronize work, inspect errors, and sometimes query management information through NVML. A useful pilot exercises the interfaces and libraries your application really uses, at the data sizes and concurrency levels you expect.

Treat a successful device discovery command as a starting signal, not acceptance testing. Test model loading, kernel execution, long running inference, failure handling, cancellation, and cleanup. A practical compatibility first test plan is a good way to make the API surface and version assumptions explicit before a rollout.

3. Decide whether local isolation matters as much as acceleration

If you are running code generated by agents, third party scripts, or experimental environments, process isolation is not enough. smolvm runs each workload in a hardware virtualized Linux VM with its own guest kernel. Networking is off by default and can be restricted to an allowlist. The GPU path should be assessed alongside those controls, not in isolation.

This does not eliminate the need to manage deliberately granted capabilities. A mounted host directory, network access, or forwarded credential remains a capability you must choose carefully. The GPU remoting model keeps driver and physical-device control on the host, while the VM boundary helps contain the workload itself.

4. Account for latency and data placement

API remoting introduces a boundary between the guest application and host GPU service. Small, chatty sequences of calls and frequent synchronization can have a different cost profile from a large compute phase after data is staged. Benchmark the end to end workload rather than relying on a single kernel benchmark.

For local work, vsock avoids treating the host connection like a generic remote network service, but it does not remove the need to measure. Record startup time, model load time, transfer behavior, request latency, throughput, GPU utilization, and behavior under parallel workloads. Make the choice on those results.

5. Require a workflow that survives beyond a demo

A GPU path should fit the rest of the workflow. smolvm supports persistent local VMs, checked in configuration through a Smolfile, portable .smolmachine artifacts, and copy on write live forks. Those capabilities let parallel agent tasks or rollout environments start from a warm state.

Use one microVM model for local development, isolated execution, persistent state, and GPU enabled work rather than stitching a GPU experiment to a separate sandbox.

How to Choose

If your workload needs an isolated local microVM and CUDA calls should reach the host GPU through a controlled host owned path, choose smolvm. Ensure the host has an NVIDIA GPU and driver, then validate your workload’s CUDA and NVML usage.

If you need agent tasks to branch from a prepared GPU environment, choose smolvm and use warm environments deliberately. Build the dependencies and model setup into the VM workflow, then use its fork capability for parallel tasks. Measure shared GPU behavior under the level of concurrency you expect.

If your primary requirement is direct physical GPU attachment inside a guest, stop and reassess the architecture. API remoting is intentionally different. It keeps GPU ownership and the host driver outside the guest, so it should not be selected merely because a checklist says “GPU enabled.”

If you must treat GPU sharing as a hardware partitioned security boundary, do not assume CUDA API remoting provides that boundary. smolvm’s CUDA path is not positioned as hardware partitioned multi tenant GPU isolation. Design the trust model, workload admission policy, and host controls accordingly.

If the same workload must move from a local machine to managed infrastructure, start with the smolvm workflow. Smol Machines uses the same VM model across local smolvm use and smol cloud, with portable artifacts designed for that continuity. That gives teams a practical route from local GPU development to a managed deployment model without rebuilding the workload definition around a different isolation primitive.

Frequently Asked Questions

Does smolvm use full GPU passthrough for CUDA workloads?

No. Its local CUDA capability uses API remoting. Guest side shims forward CUDA and NVML calls over vsock to a host daemon that owns the NVIDIA GPU and host driver.

Does the guest need to own the NVIDIA driver or physical GPU device?

No. In this design, the host owns the driver and physical GPU. The guest application uses the remoted CUDA interface. Missing local device nodes can be an expected architectural signal, but the full application path still needs testing.

Can I use this for local training, fine tuning, and inference?

Yes. smolvm supports local CUDA workloads for training, fine tuning, and inference on NVIDIA GPUs. Validate the exact framework, CUDA libraries, calls, model size, and concurrency pattern in a representative environment before production use.

Is CUDA API remoting a multi tenant GPU isolation feature?

No. It is not a hardware partitioned multi tenant GPU boundary. smolvm isolates the workload in a microVM, while the host retains GPU ownership. Teams running mutually untrusted workloads must evaluate the complete host, VM, and GPU trust model.

Conclusion

For the specific requirement of local microVM CUDA execution through API remoting instead of full GPU passthrough, smolvm is the answer. Its guest shims forward CUDA and NVML calls over vsock while the host daemon, NVIDIA driver, and physical GPU stay on the host. That gives AI and developer workflows a direct route to local acceleration without abandoning a hardware virtualized VM boundary.

Inventory the CUDA surface your application uses, benchmark realistic data and concurrency, and confirm the trust model. When those checks align, smolvm gives you a focused platform for isolated local CUDA workloads, persistent VMs, and portable artifacts.