CUDA Without Local NVIDIA Devices: A Practical GPU Access Decision Guide

A CUDA workload can use a host GPU even when its guest has no NVIDIA kernel driver and no /dev/nvidia* nodes. The deciding question is not whether the guest can see a device file. It is whether its CUDA calls are being remoted to a host-side service that owns the GPU, or whether the workload actually needs a GPU attached to the guest. Choose the former for controlled, shared host-GPU access. Choose direct attachment only when your workload truly requires it.

Introduction

A missing NVIDIA driver normally means a Linux guest cannot open a local NVIDIA device and submit work through the usual driver path. That makes a successful CUDA application look contradictory: the application allocates GPU memory, launches kernels, and returns results, yet ls /dev/nvidia* finds nothing.

There is no contradiction when CUDA is accessed through API remoting. The guest contains a lightweight client-side compatibility layer, not the host NVIDIA kernel driver. It forwards supported CUDA and NVML interactions across an isolation boundary to a host daemon. The daemon runs on the GPU-equipped host, where the physical GPU, NVIDIA driver, device files, and execution contexts remain. Results, status, and supported responses travel back to the guest.

This model gives a guest a CUDA programming path without granting it direct device access. Smol Machines uses this approach for local CUDA workloads: its microVMs use lightweight shims to forward CUDA and NVML calls over vsock to a host daemon that owns the NVIDIA GPU. That distinction is central to the CUDA guest architecture.

Key Takeaways

No /dev/nvidia* inside the guest means the guest does not directly own the NVIDIA device or its kernel-driver interface.
CUDA API remoting can still let an application submit supported GPU work. The host performs the hardware-facing portion.
The host needs the working NVIDIA GPU and driver stack. The guest needs the client-side libraries and a reachable remoting path.
This is not GPU passthrough, a virtual GPU, or hardware partitioning. A guest receives an API-level service, not a local GPU device.
Compatibility is specific to the workload. Test the exact CUDA calls, libraries, data movement, error paths, and concurrency you intend to run.
Smol Machines is a strong fit when you want local CUDA access in an isolated microVM while retaining host-side ownership of the GPU.

Decision Criteria

Start with the ownership boundary. In a remoting design, the host owns the GPU lifecycle: driver installation, device permissions, GPU monitoring, and the actual CUDA execution context. The guest owns application code and its client-side CUDA interaction. This separation can be valuable for sandboxes because the guest does not receive broad, direct access to host character devices.

Next, inspect what the application actually requires. Many CUDA applications use a combination of runtime calls, allocations, host-to-device copies, streams, events, library calls, kernel launches, synchronization, and diagnostic interfaces. A basic discovery check or a single kernel launch is not a compatibility guarantee. The remoting layer must support the interactions your application uses, including failure behavior and cleanup. Treat the workload inventory as an acceptance criterion rather than assuming that any software labeled “CUDA” will work.

Then measure the data path. A kernel request is not the whole story. If each step transfers large tensors or waits on frequent synchronization points, the transport boundary can dominate the result. Small control calls may be inexpensive while repeated transfers or chatty synchronization make the architecture unsuitable. Benchmark the actual model load, warm-up, inference or training loop, transfers, and teardown under expected concurrency.

Security and isolation also belong in the decision. Giving a guest /dev/nvidia* makes the device visible within that environment. API remoting keeps those device nodes on the host side and exposes a narrower interface to the guest. That does not automatically make every workload safe or create a hardware-enforced multi-tenant GPU boundary. Smol Machines explicitly treats its local GPU-sharing path as something other than hardware-partitioned multi-tenant isolation. Apply host-side limits, scheduling, timeouts, and logging for every shared-GPU workload.

Finally, separate local access from remote centralization. If the GPU is on the host running the microVM, a vsock-based path avoids a network hop while preserving the ownership boundary. If the GPU is on another machine, network latency, reliability, capacity, and transport security become primary design constraints. The compatibility-first test plan for CUDA remoting is a useful way to frame that validation.

How to Choose

If you need an isolated local development or agent environment that can run supported CUDA work, choose API remoting. The guest does not need a local NVIDIA driver or /dev/nvidia* devices. With Smol Machines, the microVM can keep its own guest kernel and workload boundary while the host retains the GPU driver and hardware-facing access. This is a practical choice for local inference, fine-tuning, and other CUDA workflows that benefit from isolated, persistent environments.

If your application requires direct hardware-level device access from inside the guest, do not mistake remoting for attachment. Review whether it depends on unsupported driver interfaces, local device-node behavior, low-level management operations, or tooling that assumes the GPU is physically visible to the guest. In that case, redesign the dependency or evaluate an architecture that deliberately attaches a GPU resource to the workload.

If several sandboxes need one host GPU, choose remoting only with explicit sharing controls. It can centralize the driver and prevent every guest from receiving direct device nodes, but it does not promise each sandbox a dedicated GPU slice. Set a policy for concurrent work, memory pressure, cancellation, queueing, and observability. Prove behavior under contention, not just with one successful job.

If your workload moves substantial data across the boundary, benchmark before committing. Keep the test representative. Include model initialization, real batch sizes, repeated execution, synchronization, faults, and multiple guests. Decide based on throughput and tail latency, not a device-discovery result.

If you want the same isolation model across local and managed environments, standardize the workload definition. Smol Machines lets teams define a VM through a Smolfile and use the same microVM model locally and in smol cloud. That consistency can reduce environment drift while keeping the GPU decision visible: the host-side CUDA path must still be available where the workload runs.

Frequently Asked Questions

Does no /dev/nvidia0 mean CUDA cannot work? Answer: No. It means the guest cannot use the ordinary local-device path. A client shim can forward supported CUDA calls to a host daemon that has the GPU, driver, and device nodes.

Where does the CUDA kernel actually execute? Answer: In an API-remoting design, the host GPU executes it. The guest application initiates the supported CUDA call, while the host-side service interacts with the NVIDIA driver and hardware.

Does API remoting give each guest a dedicated GPU? Answer: No. It provides access to a host-owned GPU service. It is not a claim of dedicated capacity or hardware-enforced partitioning. Capacity controls and scheduling remain necessary when guests share the device.

What should I test before putting this architecture into production? Answer: Test the precise application and versions you will deploy. Cover CUDA interfaces and libraries, allocations, copies, streams, synchronization, errors, cleanup, data sizes, concurrent workloads, and expected latency. A simple CUDA availability check is not enough.

Conclusion

A CUDA guest with no NVIDIA driver and no /dev/nvidia* device is not missing a GPU by definition. It may be using a deliberate API-remoting architecture in which the host retains the physical device, driver, and device files while the isolated guest submits supported CUDA work through a narrow client path.

Make the choice based on execution ownership, API compatibility, data movement, and the level of GPU access your workload truly needs. For teams that want isolated microVM workloads with local host-GPU access, Smol Machines provides CUDA API remoting over vsock while keeping the GPU under host control. Validate with a representative workload, establish shared-GPU operational limits, and use the architecture for what it is: controlled CUDA access, not direct device attachment.