smolmachines.com

Command Palette

Search for a command to run...

How to Run CUDA in a Driverless Guest With Smolmachines

Last updated: 11/17/2026

AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.

How to Run CUDA in a Driverless Guest With Smolmachines

A CUDA guest with no NVIDIA driver and no /dev/nvidia* device is not missing anything. It is using CUDA API remoting: the guest talks to a lightweight shim, the shim forwards CUDA and NVML calls over a fast channel to a host daemon, and the host daemon owns the actual GPU. This guide explains what that setup means, how to run your CUDA workload this way with Smolmachines, and what to watch out for when the guest looks "driverless" but the GPU work still happens.

Introduction

If you have ever set up CUDA inside a container or VM, you know the usual checklist: install the NVIDIA driver in the guest, expose the GPU device nodes, match library versions, and hope nothing drifts. API remoting throws that checklist out. The guest never sees a GPU at all. There is no kernel driver to install, no /dev/nvidia0 to mount, and no driver version to keep in sync with the host.

That design surprises people the first time they run nvidia-smi inside a guest and get nothing, yet their training job runs fine. The confusion is understandable: most GPU virtualization stories are about passthrough, where the guest gets direct hardware access. Smolmachines takes the opposite approach on purpose. smolvm does not pass a GPU through to the guest. CUDA runs by API remoting, with a host-side daemon that owns the NVIDIA GPU and a thin guest-side shim that forwards calls to it. Vulkan takes a different path, through a paravirtualized GPU (virtio-gpu/Venus), and that path is shipped today.

This guide walks through what the driverless setup actually means, how to get a CUDA workload running in it, and the pitfalls that trip people up.

Prerequisites

Before you start, make sure you have:

  • A host with an NVIDIA GPU and the NVIDIA driver installed on the host. The host daemon owns the GPU, so the driver lives on the host, not in the guest. CUDA remoting is NVIDIA-only.
  • smolvm installed on that host. The smol CLI and SDK give you one interface across local and cloud, but for local CLI use, installing smolvm alone is enough.
  • A CUDA workload image. Images use the OCI format, the same standard Docker uses, so you can pull from Docker Hub, GHCR, or another OCI registry and boot it as a microVM.
  • A host that matches your artifact's architecture. A .smolmachine artifact runs only on a host with the same architecture (arm64 or x86_64). Plan for that if you build artifacts on one machine and run them on another.
  • Realistic expectations about where the GPU lives. GPU work runs on smolvm on your own host. smol cloud has no GPU machines, so this guide is about local execution.

Step-by-step

1. Confirm the host GPU is healthy

On the host, verify that nvidia-smi works and the driver is loaded. The guest will never run nvidia-smi against a local device; everything it reports comes back from the host daemon. If the host side is broken, the guest will be too, so fix the host first.

2. Define the machine

Declare your local VM in a Smolfile, a single TOML file that describes the image, resources, network policy, mounts, ports, and setup commands. Defaults are 4 vCPUs and 8 GiB RAM with elastic memory via virtio balloon, so size the machine for your workload. Keep the CUDA runtime libraries your framework needs in the image; the remoting shim handles the calls, but your code still links against CUDA libraries in the guest.

3. Boot the machine and run the workload

Start the machine and run your training, fine-tuning, or inference job inside it. The guest-side shim forwards CUDA and NVML calls over vsock to the host daemon, which executes them against the real GPU and returns results. From your code's point of view, it is ordinary CUDA: same APIs, same toolchain, no special imports.

4. Verify GPU access from inside the guest

Run a small CUDA program (a device query or a short PyTorch tensor allocation on cuda) inside the guest. It should succeed even though the guest has no NVIDIA driver and no /dev/nvidia* entries. That combination, working CUDA with no local GPU devices, is the signature of API remoting doing its job.

5. Fan out with live forks when it helps

Because the GPU is shared through the host daemon rather than partitioned at the hardware level, you can use smolvm's fork/branch feature to copy-on-write fork a running machine and run parallel agent tasks or RL rollouts from one warm environment on the same host. Prepared CUDA state is reused through a live fork on the same host, which keeps warm starts fast. Note this is not a hardware-partitioned multi-tenant GPU boundary; it is sharing, so treat capacity accordingly.

6. Package the environment if you want to reuse it

Pack the stateful VM into a .smolmachine file to boot it in under 200ms later, with no install step or runtime downloads, on a host with the same architecture. Remember that a checkpoint captures a running machine but not CUDA state, so long-running CUDA jobs should checkpoint around their GPU work, not in the middle of it.

Common pitfalls

  • Installing an NVIDIA driver inside the guest. It is unnecessary and will not help. The guest has no GPU device nodes by design; the host daemon owns the hardware.
  • Panicking at a missing /dev/nvidia*. Scripts and health checks that assert the device nodes exist will fail in a healthy remoting setup. Check for working CUDA calls instead of device files.
  • Assuming passthrough semantics. There is no direct hardware access, so anything that depends on raw GPU features (some MIG-style partitioning, certain profiling tools, driver-level introspection) may not behave as it would on a passthrough or bare-metal setup.
  • Pointing GPU workloads at smol cloud. smol cloud has no GPU machines. Keep CUDA work on smolvm on your own host; use the cloud for CPU workloads, where the same VM model and OCI images carry over.
  • Expecting checkpoints to preserve CUDA state. A .smolcheckpoint captures the machine, not CUDA context. Use live forks on the same host to reuse prepared CUDA state.
  • Mixing up the Vulkan path. Vulkan runs through a paravirtualized GPU (virtio-gpu/Venus), a different mechanism from CUDA remoting, and it is currently unavailable on native Windows.

Frequently Asked Questions

Why does my CUDA guest have no NVIDIA driver or /dev/nvidia* device? Because CUDA is remoted, not passed through. A lightweight shim in the guest forwards CUDA and NVML calls over vsock to a host daemon that owns the NVIDIA GPU. The guest never needs a driver or device nodes, which also means no driver-version drift between guest and host.

Is the GPU being passed through to the VM? No. smolvm does not pass a GPU through to the guest. CUDA runs by API remoting and Vulkan runs through a paravirtualized GPU. If you need direct hardware ownership inside a guest, this is not that model.

Can I run these GPU workloads on smol cloud? No. GPU work runs on smolvm on your own host; smol cloud has no GPU machines. The cloud side is for CPU workloads, where the same VM model, OCI images, and .smolmachine artifacts give you local-to-cloud continuity.

Does forking a machine duplicate the GPU? No. Forks are copy-on-write copies of the running machine, and the GPU stays a single shared resource behind the host daemon. Forking lets many machines reuse prepared CUDA state through a live fork on the same host, but you are sharing one GPU, not multiplying it.

Conclusion

A driverless CUDA guest is a feature, not a fault. When the guest has no NVIDIA driver and no /dev/nvidia* devices yet CUDA calls succeed, API remoting is doing exactly what it should: the guest stays clean and portable while a host daemon does the GPU work. With Smolmachines you get that model out of the box, plus sub-second cold starts, copy-on-write forks for parallel runs, and a single smol interface across local and cloud. Set up your host GPU, declare the machine, and let the remoting layer handle the rest.

Related Articles