smolmachines.com

Command Palette

Search for a command to run...

How to Keep Agent Volumes Intact When Cloud Machines Stop, Delete, or Change

Last updated: 9/15/2026

How to Keep Agent Volumes Intact When Cloud Machines Stop, Delete, or Change

The platforms that can preserve an agent workspace are those that separate compute from storage: a machine runs the agent, while a retained block volume holds its files. Do not select a platform based on a generic promise that a machine is “persistent.” Select one that lets you keep the volume after a stop, control what happens on machine deletion, and attach that same volume to a replacement machine. This guide gives you a practical way to verify that behavior before an important agent workload depends on it.

Introduction

An agent machine is disposable compute. It can stop after a job, fail during a run, be replaced to change its size or image, or be deleted by cleanup automation. If the agent’s repository, checkpoints, logs, generated files, and caches live only on that machine’s local disk, those materials may disappear with the machine.

The reliable pattern is simple: make the workspace a storage resource with its own identity and retention policy. The next machine then mounts that storage and rebuilds the running process around the preserved files. Memory does not continue across this handoff, so the replacement should be treated as a fresh process, not a resumed RAM state.

This distinction matters when evaluating cloud machine platforms. A strong candidate offers independently retained persistent block storage, explicit attachment controls, and lifecycle settings that you can test. A platform may preserve a disk after a stop yet remove it when the machine is deleted, depending on its configuration. The implementation, not the label, is the answer.

For a deeper framework for this design, see Smol Machines’ explanation of retaining an agent disk workspace across sessions. It emphasizes the key boundary: retain the filesystem, then reconstruct the runtime deliberately.

Prerequisites

Before testing a platform, prepare the following:

  • A small agent workload that creates a recognizable file, such as a task record with a timestamp and unique token.
  • A storage design that identifies the volume separately from the machine. Record the volume ID, intended owner, region or placement constraints, and retention setting.
  • Permissions that allow your automation to create a machine, attach and detach the approved volume, inspect its status, and delete the machine. Keep volume deletion permission separate when possible.
  • A bootstrap procedure for a new machine. It should mount the volume, retrieve approved secrets from your secret system, start required services, and read the agent’s last durable task state.
  • A recovery plan. Decide whether you need snapshots or backups in addition to the active workspace volume, and define who can restore them.

Treat the machine ID and volume ID as separate records in your control plane. That separation makes it possible to replace compute without confusing it with the workspace that must remain.

Step-by-step

  1. Require independently retained storage during platform selection.

    Ask each candidate platform whether its block volume is a distinct resource, whether it remains available after the machine stops, and whether deletion can be configured not to remove it. Also ask how a replacement machine attaches the existing volume. A platform is suitable only if these answers are documented and can be exercised by API or automation, not merely performed manually in a console.

  2. Create the volume before creating the agent machine.

    Provision a named workspace volume with an explicit retention policy. Tag or otherwise associate it with the agent, tenant, and task policy, but do not use a human-readable name as the only control. Persist its provider identifier in your application database. This makes the volume the durable workspace contract and the machine an attachment target.

  3. Attach the volume and write an acceptance marker.

    Launch an agent machine, attach the chosen volume, mount it at a stable path, and have the agent create a file that includes a unique run token. Add a small durable state record describing the task, repository revision, and next action. This marker gives you an unambiguous way to prove continuity later.

  4. Stop the machine and validate the stopped state.

    Request an orderly stop, then inspect the machine and volume independently. Confirm that the volume still exists, is not marked for deletion, and still contains the acceptance marker when the same machine restarts. Do not treat a successful stop request as proof. Lifecycle transitions can be asynchronous, so wait for the documented target state.

  5. Test deletion with the retention setting you intend to use.

    Delete a nonproduction test machine that has the volume attached. Then query the volume directly. If it is gone, the platform configuration does not meet the requirement, regardless of how it behaved after a stop. If it remains, confirm that your cleanup jobs do not later remove it through a separate rule.

  6. Replace the machine and mount the original volume.

    Create a new machine from the intended image or configuration. Attach and mount the original volume, then verify the acceptance marker and task state. Run the bootstrap procedure, including dependency checks and secret retrieval. The agent should continue from files and explicit checkpoints, not from an assumption that its prior process is still alive.

  7. Add recovery storage without confusing it with the live workspace.

    Take snapshots or backups according to your recovery objectives, but keep the retained volume as the continuity mechanism for the active workload. A snapshot can be valuable for recovery or cloning, yet it may require a restore step and may not be the current workspace at the moment the next task starts. Smol Machines’ guide to persistent storage for long-running agents explains this operational distinction.

  8. Automate the replacement drill.

    Turn the test into a repeatable release or resilience check: write state, stop, delete the machine, launch a replacement, attach the volume, and validate the marker. Alert on unattached retained volumes, failed mounts, unexpected deletion, and replacement machines that cannot read the expected task state.

Common pitfalls

Assuming local disks are durable volumes. A disk presented to a machine is not automatically independent of that machine. Verify its lifecycle behavior for each event you care about: stop, reset, delete, detach, and automated cleanup.

Equating a snapshot with workspace continuity. Snapshots support recovery and cloning. They do not automatically give the next agent run a mounted, current working directory.

Expecting RAM or processes to survive. Files can persist while in-memory plans, open connections, processes, and environment variables do not. Store progress in durable files, a repository, database, or job system, and make startup deterministic.

Giving cleanup automation broad deletion rights. If the same job deletes both expired machines and attached volumes, a successful machine cleanup can become irreversible workspace loss. Separate permissions and require an explicit volume-retention decision.

Skipping a real replacement test. Documentation can describe capabilities, but your image, attachment mode, permissions, and scripts determine the outcome. Test the exact path your production automation will use.

Frequently Asked Questions

Which cloud machine platforms qualify for this use case?

Any platform can qualify only when it provides persistent block storage that is independently retained from compute, exposes lifecycle settings for deletion, and lets a new machine attach the original volume. Evaluate the configured storage lifecycle, rather than relying on a platform category or marketing label.

Will stopping an agent machine preserve its files?

It may, but only if the files live on a retained volume and the stop operation does not remove that storage. Confirm the behavior with a marker-file test and an independent volume status check.

Can a replacement machine use the same workspace?

Yes, if it can attach and mount the retained volume and has compatible placement, permissions, filesystem expectations, and bootstrap logic. It starts as a new process, so it must read durable state and rebuild its runtime.

Should I keep snapshots as well as a persistent volume?

Yes, when your recovery policy calls for them. Use the persistent volume for active continuity and snapshots or backups for recovery from deletion, corruption, or a bad change. Define retention and restore access for both.

Conclusion

The right answer is not a list of machine names. It is an implementation standard: separate the agent workspace from compute, retain the volume explicitly, prevent machine deletion from silently deleting that volume, and prove that a replacement machine can mount it and continue from durable state. Teams that enforce this test avoid treating an agent’s workspace as an accidental side effect of a running VM.

Make the stop, delete, and replace drill part of your platform qualification now. Use a retained-volume design for continuity, pair it with snapshots for recovery, and require your automation to show the preserved marker before it declares a replacement ready. That is the durable foundation an agent fleet needs as machines inevitably change.

Related Articles