Backup & restore

What to back up, how to restore it, and how long the estate keeps its own data. This guide expands the restore side that running.md → Volumes and running.md → Retention name but do not walk through.

Three named volumes hold everything worth saving, and they are not equal. One is the estate itself; losing it is total data loss. The other two hold a single generated secret each, and losing either is a recoverable inconvenience — the service mints a new one on next boot. Know which is which before you plan a backup.

The service names and volumes above are the real ones from docker-compose.yml; the commands below use them verbatim.

Backing up pgdata

The database holds no secret (running.md → Where secrets live), but it holds everything else. Take a logical dump with pg_dump inside the running postgres container — it is transactionally consistent without stopping the stack, so web and worker keep serving while it runs.

docker compose exec -T postgres \
  pg_dump -U verge -d verge --clean --if-exists \
  > verge-$(date +%F).sql

• -U verge -d verge are the shipped defaults (POSTGRES_USER / POSTGRES_DB). If you overrode either in .env, substitute your values.
• --clean --if-exists makes the dump self-restoring: it drops each object before recreating it, so a restore does not fight leftover rows.
• -T disables TTY allocation so the redirect captures clean SQL.

For a smaller, faster-restoring artifact use the custom format instead, written to a bind-mounted path so the file lands on the host:

docker compose exec -T postgres \
  pg_dump -U verge -d verge -Fc > verge-$(date +%F).dump

Store the dump off the host. A backup that lives only on the machine it protects is not a backup.

Restoring pgdata

Into the running stack

A plain-SQL dump taken with --clean --if-exists restores straight through psql:

docker compose exec -T postgres \
  psql -U verge -d verge < verge-2026-08-23.sql

For a custom-format (-Fc) dump, pipe it through pg_restore instead:

docker compose exec -T postgres \
  pg_restore -U verge -d verge --clean --if-exists < verge-2026-08-23.dump

web applies goose migrations on boot, so restoring a dump from an older schema and then starting the current image lets web migrate it forward. Restore first, then docker compose up -d.

Into a clean volume

To rebuild from scratch — corruption, a moved host, docker compose down -v — let compose recreate an empty pgdata, then load the dump before anything writes to it:

docker compose down -v          # discards the old pgdata (and both state volumes)
docker compose up -d postgres   # fresh, empty database, health-gated
docker compose exec -T postgres psql -U verge -d verge < verge-2026-08-23.sql
docker compose up -d            # bring web and worker up onto the restored data

docker compose down -v deletes all three volumes, so this path also discards web-state and worker-state. That is usually fine — see below — but if you backed them up, restore them before the first web/worker start.

The two state volumes

Neither state volume is in the database and neither is worth a scheduled backup on its own — each holds one secret the owning service regenerates. Back them up only to avoid the disruption that regeneration causes.

web-state — session signing key. Generated by web on first boot. Lose the volume and web mints a new key: every existing session cookie stops verifying, so every signed-in operator is logged out and signs in again. No data is lost and no reconfiguration is needed.

worker-state — prober SSH private key. Generated by worker at prober provisioning; only the public half ever leaves the instance (prober.md). Lose the volume and worker generates a new keypair — but every prober host still trusts the old public key in its authorized_keys, so pushes fail until you re-provision each vantage and install the new public key on it. This is the one state loss with real operational cost, proportional to how many probers you run.

If you want to spare yourself either, snapshot the volumes while the stack is down. Find their exact names first — Docker prefixes them with the compose project (the directory name unless you set COMPOSE_PROJECT_NAME):

docker volume ls | grep -E 'web-state|worker-state'

Then tar each through a throwaway container (substitute the names you saw):

docker run --rm -v verge-asm_worker-state:/state -v "$PWD:/backup" \
  alpine tar czf /backup/worker-state.tgz -C /state .

Restore is the same command with tar xzf into an empty volume before the owning service starts.

The pre-upgrade backup drill

running.md → Upgrades flags this, and it is the one time a pgdata backup is non-negotiable: web applies new migrations before the new code serves traffic, and a schema change is not always cleanly reversible. Take the dump first, then upgrade:

docker compose exec -T postgres pg_dump -U verge -d verge -Fc \
  > verge-pre-upgrade-$(date +%F).dump
git pull
docker compose up -d --build

If the upgrade misbehaves, you can roll the database back by restoring that dump into a clean volume (see above) and pinning the previous image. The state volumes need no pre-upgrade snapshot — a session re-login and, at worst, prober re-provisioning are recoverable without one.

Retention — how long the estate keeps its own data

Backups protect against loss you did not choose. Retention is loss you do choose: two sweeps inside worker that retire aged rows on a dial you set at Settings (the delivery tab; the form posts to POST /settings/retention, and the whole Settings page is admin-only). Both dials ship at 0 — unbounded — v1 grows the corpus without limit until you turn a dial up. This matters to backups because it decides how much there is to back up, and because it is the only supported way to delete estate data.

The two dials are independent and floored differently:

• Dispatch retention (dispatch_cadence_multiple) — retires expired operational dispatch rows and nothing else; the sweep's data layer exposes no observation, span or batch method, so it structurally cannot touch measured data. Stated as a multiple of the slowest enabled scan's cadence, not a day count. 0 is unbounded; any positive value below 2 cadences is rejected, because below that the coverage layer cannot answer whether the slowest scan ran.
• Observation (evidential) retention (observation_currency_days) — retires only evidential observations, in whole days. An observation is live while it is within the bound of the tightest scan covering its timeline — derivations read the live tier and it is never discarded — and evidential once past that bound, read by no derivation. Only evidential rows are eligible, and only past your dial. 0 is unbounded; a positive value below the tightest bound in force is rejected as a no-op (the whole corpus already outlives it).

A live observation is never retired no matter how low the dial goes: the delete query evaluates each row's own per-timeline bound. The dial only governs how long evidential rows — the record of what was once measured — are kept beyond that.

Checklist — what to back up, how often, how to test a restore

Test the restore, not just the backup. A dump you have never restored is a guess. Periodically:

1. Restore the latest dump into a throwaway stack — a separate directory, so COMPOSE_PROJECT_NAME differs and it gets its own volumes.
2. docker compose up -d and confirm web comes up healthy and the migrations apply cleanly (docker compose logs web).
3. Sign in and spot-check that subjects, observations and spans are present.
4. Tear it down with docker compose down -v.

If step 2 or 3 fails, the backup was not one. Find out on a rehearsal, not during an incident.

See also

• running.md → Volumes and → Retention — the operational digest this guide expands.
• prober.md — how the prober SSH keypair is generated and where the private half stays.
• troubleshooting.md — when a restore or a boot does not go to plan.