Backup and recovery

Backups are an operator function. Customer workspace roles cannot download database backups, read encryption credentials or administer the credential vault.

The local recovery tools coordinate PostgreSQL (including Keycloak), OpenBao Raft, release artifacts, both delivery nodes, transfer files, private context images and signing/configuration material. Logs, traces and historical metrics are excluded. Monitoring configuration remains in the deployment source.

Prerequisites

Use Node 24, pnpm, Docker, OpenSSL and htpasswd. Run the repository build before using recovery verifiers. Read the repository’s internal verification record before claiming deployment readiness. The supplied setup creates isolated local storage; it does not back up an existing installation automatically or create physical off-site redundancy. Keep recovery credentials outside Git and outside the archives they unlock.

pnpm build
pnpm backup:build
pnpm backup:setup
pnpm backup:check

Setup refuses an existing destination. backup:check validates repository access and WAL archiving; a complete point additionally requires all configured application participants, source directories and the vault snapshot identity. Missing participants are errors, not permission to take an inconsistent backup.

The images inventory source and images-worker maintenance participant are required for versions that support private context images. Existing installations must add the private image mount and update their inventory before the next common recovery point.

The generated private configuration names every source and separate staging, backup, restore and recovery-material directory. Set LOCALISYNC_MAINTENANCE_DIR consistently on the API, all workers, delivery nodes and vault operator commands. Never enable it on only some writers. Configure identity containers for bounded pause in the local test environment. All paths are operator-controlled; they are not accepted from customer APIs.

Run and schedule

pnpm backup:run
pnpm backup:status
pnpm backup:verify
pnpm backup:schedule

The standalone scheduler is independent of PostgreSQL and pg-boss. It requests common points every 30 minutes, full PostgreSQL backups weekly, differential backups daily, daily integrity verification and weekly clean restore drills. The last UTC day of the month also produces an independent monthly repository snapshot. Each monthly payload includes its own encrypted PostgreSQL/WAL dependencies.

Before capture, large files are staged. A lease blocks new management operations and waits for admitted work to finish. The remaining file changes, Raft snapshot, database content inventory and named PostgreSQL restore point are captured under that lease. The maximum lease is 120 seconds. A timeout fails the point; it does not kill a paid request or discard a committed publication. SDK downloads remain available. Remote transfer and repository verification occur after the lease is released.

A point is complete only after verification and a separate durable completion receipt in the encrypted repository. Interrupted uploads and unconfirmed snapshots do not count. The retained target is every point in the last 30 days and one independent archive for each of twelve months. Operator-only retention first checks verification and PostgreSQL dependencies. Source HTTP credentials cannot forget or prune restic snapshots.

Encryption and custody

pgBackRest uses an encrypted repository over mutually authenticated TLS. Restic encrypts files before sending them to an authenticated, append-only TLS endpoint. Append-only access protects this client path; it is not a claim of immutable storage against the backup-server administrator.

Keep backup passwords and the OpenBao 2-of-3 recovery shares separately from the source volumes and encrypted repository. Application containers receive neither root tokens nor recovery shares. The backup AppRole may read a Raft snapshot; it cannot decrypt or rotate Transit keys. Existing vault installations need explicit operator provisioning of this role; ordinary credential rotation does not silently create a privileged identity.

Restore first uses the recorded application and tool versions. It issues fresh local recovery TLS certificates when opening an archive; expired certificates never justify disabling verification. Old AppRole identities may also expire. Renewing these identities is a separate threshold-authorized operator procedure, not a normal backup permission.

Inspect, restore and verify

pnpm backup:restore:plan --point RECOVERY_POINT_UUID
pnpm backup:restore --point RECOVERY_POINT_UUID
pnpm backup:restore:verify --point RECOVERY_POINT_UUID
pnpm backup:drill --point RECOVERY_POINT_UUID

Add --monthly-only to inspect or recover solely from the monthly repository. Recovery points can be discovered from encrypted completion receipts without the source disk’s local catalog. A restore refuses an occupied target. Database roles, RLS, table counts and content hashes, file digests and release signatures are checked. The generic drill checks session/paid-work quarantine, database isolation, each ready image’s size/hash, image RLS, an unsealed vault and signed artifacts. The separate acceptance fixture additionally tests the original vault encryption context, identity login and signed downloads from both recovered nodes.

The keys/bootstrap directory contains recovery control definitions alongside the separately held encryption material. Preserve this directory independently of source volumes. To recover on another host, provision the recorded image digests, mount the encrypted repository and escrow files, and adjust only isolated destination paths and container addresses in the control configuration. Never use backup:setup to initialize over an existing repository. Restored deployment metadata records build/lockfile hashes and image identities; it does not download application images or run migrations.

Restored APIs and workers remain quarantined. Product sessions are revoked; Keycloak realm revocation thresholds are advanced. Incomplete assistance attempts become unknown and queued paid work is stopped without releasing uncertain usage reservations. Committed publication decisions remain intact. No production SMTP, provider request or delivery control action runs as a consequence of restoring a backup.

Verification is not traffic enablement. Review post-backup membership changes, revocations and both surviving delivery nodes. A newer catalog or policy must never be overwritten with an older backup. Missing freshness evidence blocks reopening; creating an empty replacement node does not prove a revoked key is safe to accept. Resolve these differences through an approved recovery review before starting writers or moving traffic.

For the local isolated workflow, prepare a private JSON review file with version: 1, the point UUID, reviewedAt as Unix milliseconds, absolute livePrimary and liveSecondary directory paths, and explicit accessReviewed, revocationsReviewed, publicationDecisionsReviewed and externalEffectsReviewed confirmations. All four must be true. They represent the operator’s completed review, not an automatic proof that no access changes occurred after the backup. Review evidence expires after five minutes. Both live directories must be independent of the restored target.

pnpm backup:restore:reconcile --point RECOVERY_POINT_UUID --review /private/recovery-review.json
pnpm backup:restore:enable --point RECOVERY_POINT_UUID --review /private/recovery-review.json --confirm RECOVERY_POINT_UUID

Enablement repeats verification and reconciliation under the maintenance barrier. It only removes the selected local target’s quarantine file; it does not start workers, switch DNS, route traffic or approve production. A newer signed policy, missing node, stale review or changed catalog blocks the operation. If both original nodes are lost and current revocation/access state cannot be established, leave the target quarantined.

Alarms

Enable the optional backup scrape only after the scheduler is running: BACKUP_MONITORING_ENABLED=true pnpm observability:setup. Local metrics listen on loopback port 4407. Do not expose them publicly; production telemetry uses the separately configured verified transport.

The operations dashboard shows recovery-point age, backup and maintenance duration, WAL archive age, integrity verification and restore-drill results. A failed run warns; 45-minute point age warns and 60-minute age is critical. Heartbeat loss, integrity errors and failed or overdue restore drills require operator attention. Local notifications use Mailpit only.

For an incident: inspect backup:status, preserve the last verified point, check capacity, repository connectivity, vault seal state and participants, then rerun the failed step. Do not remove old points to make a failed backup appear successful. If a coordinator crashes, its lease expires; backup:recover-lock accepts only a proven-dead local process. Never steal a lock merely because a backup is slow.

A stale application participant also blocks capture, including when another instance of the same worker is healthy. Stop or recover the stalled instance and verify its identity before removing stale participation records. Never discard them solely because their heartbeat is old. Restarting the scheduler recovers only expired identity freezes whose container ownership matches the configured isolated environment.

AppRole credentials and TLS certificates still require operator rotation before expiry. For an old archive with expired AppRole SecretIDs, --renew-vault-identities is a separate break-glass recovery option requiring custody of the threshold shares. It is not invoked by the scheduler. Use it only with an explicitly authorized, isolated restore:

pnpm backup:restore --config /private/operator/config.json --point UUID --renew-vault-identities

The operation renews API/worker AppRole credentials in the restored copy, checks login, revokes the temporary root token and verifies that both that token and the temporarily opened recovery endpoint are rejected. Original vault credentials and encrypted archives remain unchanged. The copied client profiles use the recovered vault and fresh TLS trust. This follows OpenBao’s threshold recovery procedure.

Any failed credential replacement, unproven token revocation or endpoint closure stops the isolated vault and leaves recovery quarantined. A timeout is not proof of revocation. If the operator process is forcibly terminated, stop that isolated vault and perform a fresh restore; do not reopen a partly completed privileged recovery. Verification and delivery-state reconciliation are still required before the separate enable step.

Deployment acceptance

One hour of data loss and four hours to recovery are targets. Small local restore timings are not a production guarantee. Before production acceptance, test actual data size, separate backup-host loss domains, credential custody, retention across calendar boundaries, network throughput, restore capacity and the complete traffic-reopening procedure.

The technical mechanisms follow the official pgBackRest retention rules, OpenBao Raft snapshot API and restic retention guidance. Keycloak realm exports are not a replacement for its database backup.