Publishing and delivery setup

Complete local setup and worker setup first. Commands run from the repository root. Delivery nodes need no PostgreSQL, Keycloak or management API at runtime.

Generate local credentials

pnpm releases:prepare
pnpm build:delivery

The generator creates a private .local/release-config directory, one Ed25519 signing key, a public trust map, distinct node control credentials and host/container worker configuration files. Re-running preserves existing secrets. It prints paths, never credentials. Set these in your private root .env using the absolute paths printed by the command:

RELEASE_DELIVERY_CONFIG=/absolute/path/to/.local/release-config/worker.json
RELEASE_STORAGE_DIR=/absolute/path/to/.local/releases

The signing key is read only by the publishing worker. Delivery nodes receive trust.json and their own control credential, never the signing key. Keep generated files out of Git and public assets. Test environments use separate directories and keys.

Migrate an existing installation

1. Stop API and worker writers, including the old branch snapshot cleanup worker. Keep delivery nodes serving existing packages.
2. Back up PostgreSQL, the permanent release store, and both nodes’ durable state and blobs as one coordinated recovery point. Record the application build, trust map and maximum channel/policy revisions. Protect private keys separately.
3. Record tenant/content row counts and content hashes. Apply pnpm db:migrate and pnpm db:queue as separate operations using the migration role.
4. Verify unchanged existing identities, memberships, branches, translations and histories. The publishing migration adds tables and retention roots; it does not rewrite customer content.
5. Start the new API/worker build and verify all three queues and both delivery nodes. Do not restart an old snapshot cleanup process against the new data.

Migration does not run on application startup. Application and queue roles remain restricted; neither is a table owner or BYPASSRLS. The publishing queue runs with concurrency one and carries only an outbox ID.

Run two local nodes

In separate terminals:

pnpm dev:delivery:primary
pnpm dev:delivery:secondary
pnpm dev:worker

The independent loopback nodes use ports 4021 and 4022, separate persistent directories and explicit development HTTP. The worker configuration points to these control addresses. The common protocol verifier can use either node for reads.

For Nginx in front of two container nodes:

LOCAL_UID=$(id -u) LOCAL_GID=$(id -g) \
  docker compose -f compose.distribution.yaml up -d --build

Stop host-run distribution processes before using the same ports in Docker. Public read addresses are http://127.0.0.1:4031 and http://127.0.0.1:4032. Control ports 4021/4022 remain loopback-only. Each node has a different private persistent directory. Do not mount the primary directory into the secondary.

To run the container worker with the existing local database:

LOCAL_UID=$(id -u) LOCAL_GID=$(id -g) \
  docker compose -f compose.yaml -f compose.distribution.yaml \
    -f compose.worker.yaml up -d --build worker

The worker uses worker-container.json and the isolated control network. Only one local worker mode should run at a time. The combined worker readiness endpoint requires transfer, branch and release queues to be healthy; node readiness checks its local storage.

Production transport boundary

Real domains, hosts, certificates and production deployment require separate environment setup. Do not expose development control ports publicly.

infra/distribution/nginx.production.conf separates public HTTPS reads on port 443 from management mTLS on 8443. Use a dedicated management client CA, a worker client certificate/key and verified server trust. Bind the Node process to loopback behind the proxy. Nginx overwrites the client-verification header; a public request cannot reach an internal write route or forge that header through the public listener.

Set development: false in the worker configuration and DELIVERY_DEVELOPMENT=false on nodes. Worker node entries require url with HTTPS, tokenFile, caFile, certFile and keyFile. Node settings are DELIVERY_STORAGE_DIR, DELIVERY_TRUST_KEYS_FILE, DELIVERY_CONTROL_TOKEN_FILE, DELIVERY_HOST and DELIVERY_PORT. The trust file is an object mapping trusted signing key IDs to public PEM strings. TLS verification remains enabled.

The worker sends bounded uploads and verifies durable preparation receipts. No database transaction remains open while contacting a node. Read keys and internal control credentials are separate; neither is sent in query strings or logged.

Preparation checks the same catalog revision, immutable package identities, file integrity and resulting durable index as activation. Each node bounds that index at 100 MiB, including retained release access mappings. Exceeding the bound fails preparation before a new activation decision is committed; it does not prune release history. Treat it as an operator capacity limit, not a customer quota.

Package construction runs in a child process with a 90-second deadline, 384 MiB JavaScript heap and 768 MiB sampled resident-memory guard. Input/output and final package size limits are also enforced. The parent worker and its independent transfer/branch queues need additional memory; the 100 MiB content bound is not a process-memory limit. Structured release_build_resources events report duration and observed peak RSS without customer content. Operation events report queue wait, retries, completion and fixed failure reasons.

Recovery and retention

Release packages, referenced snapshots/ancestors, committed publication decisions and audit events are permanent. Transfer cleanup must never touch the release store. Expired publication previews can be removed after 24 hours only when no unfinished operation references them. Do not manually prune blobs using branch age or transfer retention rules.

Before commit, a failed job can be cancelled or retried as allowed by its state. After commit, restore connectivity and retry the recorded decision. Do not decrement a node’s revision, mutate its catalog or erase the decision to bypass a partial activation.

For disaster recovery, stop management writers, restore the coordinated database/package backup into an isolated environment, restore each node’s matching state and blobs, and validate signatures, hashes, revision monotonicity and scoped reads before switching traffic. Preserve post-backup data for reconciliation. An older database backup must not issue a revision below one clients or surviving nodes already accepted. Replay preserved committed decisions, or recover a later coordinated backup; never silently reset client revision state.

There is no destructive downgrade. A previous build can be used only after verifying schema and retention compatibility. Keep delivery available while recovering management services. A same-machine local test cannot establish two-physical-server availability or production readiness.

Coordinated recovery

Follow the backup and recovery guide to capture the database, credential vault and durable release state together. Restore into quarantine, preserve post-backup revocations and require explicit reconciliation before reopening writes.