Alarm response

These runbooks are for authorized infrastructure operators. Open the alert’s dashboard and verify its time range before changing a service. All thresholds are configurable operating defaults, not availability promises.

Notifications and maintenance

Alertmanager groups by alert name and service, sends firing and resolved notifications, and repeats unresolved incidents. Critical delivery failure inhibits the per-node warning; critical severity inhibits the same warning condition. A Keycloak incident suppresses related API error-rate notifications, not unrelated queue failures.

Create a time-bounded silence in Alertmanager before planned maintenance. Specify exact matchers, an owner/reason and expiry, then verify it expires. Never remove rules or mute all services permanently. After maintenance, verify probes, fresh logs/traces and one test alarm. Local alarm verification uses Mailpit only.

ServiceDown

Check the indicated readiness URL, process status and the latest deployment. API readiness uses a bounded background database check. A healthy process alone does not establish dependency readiness. Restore the dependency or roll back the application version without deleting data.

DependencyDown

Check Keycloak independently, then its database/network/certificate. Avoid mass session or invitation retries. API HTTP error-rate notifications are inhibited while this dependency incident is active; queue alarms remain visible.

WorkerHeartbeatMissing

Inspect the worker process and private queue database connection. Check all three queue heartbeats. Restart only with the current schema/application version; persisted publication decisions must be completed unchanged.

DeliveryNodeDown

Use the other delivery address and inspect the failed node locally. Check storage, certificate and process readiness. Do not remove healthy immutable artifacts.

BothDeliveryNodesDown

Treat delivery as critical. Check independent node storage/network/TLS. Restore one node first; existing SDK caches continue working. Do not require the management API or Keycloak to recover reads.

HttpErrorRate

At least 20 requests in five minutes and more than 5% server failures triggered this threshold. Use the request reference, safe code and trace; distinguish 500 defects from 503 dependency outages. This threshold is not an SLA.

OutboxStalled

Inspect the queue connection and dispatcher. The oldest undelivered outbox entry exceeded two minutes. Retry delivery through the existing outbox mechanism; do not recreate the user operation.

QueueStalled

A queued job has not started for five minutes. Check worker heartbeat, capacity and retries. Preserve job identity and confirmed preview/version.

PartialActivation

A committed publication or policy lacks all node acknowledgements for over two minutes (critical after five). Restore the failed node and retry the SAME job. Never cancel or reinterpret the committed decision.

DiskSpace

Check the affected filesystem. Warning begins above 85%, critical above 95%. Add capacity or apply documented telemetry retention. Never delete release artifacts or live database files as an emergency log cleanup.

DiskInodes

Check inode consumption and the affected mount. Warning begins above 85%, critical above 95%. Remove only known expired temporary data using its supported cleanup procedure.

CertificateExpiry

Renew and verify the complete certificate chain before expiration. Warning: less than 14 days; critical: less than three days. Rotate mTLS clients and servers without disabling verification.

TelemetryDropped

Inspect application dropped counters, Collector exporter failures, queue capacity and backend storage. Restore the backend and verify new events. Previously dropped data cannot be reconstructed or treated as delivered.

NotificationFailed

Inspect Alertmanager locally and through the independent watchdog. Check SMTP TLS, runtime credentials and recipient policy. An email delivery failure cannot reliably notify through the same broken email path.

CollectorQueuePressure

A persistent exporter queue has stayed above 80% capacity for two minutes. Check Loki/Tempo availability, free disk and export throughput. Restore downstream service before increasing a bounded queue. This warning does not mean events have already been lost; check TelemetryDropped separately. The pinned Collector exposes raw metric names with type/unit suffixes disabled explicitly.

TerminalWorkerError

Inspect the stable error code, job reference, source locations and trace. Validation/conflict results are not unexpected crashes. Retry only when the job state permits it.

ScrapeFailed

Check scraper credentials, mTLS, network route and exporter readiness. A missing scrape is unknown health, not zero errors. Do not disable authentication to make a target green.

ProbeMissing

Restore required health targets and their labels. Verify both delivery nodes are independently configured. Missing time series must not be interpreted as successful service operation.

SyntheticDeliveryFailed

Check the configured synthetic distribution, channel, read key, trusted keys, version and published locale. A missing configuration is not success. Inspect each node separately; signature/hash/scope verification must pass before the gauge becomes 1.

MonitoringServerDown

The independent watchdog could not reach the monitoring server for two minutes. Check the monitoring host, firewall, reverse proxy and persistent volumes from the separate delivery host. Its Alertmanager and SMTP path must remain independent. Recover monitoring without restarting healthy translation delivery.

Signed delivery probe

Run pnpm observability:synthetic with OBS_SYNTHETIC_CONFIG pointing to a private operator JSON file. Specify project/distribution IDs, channel, exactly two addresses, read-key tokenFile, appVersion, locale and trusted public keys. Configure a deliberately small published synthetic release. The separate process verifies each node’s signed catalog, immutable manifest, scopes, hashes and content under a 30-second process deadline. Scrape its protected metrics on port 4396. No configured distribution produces configured=0 and unsuccessful gauges, never a fabricated success.

The probe does not edit customer content. It does not automatically publish, rotate credentials or roll back releases.

Verification and escalation

pnpm observability:test exercises safe OTLP receipt, stored Loki events, Tempo traces and a real local readiness failure followed by Mailpit firing/resolved delivery. Record the environment and results. Production SMTP, firewall, TLS and independent-host failures require their own deployment acceptance test. See monitoring for setup and SDK diagnostics for untrusted client reports.

AggregateCollectionMissing

Numeric database and queue metrics have not been refreshed. Check the worker database connection, the observability migration and queue role permissions. Previous metric values may be stale; do not interpret zero or absent values as healthy queues.

Translation assistance

AssistanceOutcomeUnknown means the provider may have processed and charged a request. Inspect the job reference and retained reservation. Never replay the paid call automatically; the user must approve additional possible cost.

AssistanceVaultUnavailable means encryption/decryption could not complete. Check private TLS, AppRole expiry and seal state using the vault runbook. Do not disable certificate verification or grant the API decryption privileges. Editing and existing delivery should remain available.

AssistanceWorkerHeartbeatMissing checks the separate queue worker. Check its database/queue connection and outbox age. Restarting it must recover a sent attempt as uncertain rather than sending it twice. Provider outcome/duration and vault operation panels use bounded labels without tenant IDs or content.

Team notifications

NotificationWorkerMissing means the combined worker responds but its notification queue has no heartbeat. Check queue migrations, database access and the worker’s safe error references. Keep notifications-worker in backup maintenance participants.

NotificationDeliveryUnknown reports an increase in uncertain SMTP deliveries. Check SMTP reachability and credentials without logging messages or secrets. These records must not be reset to pending: SMTP may already have accepted them. Users can read the inbox while operators repair sending for future notifications. An unchanged historical count does not repeatedly fire this alert. Existing outbox-age and terminal worker-error rules also cover the fixed notifications queue label.

Private images

ImageWorkerMissing detects a missing image heartbeat while the worker service is reachable. ImageProcessingBacklog warns when the oldest queued image remains over five minutes for one minute. Check queue configuration, DB permissions, storage mounts and decoder resource limits. Processing failures also use the bounded images queue result counter. Inspect aggregate localisync_image_storage_bytes (stored/reserved), pending count and queue age; never add workspace IDs as metric labels. Lowering quotas or deleting referenced files is not a queue repair. Resume the worker after verifying current maintenance state.