Monitoring and error references

Localisync provides structured errors, request references, metrics, traces and an operator-only Grafana dashboard. Customer roles never grant access to Grafana. Public-site analytics is separate from operational monitoring: the landing and documentation sites load it only with explicit visitor permission and a reviewed deployment configuration. See Website privacy setup.

Before you start

An infrastructure operator needs Docker, the repository’s Node/pnpm versions, private storage and access to the application hosts. Apply the additive observability migration with the existing migration command while old API/worker processes are stopped. It adds diagnostic credentials, bounded rate-limit buckets, outbox trace context and numeric monitoring helpers; it does not store raw error events in the product database.

The local configuration uses loopback ports and isolated Mailpit. Production requires a separate monitoring host, explicit public/private addresses, a firewall, verified TLS/mTLS, runtime secrets and deployment-specific storage quotas. Local verification is not production approval.

Start the isolated stack

pnpm observability:setup
pnpm observability:check
pnpm observability:up

Open Grafana at http://127.0.0.1:4390. Use the operator account and the generated password in .local/observability/secrets/grafana-password. Do not commit or share that file. Grafana authentication is independent of Keycloak. The operator must provision accounts and rotate the initial password before production use.

Local Prometheus is on 4391, Alertmanager on 4393, Loki on 4394, Tempo on 4395, and Mailpit on 4397. These ports bind only to loopback. The OTLP HTTP endpoint is http://127.0.0.1:4398. Only Mailpit receives local alarm messages.

Local setup also uses OpenSSL to create a private, 30-day development CA and client/server certificates under .local/observability/tls. It never installs a system trust certificate. The additional https://127.0.0.1:4399 Collector listener requires those client certificates, allowing compiled production-mode application processes to keep their mTLS checks enabled locally. Expired certificates produce an explicit setup error; stop the affected services, rotate that local TLS directory, rerun setup and restart the processes. Production generation removes this development listener and uses the operator’s production certificates.

After building, start the host API and panel in separate terminals with pnpm observability:run api and pnpm observability:run panel. Stop any prior process on their ports first. The launchers load the generated private process.env; they do not change your existing .env. To connect the local container worker and delivery nodes, add -f .local/observability/applications.compose.json after the existing application Compose files. This override mounts only the client TLS directory and loads the generated container environment. SDK reporting remains disabled until an application explicitly opts in.

Configure each application process before restarting it:

Setting	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	Private Collector origin; leave unset to disable export.
OBS_ALLOW_LOCAL_HTTP=true	Explicit development-only loopback exception.
OBS_METRICS_TOKEN	Runtime value from the generated metrics-token secret, shared with the private scraper.
OBS_INTERNAL_TOKEN	Runtime value from the internal-token secret, shared only by the panel and API. Public headers are not trusted.
LOCALISYNC_BUILD_ID	Stable release identifier for logs, traces and private source maps.
LOCALISYNC_NODE	Distribution identity: primary or secondary. Set explicitly on separate production hosts; local launchers already set it.
OBS_TRACE_RATIO	Root-trace sampling ratio, from 0 to 1; default 0.1.
PANEL_METRICS_PORT	Dedicated Node metrics listener, normally 3012 locally.
PANEL_METRICS_HOST	Loopback by default. An isolated container network may require an explicit reachable bind.

The API and distribution expose protected /metrics routes; the worker uses its existing health listener. Health endpoints distinguish liveness from readiness. API database readiness is refreshed in the background instead of querying on every probe. Monitoring outages do not change readiness.

Find an error

1. Ask for Copy reference or the request’s X-Request-ID, plus the approximate time. Never ask for cookies, tokens or translation content.
2. Open the Localisync operations dashboard and select the service and time range.
3. In Loki, filter structured JSON by requestId; use fingerprint to group repeated failures and build/clientBuild to distinguish releases.
4. Follow the trace link to Tempo. Trusted internal requests and persisted outbox context connect work across a worker restart. Separate attempts have separate spans.
5. For a panel frame, use the matching private source-map artifact described below.

Expected validation, sign-in, permission, not-found, conflict and rate-limit responses keep their existing behavior. They are not application crashes. Unexpected defects return 500; recognized unavailable dependencies return 503. Retrying a write must preserve its idempotency key and version; never replay an already committed publication as a new operation.

Data and retention

Events contain allowlisted service/build/operation/codes and sanitized code locations. Raw exception messages, stack headers, URLs with queries, SQL parameters, email addresses, credentials, translation values and ICU arguments are excluded. Client reports are untrusted and cannot trigger automatic rollback.

Defaults are logs 30 days, traces 7 days, and metrics 90 days. Operators can change Loki retention_period, Tempo block_retention and Prometheus --storage.tsdb.retention.time in their generated configuration. Metrics also have a 20 GiB size ceiling locally. Apply filesystem quotas and alerts to every persistent volume, including Collector storage; retention alone is not a disk-capacity guarantee. Delivery artifacts and translation data have separate volumes and retention rules.

Application logs are buffered up to 512 events / 512 KiB; traces use a bounded 1,024-span queue. Collector exporters use 256-entry persistent queues with bounded retries. Overflow and delivery failures are counted. A Collector receipt is not a guarantee of final durable storage. There is no unlimited or lossless telemetry promise.

Private panel source maps

The production webpack build emits hidden source maps. The standalone preparation command removes browser .map files from public assets and stores them under .local/panel-source-maps/<build-id>/ with a build manifest. Archive this directory in restricted operator storage before removing a build workspace. Do not publish it through the panel, landing or documentation servers. Client events carry the same build identifier.

Production configuration and watchdog

pnpm observability:production -- /path/to/operator.json validates and generates deployment files without starting services or sending email. The input supplies monitoring/docs origins, all health/metrics targets, Collector bind, separate server/client mTLS certificates, SMTP settings and secret file paths. Run local setup first to create the configuration template. Production application export requires OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE and OTEL_EXPORTER_OTLP_CLIENT_KEY files. TLS verification cannot be disabled.

The generated Collector runs as UID 10001. Provision access to its storage and mounted certificates explicitly. Backends remain private. Protect remote metrics endpoints with a verified mTLS reverse proxy and the metrics bearer token. Protect Grafana with HTTPS and operator authentication. Never expose Loki, Tempo, Collector or Prometheus as customer APIs.

Run pnpm observability:watchdog on a separate delivery host with OBS_WATCHDOG_TARGET, OBS_WATCHDOG_ALERT_CONFIG and OBS_WATCHDOG_SECRETS. Its Alertmanager needs independently reachable SMTP and its own durable storage. It must not depend on the monitoring host’s Mailpit or network. Production notifications go to info@localisync.com; test configurations must remain isolated.

Use alarm runbooks for failures and SDK diagnostics for optional application reporting. pnpm observability:down stops the local stack without deleting its volumes.

Deployment file ownership

Generated production configuration files are readable without containing secret values. Before startup, assign the secrets directory and its files to group 10001 (directory mode 0750, file mode 0440); the generated services receive this supplementary group. Assign the private collector-data directory to UID/GID 10001 (mode 0700). Keep the enclosing operator directory private. Configure filesystem quotas for backend volumes. Review generated files before starting them; generation does not contact SMTP or deploy services.

Use a private mTLS reverse proxy for cross-host metrics scraping. Forward only /metrics to each internal listener, require the monitoring client certificate, preserve the bearer header, and keep the listener itself firewalled. TLS termination must verify certificates; the application metrics token is an additional credential. Public readiness probes use separate paths and credentials from metrics/control traffic.

Translation assistance

Translation assistance adds bounded provider/result metrics, a separate worker heartbeat and queue ages. A provider or vault outage must not make existing translation delivery unavailable. Investigate unknown paid outcomes without automatically retrying them; see assistance recovery.

Coordinated recovery

Follow the backup and recovery guide to capture the database, credential vault and durable release state together. Restore into quarantine, preserve post-backup revocations and require explicit reconciliation before reopening writes.