Content worker and storage

Import/export requires the API, PostgreSQL and a separate transfer worker. Reading this documentation does not require them. Complete local setup first. Commands below run from the repository root.

Prepare existing installations

pnpm transfers:prepare
pnpm db:migrate
pnpm db:queue

transfers:prepare preserves existing .env settings, generates a dedicated queue credential when missing, and creates a private directory with permission 0700. It never prints passwords. db:migrate applies additive application migrations. db:queue separately provisions the pg-boss schema and restricted queue role; neither API nor worker startup runs migrations.

Keep DATABASE_MIGRATION_URL private. API/worker application queries use DATABASE_URL, whose role is neither a table owner nor BYPASSRLS. QUEUE_DATABASE_URL uses localisync_queue, which has queue-schema permissions and narrowly scoped dispatch/cleanup functions, not direct access to tenant tables.

Run locally

Keep the API and panel running in their existing terminals. Start another terminal:

pnpm releases:prepare
# Configure publishing and start delivery nodes as described below.
pnpm dev:worker

Complete publishing configuration before starting the combined worker. It requires RELEASE_DELIVERY_CONFIG and a permanent RELEASE_STORAGE_DIR.

After building, use pnpm start:worker. The default readiness URL is http://127.0.0.1:4012/health/ready. A queued job remains queued if no worker is running; the panel never claims it completed.

Variable	Purpose
IMAGE_STORAGE_DIR	Separate persistent image store; API/content worker read-write, assistance worker read-only.
TRANSFER_STORAGE_DIR	Absolute private directory shared by API and worker; transfers:prepare resolves it. Never point to a public assets directory.
QUEUE_DATABASE_URL	Restricted queue connection.
QUEUE_DATABASE_PASSWORD	Same generated queue password, used by the optional local Docker worker configuration.
TRANSFER_CONCURRENCY	Simultaneous transfer jobs; default 2, allowed 1–8.
WORKER_HOST	Health listener; local default 127.0.0.1.
WORKER_HEALTH_PORT	Health listener port; default 4012.

Optional Docker worker

The API image’s worker target runs the same use cases in a separate NestJS process. For a host-run API, the local Compose overlay shares its private directory and numeric host user:

LOCAL_UID=$(id -u) LOCAL_GID=$(id -g) \
  docker compose -f compose.yaml -f compose.distribution.yaml -f compose.worker.yaml up -d --build worker

Run transfers:prepare and both migrations first. Use one local worker mode at a time for predictable development resource use. The overlay mounts persistent storage, runs without Linux capabilities, uses a read-only root filesystem and waits up to 330 seconds for shutdown. The host API and container worker must refer to the same storage contents.

Build targets are explicit:

docker build -f apps/api/Dockerfile --target runtime -t localisync-api .
docker build -f apps/api/Dockerfile --target worker -t localisync-worker .

The migration and queue-migration targets are separate operational jobs. This guide describes local development; public origins, TLS, production credentials, backups and deployment approval remain environment-specific setup work.

Processing and retention

Application transactions create a job and outbox record together. The dispatcher submits only an outbox ID to pg-boss. The worker rechecks current actor/project permission, revision and action before executing. A retry cannot create a second committed import or history event.

Parsing runs in a separate process with a 30-second deadline, a 256 MiB JavaScript heap limit and a 512 MiB RSS guard. Compressed spreadsheet content is bounded at 100 MiB. XML external entities, YAML aliases/explicit tags, spreadsheet formula cells and macros are rejected. Plaintext translations and file bodies are not logged.

Writes complete atomically in private storage. Temporary upload/output files and preview records expire after 24 hours; cleanup runs approximately every minute and retries storage failures. Content-free job summaries expire after 30 days. Translation history and audit are not deleted by transfer cleanup. Keep the worker running for retention processing.

Recover from failures

• Queue unavailable: readiness reports unavailable; repair PostgreSQL or credentials and restart the worker. Outbox dispatch resumes.
• Worker stopped during Apply: PostgreSQL rolls back an unfinished transaction. A completed receipt suppresses duplicate application after a retry.
• File permissions/storage failure: restore the configured directory and the API/worker user’s access. Never make it publicly readable to bypass the error.
• Conflict or access changed: obtain current permission/state and regenerate the preview. Old approval cannot authorize changed data.
• Rollback: stop API/worker writers. A prior application version must support the installed schema. The branch migration is a breaking change: use the branch recovery procedure with a verified backup and matching build. Preserve post-backup data for an explicit recovery decision; do not run destructive down migrations or start a branchless writer.

Automated tests use node scripts/e2e-stack.mjs, a separate database/storage directory and isolated Mailpit. They do not use production SMTP.

Branch operations

The same process consumes localisync-branches with concurrency one. db:queue creates this separate queue alongside transfers. Transfers, branches, releases and team notifications must be healthy for worker readiness. Branch operations use atomic outbox delivery, bounded retries, ordered context locks and current membership checks. Processing state is durable, and jobs survive process restarts. Previews expire after 24 hours; job summaries after 30 days. Live or shared ancestry is not expired by time. Unreferenced snapshot leaves are reclaimed in bounded batches. See branch migration and recovery before upgrading an existing installation.

Structured branch events report the job ID, revision, processing duration, retry count and pg-boss queue wait. Queue wait starts at queue insertion and excludes earlier outbox dispatch delay. Failure events use fixed reason codes; comparison summaries report unresolved conflicts and integrity issues. Translation content is not logged. API dependency diagnostics include the request ID, route template and a restricted error code when available, never exception messages or query parameters. Size worker memory from measured content workloads and leave headroom for the independent transfer queue; the serialized snapshot limit is not a process memory limit.

Publication decisions

The localisync-releases queue has concurrency one. Package building runs separately from transfer parsing; release blobs and snapshot roots do not expire with transfer data. Both nodes prepare before a short transaction commits an activation. After commit, narrow queue functions record acknowledgements even if the initiating user loses membership. See publishing setup and recovery.

Operational diagnostics

Monitoring and error references explains safe request references, private logs/traces and failure recovery. Optional SDK diagnostics remain off unless the application explicitly configures a write-only diagnostic key. A reporting outage does not change verified translations or the publication commit boundary.

Translation assistance

Translation assistance runs in a separate process/queue with DB-enforced permits and a private Transit vault. Follow assistance worker setup; do not give the API decryption authority or automatically retry uncertain paid calls.

Team notifications

The combined worker also runs localisync-notifications with one concurrent email sender. Apply application migrations and rerun pnpm db:queue before restarting it. Supply the same validated PANEL_ORIGIN and SMTP settings used by invitation email. Local acceptance uses only Mailpit; production SMTP still requires authenticated TLS. Email is disabled in each account until its user opts in on Notifications.

For existing backup installations, add notifications-worker to the required maintenance participants before resuming backups. Task state checks and mail dispatch pause behind the barrier. Restore quarantine marks pending email as skipped and in-flight delivery as unknown; it does not send a backlog automatically.

Only notification IDs enter the queue. Database records deduplicate events and claim sending durably before SMTP. A crash after that claim produces Email delivery is uncertain rather than another automatic send. Repair the mail configuration for new events; users can still read existing notifications in the panel. Queue, oldest pending age and heartbeat metrics use the fixed notifications label.

Knowledge file previews live in the product database with resource RLS. The worker expires at most 100 overdue previews per pass behind its maintenance barrier; input files are processed in a bounded child process with no application credentials. No new filesystem volume is needed for CSV/TMX preview records. They participate in the existing physical database backup and never apply themselves after restore.

Image processing and cleanup run on localisync-images. Before upgrading, add the private image mount and backup inventory described in private image storage.