Docker

Deploy SteelEngine with Docker Compose

For Azure DevOps pipeline authentication, follow Entra workload identities. The Compose stack forwards WORKLOAD_AUTH_ENABLED and WORKLOAD_AUTH_PROFILE from .env to web and the worker. Configure organization identities and grants in the application after installation trust is enabled.

Quick Start

The production Compose file is docker/selfhost/compose.yml. It pulls prebuilt images from ghcr.io/steelengineai and connects to a Temporal server that you run separately. Set up Temporal first. Then write the .env from 1. Configure Environment and start the stack.

# Clone, sign in, configure, start
git clone https://github.com/steelengineai/steelengine.git && cd steelengine
docker login ghcr.io
# Write .env (see Production Setup below)
docker compose -f docker/selfhost/compose.yml up -d

Open http://localhost:3000

docker compose up fails immediately when TEMPORAL_WORKER_BROKER_SECRET or TEMPORAL_WORKER_APP_URL is missing. The steelengine-worker container restarts until TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_AUTH_ISSUER, TEMPORAL_AUTH_JWKS_URL, and TEMPORAL_BUILD_ID are set. There is no working bare up -d.

Production Setup

1. Configure Environment

# Generate secrets
cat > .env << EOF
BETTER_AUTH_SECRET=$(openssl rand -hex 32)
ENCRYPTION_KEY=$(openssl rand -hex 32)
INTERNAL_API_SECRET=$(openssl rand -hex 32)
TEMPORAL_WORKER_BROKER_SECRET=$(openssl rand -hex 32)
POSTGRES_PASSWORD=$(openssl rand -hex 16)
NEXT_PUBLIC_APP_URL=https://steelengine.yourdomain.com
BETTER_AUTH_URL=https://steelengine.yourdomain.com
NEXT_PUBLIC_SOCKET_URL=https://steelengine.yourdomain.com
TEMPORAL_WORKER_APP_URL=https://steelengine.yourdomain.com
TEMPORAL_ADDRESS=temporal.yourdomain.com:7233
TEMPORAL_NAMESPACE=steelengine-production
TEMPORAL_SERVER_NAME=temporal.yourdomain.com
TEMPORAL_AUTH_ISSUER=https://steelengine.yourdomain.com
TEMPORAL_AUTH_JWKS_URL=https://steelengine.yourdomain.com/api/auth/jwks
TEMPORAL_BUILD_ID=<image digest or git sha>
EOF

The Compose file builds DATABASE_URL from POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB. The defaults are postgres, postgres, and steelengine. It always points at the bundled db service, so a DATABASE_URL in .env has no effect. To use an external database, edit the Compose file or deploy with Helm.

TEMPORAL_WORKER_APP_URL must be an HTTPS URL reachable from the Worker. The Worker authenticates to that web endpoint with the normal internal-service token plus a separate, purpose-bound proof signed by TEMPORAL_WORKER_BROKER_SECRET. Use different random values for the two secrets. Only web and Worker receive the broker secret. Production Compose does not accept a static TEMPORAL_API_KEY for the Worker.

The TEMPORAL_* settings describe the server you set up on the Temporal page:

  • TEMPORAL_ADDRESS is host:port with no scheme. TEMPORAL_TLS_ENABLED defaults to true. TEMPORAL_SERVER_NAME is the TLS name on the server certificate.
  • TEMPORAL_AUTH_ISSUER is the issuer the app writes into every Temporal token. The app defaults it to http://localhost:3000 and the Worker has no default. Set it to your app URL and give the Temporal server the same value.
  • TEMPORAL_AUTH_JWKS_URL is the app's JWKS endpoint, <app URL>/api/auth/jwks. The Worker and the Temporal server both verify tokens against it, so both must reach it.
  • TEMPORAL_BUILD_ID identifies the Worker build to Temporal. Change it whenever you pull a new worker image.

2. Start Services

Sign in to the registry with the credentials from your subscription, then start the stack.

docker login ghcr.io
docker compose -f docker/selfhost/compose.yml up -d

3. Set Up SSL

Caddy automatically handles SSL certificates.

# Install Caddy
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install caddy

Create /etc/caddy/Caddyfile:

steelengine.yourdomain.com {
    reverse_proxy localhost:3000

    handle /socket.io/* {
        reverse_proxy localhost:3002
    }
}
sudo systemctl restart caddy

Temporal Worker

The steelengine-worker service runs ghcr.io/steelengineai/worker with a 4 GB memory limit. It executes workflows, schedules, knowledge connector syncs, webhook ingestion, notifications, document processing, and platform jobs. It polls eight Temporal Task Queues over one connection. It publishes no ports. Its health server listens on port 3003 inside the container at /health/live and /health/ready. It also serves the backlog of its Task Queues at /metrics/task-queue-backlog, which the Helm chart's autoscaler reads.

Scheduled Sub Agent tasks also execute their complete assistant loop and tool calls in this container. The app and Worker images include the shared assistant-runtime, platform-tools, and server-email packages. When building custom images, include the Worker's workspace dependencies as well as the web app's dependencies.

Give the Worker access to the model providers, integration services, and storage used by your tasks. Workspace and organization BYOK keys are resolved at run time; provider keys supplied through environment variables must also be available to the Worker. The Worker still contacts the app for scoped credentials when an assistant starts a Temporal workflow. See Scheduled Sub Agents.

# Confirm the Worker is polling
docker compose -f docker/selfhost/compose.yml exec steelengine-worker curl -fsS http://127.0.0.1:3003/health/ready

The Worker reads the same ENCRYPTION_KEY, API_ENCRYPTION_KEY, INTERNAL_API_SECRET, and EGRESS_ALLOWLIST as the app. Keep those values identical. These variables set how many Activities each queue runs at once (defaults in parentheses):

VariableQueue
TEMPORAL_MAX_CONCURRENT_ACTIVITIES (20), TEMPORAL_MAX_CONCURRENT_WORKFLOW_TASKS (20)Workflows
TEMPORAL_KNOWLEDGE_CONNECTOR_MAX_CONCURRENT_ACTIVITIES (5)Knowledge connector syncs
TEMPORAL_WEBHOOK_INGESTION_MAX_CONCURRENT_ACTIVITIES (30)Webhook ingestion
TEMPORAL_NOTIFICATION_MAX_CONCURRENT_ACTIVITIES (10)Notifications
TEMPORAL_DOCUMENT_PROCESSING_MAX_CONCURRENT_ACTIVITIES (20)Document processing
TEMPORAL_COPY_BACKFILL_MAX_CONCURRENT_ACTIVITIES (5)Copy backfill
TEMPORAL_SCHEDULED_TASKS_MAX_CONCURRENT_ACTIVITIES (20)Scheduled tasks
TEMPORAL_STEELENGINE_PLATFORM_MAX_CONCURRENT_ACTIVITIES (10)Source control and platform jobs

Set TEMPORAL_ACTIVITY_MEMORY_TARGET to a fraction greater than 0 and less than 1, such as 0.85, to limit new Activity reservations by memory as well. At or above that fraction of the container's memory limit, each queue pauses new slot reservations above a minimum of one until usage drops. Already-reserved slots and running Activities are retained. Work that has not been picked up stays on the Temporal queue, where another Worker replica can take it. This needs a readable container memory limit. Without one, or when unset, only the counts above apply.

Each queue Worker also keeps up to TEMPORAL_MAX_ACTIVITY_TASK_POLLERS Activity Task polls and TEMPORAL_MAX_WORKFLOW_TASK_POLLERS Workflow Task polls open on the Temporal server. The defaults suit one or a few Worker replicas. Lower them before running many replicas, because the server limits the polls a namespace may hold. See Temporal → Scaling Workers.

The Worker's database pool defaults to DATABASE_POOL_MAX=25 with a 60,000 ms statement timeout. The app uses 20 and 30,000 ms. The Worker needs more connections because it runs Activities for every queue at once. Set OTEL_EXPORTER_OTLP_ENDPOINT to export traces and metrics over OTLP. HYPERDX_API_KEY and OTEL_EXPORTER_OTLP_HEADERS carry the collector credentials.

Private Network Configuration

Set deployment policies in the Compose .env file, then recreate the app and Worker containers:

EGRESS_ALLOWLIST=https://erp.corp.internal:8443=10.0.42.0/24
ALLOWED_MCP_DOMAINS=mcp.corp.internal
# Optional: when API and webhook callers use a different host from the app.
PUBLIC_API_URL=https://api.example.com

The production Compose file passes both policies to the app and Worker. EGRESS_ALLOWLIST matches a workflow HTTP destination's scheme, host, and port; the optional CIDR pin restricts its resolved address. ALLOWED_MCP_DOMAINS separately permits MCP hostnames, including approved private destinations. Neither setting changes network routing or firewalls. See Workflow Egress Allowlist and Allowed Server Domains for the policy limits.

PUBLIC_API_URL defaults to NEXT_PUBLIC_APP_URL. Compose gives the app the same effective value as NEXT_PUBLIC_API_URL and passes PUBLIC_API_URL to the Worker. Keep TEMPORAL_WORKER_APP_URL as the separate HTTPS address the Worker uses to contact the app, and keep the Temporal issuer and JWKS settings consistent with your Temporal server.

docker compose -f docker/selfhost/compose.yml up -d --force-recreate steelengine steelengine-worker

Verify an approved and an unapproved private destination through both editor and deployed execution, and inspect the egress audit log. Test MCP connection and tool execution using the final endpoint URL; redirects are rejected. Incoming webhooks still need a reachable public endpoint.

Ollama

Point the stack at an Ollama server the containers can reach. OLLAMA_URL defaults to http://localhost:11434. Inside the container that address is the container itself, so you must override it.

docker/selfhost/compose.ollama.yml bundles an Ollama container with the app, realtime, migrations, and PostgreSQL. It has no steelengine-worker service and builds every image from source. A stack started from it cannot run workflows. Use the production Compose file and point it at Ollama instead.

# macOS/Windows
OLLAMA_URL=http://host.docker.internal:11434 docker compose -f docker/selfhost/compose.yml up -d

# Linux - use your host IP
OLLAMA_URL=http://192.168.1.100:11434 docker compose -f docker/selfhost/compose.yml up -d

Inside Docker, localhost refers to the container, not your host. Use host.docker.internal or your host's IP. The production Compose file does not define host.docker.internal. On Linux Docker Engine, use the host IP.

Laya Decision

The Laya Decision block needs the Laya decision server, which you run next to the stack. Until the app and the Temporal Worker both have LAYA_URL, the block stays hidden from the toolbar, search, and Mothership, and Laya does not appear in the model controls.

docker/selfhost/compose.laya.yml adds the server and wires both services to it:

export LAYA_API_KEY=$(openssl rand -hex 24)
docker compose -f docker/selfhost/compose.yml -f docker/selfhost/compose.laya.yml up -d

The server is CPU only. Reserve about 1 vCPU and 4 GB of RAM; it loads two models (English and multilingual, about 3 GB) at startup, which takes a minute or two. The image (ghcr.io/steelengineai/laya, tagged by laya version and model commit, override with LAYA_IMAGE_TAG) contains the models and makes no network calls, so the server needs no outbound access. Keep LAYA_API_KEY identical on all three services.

Once it runs, organization admins can still turn Laya off under Organization Settings → Models, and workspaces under Settings → BYOK → Workspace model overrides, like any model.

Workflow Checkpoint Upgrades

Checkpoint state V4 reduces repeated workflow payload storage. In releases containing V4 support, migration 0288 prepares the database but leaves the write version unchanged. An operator activates V4 once per database after upgrading all application readers and workers, including any older draining processes.

Use docs/runbooks/workflow-state-performance.md from your compatible release checkout or request it with your dedicated-installation release package. The repository runbook includes source and Compose commands, database selection, verification, and an operator completion record. Use your installation's existing Compose project, environment, overrides, and compatible image tags.

After activation, rollback builds must still support V4; lowering the gate is not a supported rollback. Existing V2/V3 execution chains retain their version, and historical checkpoints are not converted. The always-on cloning improvements do not require activation.

Commands

# View logs
docker compose -f docker/selfhost/compose.yml logs -f steelengine
docker compose -f docker/selfhost/compose.yml logs -f steelengine-worker

# Stop
docker compose -f docker/selfhost/compose.yml down

# Update
docker compose -f docker/selfhost/compose.yml pull && docker compose -f docker/selfhost/compose.yml up -d

# Re-run migrations by hand
docker compose -f docker/selfhost/compose.yml run --rm migrations

# Backup database
docker compose -f docker/selfhost/compose.yml exec db pg_dump -U postgres steelengine > backup.sql

Common Questions

Five services are started: steelengine (main app on port 3000, 8 GB memory limit), steelengine-worker (Temporal Worker, 4 GB memory limit, internal only, health on port 3003), realtime (WebSocket server on port 3002, 1 GB memory limit), db (PostgreSQL 17 with pgvector on port 5432), and migrations (runs once to apply database schema changes, then exits). The Temporal server is not included. The TEMPORAL_* variables in .env point the app and the Worker at the one you run.
You can use either Caddy (recommended, handles certificates automatically) or Nginx with Certbot. Both need to reverse-proxy port 3000 for the main app and port 3002 for WebSocket connections at the /socket.io/ path.
Inside a Docker container, localhost refers to the container itself, not your host machine. On macOS and Windows, use http://host.docker.internal:11434. On Linux, use your host machine's actual IP address (e.g., http://192.168.1.100:11434). The production Compose file does not define host.docker.internal.
docker/selfhost/compose.yml marks TEMPORAL_WORKER_BROKER_SECRET and TEMPORAL_WORKER_APP_URL as required. Compose stops with 'must be at least 32 bytes' or 'must be an HTTPS URL' when either is unset. Add both to .env. The broker secret must differ from INTERNAL_API_SECRET. The URL must be the HTTPS address of the app as seen from the Worker.
Run docker compose -f docker/selfhost/compose.yml pull to fetch the latest images, then docker compose -f docker/selfhost/compose.yml up -d to restart with the new versions. The migrations container will automatically apply any new database schema changes on startup. Set a new TEMPORAL_BUILD_ID whenever the worker image changes.
Back up with: docker compose -f docker/selfhost/compose.yml exec db pg_dump -U postgres steelengine > backup.sql. Restore with: docker compose -f docker/selfhost/compose.yml exec -T db psql -U postgres steelengine < backup.sql. The database data is persisted in a Docker volume named postgres_data.
Yes. docker/selfhost/compose.yml uses environment variable defaults: POSTGRES_USER (default: postgres), POSTGRES_PASSWORD (default: postgres), POSTGRES_DB (default: steelengine), and POSTGRES_PORT (default: 5432). Set these in your .env file to override them. The Compose file assembles DATABASE_URL for every service from these values.

On this page