Troubleshooting

Common issues and solutions

The commands below use the production Compose file. Run them from the repository root.

Entra Workload Access Fails

Check that web and every worker received the same WORKLOAD_AUTH_ENABLED=true and valid WORKLOAD_AUTH_PROFILE, then redeploy after environment changes. A successful workspace-read verification does not establish import, deployment, environment-write or execution permission. Check the registered application's enabled state, current revision, target grants and workspace exclusions. See workload troubleshooting for token and worker-admission failures. Keep tokens and secret values out of diagnostic output.

Database Connection Failed

# Check database is running
docker compose -f docker/selfhost/compose.yml ps db

# Test connection
docker compose -f docker/selfhost/compose.yml exec db psql -U postgres -c "SELECT 1"

Verify DATABASE_URL format: postgresql://user:pass@host:5432/database

The published images run with NODE_ENV=production. The connection pool then requires TLS unless the URL ends in ?sslmode=disable. A plaintext PostgreSQL outside the Compose file fails with an SSL error until you add that parameter. The bundled db service works because the Compose file sets the URL for you.

On Kubernetes with PgBouncer, runs that fail with connection errors right after a PgBouncer pod stops (a node drain, scale-down or rollout) were still using that pod when it was killed. Set DATABASE_MAX_LIFETIME_SECONDS below pgbouncer.terminationGracePeriodSeconds; see Connection pooling.

Ollama Models Not Showing

Inside Docker, localhost = the container, not your host machine.

# For host-machine Ollama, use:
OLLAMA_URL=http://host.docker.internal:11434  # macOS/Windows
OLLAMA_URL=http://192.168.1.x:11434           # Linux (use actual IP)

WebSocket/Realtime Not Working

  1. Check NEXT_PUBLIC_SOCKET_URL matches your domain
  2. Verify realtime service is running: docker compose -f docker/selfhost/compose.yml ps realtime
  3. Ensure reverse proxy passes WebSocket upgrades (see Docker guide)

The realtime container exits with Invalid environment configuration for the realtime server in three cases:

  • BETTER_AUTH_SECRET is shorter than 32 characters
  • INTERNAL_API_SECRET is shorter than 32 characters
  • DATABASE_URL is missing

It exits with set APP_URL (preferred) or NEXT_PUBLIC_APP_URL when neither URL is set. BETTER_AUTH_SECRET must equal the app's value. The health endpoint is GET /health on port 3002. Add extra browser origins with ALLOWED_ORIGINS.

502 Bad Gateway

# Check app is running
docker compose -f docker/selfhost/compose.yml ps steelengine
docker compose -f docker/selfhost/compose.yml logs steelengine

# Common causes: out of memory, database not ready

The steelengine container waits for migrations to exit successfully and for realtime to report healthy. A realtime configuration error therefore keeps the app down too. Check ps for the service that is not healthy.

Migration Errors

# View migration logs
docker compose -f docker/selfhost/compose.yml logs migrations

# Run manually
docker compose -f docker/selfhost/compose.yml run --rm migrations

Migrations run from the ghcr.io/steelengineai/migrations image, not from the app container. The app image ships only the compiled server, so bun run db:migrate is not available inside steelengine.

pgvector Not Found

Use the correct PostgreSQL image:

image: pgvector/pgvector:pg17  # NOT postgres:17

Temporal Worker Not Ready

Workflows stay queued and schedules never fire when the Temporal Worker is not polling.

# Ready only when every queue Worker is polling
docker compose -f docker/selfhost/compose.yml exec steelengine-worker curl -fsS http://127.0.0.1:3003/health/ready

docker compose -f docker/selfhost/compose.yml logs steelengine-worker

Match the log message to its cause:

  • Missing required environment variable TEMPORAL_ADDRESS (or TEMPORAL_NAMESPACE, TEMPORAL_BUILD_ID): the variable is empty in .env. The Compose file passes an empty TEMPORAL_BUILD_ID through, which overrides the image default.
  • Temporal authentication requires TEMPORAL_AUTH_ISSUER and exactly one of TEMPORAL_AUTH_JWKS_URL or TEMPORAL_AUTH_JWKS_FILE: set TEMPORAL_AUTH_ISSUER and TEMPORAL_AUTH_JWKS_URL. The Worker has no issuer default. The app defaults to http://localhost:3000. Set the same value on both.
  • Temporal Worker credential broker request failed for <url>: the Worker cannot reach TEMPORAL_WORKER_APP_URL. It must be the app's HTTPS URL as seen from inside the Worker container. A 503 from that endpoint means the app lacks TEMPORAL_WORKER_IDENTITY or TEMPORAL_WORKER_BROKER_SECRET.
  • TEMPORAL_IDENTITY must match the Temporal Worker JWT subject: the Worker's identity and the subject in its token differ. With the broker, keep TEMPORAL_WORKER_IDENTITY (app) equal to TEMPORAL_IDENTITY (Worker). With a static token, issue it for the Worker's identity.
  • Production Temporal Workers require bearer authentication: no credential source is configured. The Compose file enables the broker. Helm needs worker.auth.credentialBroker or worker.auth.apiKeySecret.
  • TEMPORAL_ALLOW_INSECURE_AUTH is incompatible with TEMPORAL_TLS_ENABLED=true: the plaintext escape hatch is on while TLS is on. Remove it outside local development.

TEMPORAL_ADDRESS is host:port with no http:// or grpc:// prefix. If the Worker connects but Temporal rejects workflows for missing search attributes, run bash scripts/temporal-production-preflight.sh. It needs TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, and an operator TEMPORAL_API_KEY. See Temporal.

Added Worker Replicas Get No Work

If Worker logs repeat ResourceExhausted: namespace concurrent poller limit exceeded, the Worker fleet holds more polls than the Temporal server allows for the namespace. Replicas added past that point get little or no work. Lower TEMPORAL_MAX_ACTIVITY_TASK_POLLERS and TEMPORAL_MAX_WORKFLOW_TASK_POLLERS on the Worker, or raise frontend.namespaceCount on the server. See Temporal → Scaling Workers for the arithmetic.

Runs Fail After the Worker Restarts

If scheduled or API runs fail with only Workflow execution failed, check whether the Worker is restarting. On Kubernetes, kubectl describe pod shows Last State: Terminated, Reason: OOMKilled. The container hit its memory limit mid-run, and the runs it held were marked failed afterwards.

Give the Worker more memory and keep NODE_OPTIONS=--max-old-space-size at about 75% of the limit. The rest covers memory outside the JavaScript heap. To stop one Worker taking on more than it can hold, set TEMPORAL_ACTIVITY_MEMORY_TARGET and run more than one replica. See Kubernetes.

Blank Page After Login

  1. Check browser console for errors
  2. Verify NEXT_PUBLIC_APP_URL matches your actual domain
  3. Clear browser cookies and local storage
  4. Check that all services are running: docker compose -f docker/selfhost/compose.yml ps

Windows-Specific Issues

Running from source on Windows:

# Use WSL2 for better compatibility
wsl --install

Line ending issues:

# Configure git to use LF
git config --global core.autocrlf input

Workflow Cannot Reach an Internal Service

An API block or a function-block fetch fails with cannot point to private IP addresses, resolves to a blocked IP address, or must use https:// protocol when the target is on your own network.

Workflow egress is deny-by-default for private addresses. Add the destination to EGRESS_ALLOWLIST on both the app and the Temporal Worker. With production Compose, set it in .env and recreate both containers so they receive the changed value. See Private Network Configuration and Workflow Egress Allowlist.

If the error is resolves outside the allowlisted CIDR pin, the host now resolves to an address outside the range you pinned. Confirm the DNS change is expected before widening the pin.

MCP Cannot Connect to an Internal Server

MCP uses ALLOWED_MCP_DOMAINS, independently of EGRESS_ALLOWLIST. Set the approved hostname on both the app and Worker, then recreate their containers. The containers still need network access to that server.

If the server redirects, use its final HTTP or HTTPS endpoint URL. Put authentication in headers, without credentials embedded in the URL. Every request and reconnect validates and pins DNS results. Metadata and other absolute-deny addresses stay blocked even for an approved hostname; only an explicit localhost or loopback IP URL may reach loopback. See Allowed Server Domains.

View Logs

# All services
docker compose -f docker/selfhost/compose.yml logs -f

# Specific service
docker compose -f docker/selfhost/compose.yml logs -f steelengine
docker compose -f docker/selfhost/compose.yml logs -f steelengine-worker

Getting Help

On this page