Supervise the engine's host: deep health, loop watchdog, one worker

The API image ran four uvicorn workers, and each one built a full flow
controller — four sets of MQTT subscriptions, cron ticks and webhooks.
Runs one worker now; scaling out is the worker split, not more processes.

Adds a loop-lag watchdog and a deep /utils/health/ that fails when the
event loop is wedged or Redis is unreachable, the two failure modes a
process-alive check never sees. Autoheal restarts on that signal, behind
a compose profile because it mounts the Docker socket.

The private user-seeding routes now need an explicit opt-in rather than
just ENVIRONMENT=local, so a deployment that kept the default never
exposes them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
This commit is contained in:
root
2026-08-16 07:14:28 +02:00
co-authored by Claude Fable 5
parent 8d82d6c4ec
commit 5462842b8a
16 changed files with 325 additions and 13 deletions
+27 -1
View File
@@ -154,16 +154,20 @@ services:
volumes:
- app-flow-data:/data
# Deep health: fails when the event loop is wedged or Redis is gone, not
# just when the process is dead. Autoheal restarts on unhealthy.
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/utils/health-check/"]
test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/utils/health/"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
build:
context: ..
dockerfile: backend/Dockerfile
labels:
- autoheal=true
- traefik.enable=true
- traefik.docker.network=proxy
- traefik.constraint-label=proxy
@@ -212,6 +216,28 @@ services:
- traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.middlewares=https-redirect
# Docker never restarts a merely *unhealthy* container on its own; autoheal
# closes that gap for the services labeled autoheal=true.
#
# Behind a profile because it needs the Docker socket, which is host-wide
# authority: on a machine that runs anything besides this stack, that is a
# deliberate operator decision. `make up` opts in; the dev stacks do not.
autoheal:
image: willfarrell/autoheal:latest
container_name: fluksio-autoheal
profiles: ["autoheal"]
restart: always
security_opt:
- no-new-privileges:true
networks:
- default
environment:
# Scoped by label, so it only ever restarts this stack's backend.
- AUTOHEAL_CONTAINER_LABEL=autoheal
- AUTOHEAL_INTERVAL=15
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
app-db-data:
app-redis-data: