Supervise the engine's host: deep health, loop watchdog, one worker
The API image ran four uvicorn workers, and each one built a full flow controller — four sets of MQTT subscriptions, cron ticks and webhooks. Runs one worker now; scaling out is the worker split, not more processes. Adds a loop-lag watchdog and a deep /utils/health/ that fails when the event loop is wedged or Redis is unreachable, the two failure modes a process-alive check never sees. Autoheal restarts on that signal, behind a compose profile because it mounts the Docker socket. The private user-seeding routes now need an explicit opt-in rather than just ENVIRONMENT=local, so a deployment that kept the default never exposes them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
This commit is contained in:
@@ -77,6 +77,8 @@ services:
|
||||
SMTP_PORT: "1025"
|
||||
SMTP_TLS: "false"
|
||||
EMAILS_FROM_EMAIL: "noreply@fluksio.com"
|
||||
# Test-only user seeding, needed by the Playwright suite. Dev stack only.
|
||||
PRIVATE_API_ENABLED: "true"
|
||||
|
||||
mailcatcher:
|
||||
image: schickling/mailcatcher
|
||||
|
||||
+27
-1
@@ -154,16 +154,20 @@ services:
|
||||
volumes:
|
||||
- app-flow-data:/data
|
||||
|
||||
# Deep health: fails when the event loop is wedged or Redis is gone, not
|
||||
# just when the process is dead. Autoheal restarts on unhealthy.
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/utils/health-check/"]
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/api/v1/utils/health/"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: backend/Dockerfile
|
||||
labels:
|
||||
- autoheal=true
|
||||
- traefik.enable=true
|
||||
- traefik.docker.network=proxy
|
||||
- traefik.constraint-label=proxy
|
||||
@@ -212,6 +216,28 @@ services:
|
||||
|
||||
- traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.middlewares=https-redirect
|
||||
|
||||
# Docker never restarts a merely *unhealthy* container on its own; autoheal
|
||||
# closes that gap for the services labeled autoheal=true.
|
||||
#
|
||||
# Behind a profile because it needs the Docker socket, which is host-wide
|
||||
# authority: on a machine that runs anything besides this stack, that is a
|
||||
# deliberate operator decision. `make up` opts in; the dev stacks do not.
|
||||
autoheal:
|
||||
image: willfarrell/autoheal:latest
|
||||
container_name: fluksio-autoheal
|
||||
profiles: ["autoheal"]
|
||||
restart: always
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
networks:
|
||||
- default
|
||||
environment:
|
||||
# Scoped by label, so it only ever restarts this stack's backend.
|
||||
- AUTOHEAL_CONTAINER_LABEL=autoheal
|
||||
- AUTOHEAL_INTERVAL=15
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
|
||||
volumes:
|
||||
app-db-data:
|
||||
app-redis-data:
|
||||
|
||||
Reference in New Issue
Block a user