Supervise the engine's host: deep health, loop watchdog, one worker

The API image ran four uvicorn workers, and each one built a full flow
controller — four sets of MQTT subscriptions, cron ticks and webhooks.
Runs one worker now; scaling out is the worker split, not more processes.

Adds a loop-lag watchdog and a deep /utils/health/ that fails when the
event loop is wedged or Redis is unreachable, the two failure modes a
process-alive check never sees. Autoheal restarts on that signal, behind
a compose profile because it mounts the Docker socket.

The private user-seeding routes now need an explicit opt-in rather than
just ENVIRONMENT=local, so a deployment that kept the default never
exposes them.
This commit is contained in:
root
2026-08-16 07:14:28 +02:00
parent 6d2e25c2ca
commit 2a224ac94e
16 changed files with 325 additions and 13 deletions
+5
View File
@@ -19,6 +19,7 @@ from app.flow.plugins import load_plugins
from app.flow.secrets import init_secrets
from app.flow.state import MemoryState, RedisState, StateBackend
from app.flow.store import FlowStore
from app.flow.watchdog import LoopWatchdog
def custom_generate_unique_id(route: APIRoute) -> str:
@@ -62,6 +63,9 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
fastapi_app=app,
)
app.state.flow_controller = controller
watchdog = LoopWatchdog(event_bus)
app.state.watchdog = watchdog
watchdog_task = asyncio.create_task(watchdog.run(), name="loop-watchdog")
await controller.start()
try:
# A mounted sub-app gets no lifespan of its own, so the MCP session
@@ -69,6 +73,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
async with _mcp_sessions():
yield
finally:
watchdog_task.cancel()
await controller.stop()
if settings.MCP_ENABLED:
from app.mcp.http import aclose