Supervise the loops a flow starts, and give up loudly

A node's subscription, schedule or poll loop was a bare asyncio task: one
that raised outside its own retry handling was simply gone, and the node
went on being listed as running while nothing listened any more.

Those loops now run under a supervisor that restarts them with a growing
delay and quarantines a flow that burns through five restarts in five
minutes — a flow crash-looping every second is worse than one that is
visibly stopped, and the dashboard can now say which.

The MQTT subscription loses its private five-second reconnect in the
process: one backoff policy per socket, and it belongs to the supervisor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
This commit is contained in:
root
2026-08-16 07:23:02 +02:00
co-authored by Claude Fable 5
parent 5462842b8a
commit e7e48c4f13
10 changed files with 358 additions and 61 deletions
+4
View File
@@ -76,6 +76,10 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend M
reachability and fails the container healthcheck, so a wedged engine is restarted
rather than counted as up. One engine per deployment — the API image runs a single
worker, because a second one would be a second engine
- [x] Supervised background tasks: a node's subscription, schedule or poll loop is
restarted with growing delay when it dies, and a flow that spends its failure
budget is quarantined and surfaced rather than left crash-looping. The loops
themselves no longer carry private retry logic
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
deployment on failure
- [ ] User management scoped per flow and per data set