Supervise the loops a flow starts, and give up loudly

A node's subscription, schedule or poll loop was a bare asyncio task: one
that raised outside its own retry handling was simply gone, and the node
went on being listed as running while nothing listened any more.

Those loops now run under a supervisor that restarts them with a growing
delay and quarantines a flow that burns through five restarts in five
minutes — a flow crash-looping every second is worse than one that is
visibly stopped, and the dashboard can now say which.

The MQTT subscription loses its private five-second reconnect in the
process: one backoff policy per socket, and it belongs to the supervisor.
This commit is contained in:
root
2026-08-16 07:23:02 +02:00
parent 2a224ac94e
commit 3a8edae928
10 changed files with 358 additions and 61 deletions
+4
View File
@@ -76,6 +76,10 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend – M
reachability and fails the container healthcheck, so a wedged engine is restarted
rather than counted as up. One engine per deployment — the API image runs a single
worker, because a second one would be a second engine
- [x] Supervised background tasks: a node's subscription, schedule or poll loop is
restarted with growing delay when it dies, and a flow that spends its failure
budget is quarantined and surfaced rather than left crash-looping. The loops
themselves no longer carry private retry logic
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
deployment on failure
- [ ] User management scoped per flow and per data set