Supervise the loops a flow starts, and give up loudly
A node's subscription, schedule or poll loop was a bare asyncio task: one that raised outside its own retry handling was simply gone, and the node went on being listed as running while nothing listened any more. Those loops now run under a supervisor that restarts them with a growing delay and quarantines a flow that burns through five restarts in five minutes — a flow crash-looping every second is worse than one that is visibly stopped, and the dashboard can now say which. The MQTT subscription loses its private five-second reconnect in the process: one backoff policy per socket, and it belongs to the supervisor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
This commit is contained in:
@@ -76,6 +76,10 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend – M
|
||||
reachability and fails the container healthcheck, so a wedged engine is restarted
|
||||
rather than counted as up. One engine per deployment — the API image runs a single
|
||||
worker, because a second one would be a second engine
|
||||
- [x] Supervised background tasks: a node's subscription, schedule or poll loop is
|
||||
restarted with growing delay when it dies, and a flow that spends its failure
|
||||
budget is quarantined and surfaced rather than left crash-looping. The loops
|
||||
themselves no longer carry private retry logic
|
||||
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
|
||||
deployment on failure
|
||||
- [ ] User management scoped per flow and per data set
|
||||
|
||||
Reference in New Issue
Block a user