Bound the pipeline teardown so a stuck node cannot wedge the controller

A node's stop() and a supervised task's cancellation are both waited on
inside the rebuild lock, and neither had a deadline: an MQTT client whose
broker never acknowledges the disconnect leaves aiomqtt's __aexit__
waiting forever, so reload() never returned and every start, stop or
publish behind it hung until the container was restarted.

Each node now gets five seconds to close and is abandoned after that, and
cancel_all reports what is still running rather than waiting on it — it
also no longer swallows a cancellation aimed at the caller, which used to
make the lock holder unkillable. A rebuild asked for by a request gives up
on the lock after fifteen seconds with RebuildBusy, answered as a 503.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StpRc2C6au1WJ1EUU7fsfu
This commit is contained in:
2026-08-23 16:36:48 +02:00
co-authored by Claude Opus 5
parent e9e13371fb
commit c81d6cb21a
5 changed files with 142 additions and 23 deletions
+18 -11
View File
@@ -15,7 +15,8 @@ Deferring because out of scope is fine, but don't mention deferring than.
- FEAT/UI: we promise testing, but currently don't provide an UI for testing e.g. mock values or probing edge cases of a flow. This should be resolved (in a dedicated session); I'm thinking of a "Labs" page, which allows simulating an installation with all the flows (using their draft states) and which allows injecting values or mocking values based on events in the past
- FEAT/UI: check if PWA (https://whatpwacando.today/) notifications could be used to have a panel sending notifications to the device event bus (or generally using PWA to retrieve e.g. location etc). We could introduce a general concept of having a panel (a device, like a wall panel or a phone where the pwa (dashboard) runs) being effectively a node with various outputs. Then various inputs could trigger actions like authentification (i.e. you get home and get a notification which allows you to authenticate the door unlock), get notified on alarms (native alarm connector) or to query geolocation (check where the user is before turning of all lights) etc
- BUG/UI sync the theme state between panels and installations
- BUG/UI some dashboard widgets (like the color picker) are scrollable; we should make sure that no widgets (except for text widgets or list-related widgets) are scrollable
- INFRA document `make update` in the docs; this command is intended to run as a fire-and-forget command when updating a local installation
- CHORE/UI: the house panels are laid out for 1280x800 — twelve columns, twelve
rows. A chart's fixed chrome is now its title line and legend: the range picker
moved up onto the title and gave back its row, so the budget is nearer forty
@@ -34,16 +35,22 @@ Deferring because out of scope is fine, but don't mention deferring than.
`Segment` now takes an optional `label`; the house dashboard sets one
(`Solar`), and it shows on the next frontend build.
- BUG/FLOW: **a cancelled request can leave `FlowController._lock` held forever.**
Seeding nineteen flows over a client that timed out mid-request left the next
`POST /flows/{name}/start` waiting on the lock indefinitely — ten minutes, until
the container was restarted. A py-spy dump showed *no* thread in the reload path,
so the coroutine that holds it is suspended at an `await` inside `reload()`, most
likely in `_teardown()` awaiting a supervised task's cancellation. Everything else
kept working — health, reads, MQTT — so the engine looked fine and only anything
needing a rebuild hung. Two things worth doing: release the lock on cancellation
(`asyncio.timeout` around the teardown, or a `finally` that cannot be skipped), and
fail a `start` that waits more than a few seconds for the lock rather than hanging.
- BUG/NODE: **an MQTT client can hang forever on the way out.** `MqttNode` builds
`aiomqtt.Client` without a `timeout`, so `Client.__aexit__` waits for the broker's
disconnect acknowledgement with no deadline — `_wait_for(..., timeout=None)` falls
through to `self.timeout`, which is `None` too. A subscriber cancelled while its
socket is dead never finishes unwinding. This was what wedged the rebuild lock;
the teardown now abandons such a task after five seconds rather than waiting on
it, so what is left is the task itself, which is stopped only by the second
cancellation it is sent on the way out. Passing a `timeout` to the client fixes it
at the source, but the same number also bounds `subscribe` and `publish`, so it
wants choosing deliberately.
- BUG/FLOW: `MqttNode.stop_publisher`, `MqttNode.stop_subscription` and
`DelayNode.stop_cron` catch `CancelledError` around the task they have just
cancelled, which swallows a cancellation meant for the caller — the trap
`Supervisor.cancel_all` was just fixed for. Latent rather than live: a supervised
node leaves those handles `None`, so only a node built on its own (a test, a
preview) awaits there.
- BUG/INFRA: **475 zombie `git` processes** in the API container after a seeding
session. `FlowStore._git` uses `subprocess.run`, which reaps its own child — these
are the `git gc --auto` daemons `git commit` spawns, reparented to PID 1 when their