Bound the pipeline teardown so a stuck node cannot wedge the controller

A node's stop() and a supervised task's cancellation are both waited on
inside the rebuild lock, and neither had a deadline: an MQTT client whose
broker never acknowledges the disconnect leaves aiomqtt's __aexit__
waiting forever, so reload() never returned and every start, stop or
publish behind it hung until the container was restarted.

Each node now gets five seconds to close and is abandoned after that, and
cancel_all reports what is still running rather than waiting on it — it
also no longer swallows a cancellation aimed at the caller, which used to
make the lock holder unkillable. A rebuild asked for by a request gives up
on the lock after fifteen seconds with RebuildBusy, answered as a 503.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StpRc2C6au1WJ1EUU7fsfu

backend/fluksio/flow/supervision.py

@@ -29,6 +29,11 @@ BACKOFF = (1.0, 5.0, 30.0, 60.0)
 # recover by being restarted again.
 FAILURE_BUDGET = 5
 FAILURE_WINDOW = 300.0
+# How long a cancelled task gets to notice. A loop that is still waiting after
+# this is not going to stop on its own — a client closing a socket the broker
+# no longer answers on is the case seen in the wild — and the rebuild asking
+# for it must not wait on that forever.
+CANCEL_GRACE = 5.0
 
 TaskFactory = Callable[[], Coroutine[Any, Any, None]]
 
@@ -113,11 +118,28 @@ class Supervisor:
         self._tasks.clear()
         for task in tasks:
             task.cancel()
-        for task in tasks:
-            try:
-                await task
-            except (asyncio.CancelledError, Exception):  # noqa: B014 - shutting down
-                pass
+        if not tasks:
+            return
+        # `wait` hands back what is still going instead of waiting on it, and
+        # lets a cancellation aimed at *this* coroutine through — the
+        # `except CancelledError` it replaces swallowed that, which left
+        # whoever asked for the teardown holding their lock and unkillable.
+        done, pending = await asyncio.wait(tasks, timeout=CANCEL_GRACE)
+        for task in done:
+            if not task.cancelled():
+                # Retrieved so a crash on the way out is not reported at exit;
+                # the supervisor has already said what it was.
+                task.exception()
+        for task in pending:
+            # Cancelled once and still running means its shutdown is waiting on
+            # something that is not answering. A second cancellation interrupts
+            # that wait; whether it takes is no longer the rebuild's problem.
+            task.cancel()
+            logger.warning(
+                "Supervised task '%s' did not stop within %.0fs — abandoned",
+                task.get_name(),
+                CANCEL_GRACE,
+            )
 
     def _publish(self, event: dict[str, Any]) -> None:
         if self._events is not None: