Retire the GPU workers when a run that held a card finishes

A worker that has run a jax node keeps holding the GPU after the run: XLA
preallocates most of the VRAM at import and never releases it, so the next
process OOMs on preallocation while a warm idle worker sits on the card.
Pools are kept warm on purpose — a library reads its environment at import,
so a warm worker cannot be re-told — but the end of a run is a point where
the memory should go back, and the environments carrying a GPU assignment
are exactly the pools that ran on one. Idle ones go now, busy ones when
they return.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc

backend/fluksio/flow/runs.py

@@ -1190,6 +1190,7 @@ class RunService:
             self._finish(run_id, status, reason, result, duration)
             run.status = status
             self._publish(run, "run_finished")
+            self._release_cards(run)
             # Its values were only ever this run's; nothing reads them once it
             # has a result. On Redis the namespace would expire anyway.
             if state is not None and status != "error":
@@ -1198,6 +1199,29 @@ class RunService:
                 except Exception:
                     logger.warning("Could not clear state of run %s", run_id)
 
+    def _release_cards(self, run: Run) -> None:
+        """Hand a GPU run's device memory back when the run is over.
+
+        The accountant frees the card the moment the node returns, but the
+        worker that ran on it is kept warm and a library that preallocated
+        most of the VRAM never gives it up — so the next process to want the
+        card found it taken by one sitting idle.
+        """
+        # ponytail: retires every CUDA pool rather than the ones this run used,
+        # which needs no bookkeeping — a concurrent GPU run's busy worker only
+        # dies when it returns, which is when its own memory should go back
+        # anyway. What it costs is the warm worker of a *live* flow's GPU node.
+        # Track the pools per run if that ever matters.
+        if not (run.needs or {}).get("gpus"):
+            return
+        pool = getattr(self.controller, "workers", None)
+        if pool is None:
+            return
+        try:
+            pool.retire_gpu_children()
+        except Exception:
+            logger.warning("Could not retire the GPU workers of run %s", run.id)
+
     def _record_node(self, run_id: str, outcome: NodeOutcome) -> None:
         outputs = _cacheable(outcome)
         row = RunNode(