Close eight open SDK tasks: the pidfile, the log, cards, names and a live curve

Each was a loose end recorded under `### SDK` in the notepad.

`serve` takes its own pidfile down on SIGTERM. uvicorn restores the handler it
found and re-raises the signal it stopped on, so the default handler ended the
process without unwinding and the `finally` never ran — which is what a stop
sends, and what left `serve.pid` behind.

`serve.log` is cut back past 5 MB by the engine rather than by the screen that
started it, so an adopted engine is bounded too. Gated on its own stdout being
an appended regular file, which is what makes the cut safe: the kernel then
puts the next write at the new end.

Cards are counted from `/dev/nvidia[0-9]*`, so `FLOW_GPUS`/`--gpus` of 0 means
"work it out" the way `FLOW_CPUS` always has. The engine counts, not the
accountant — a remote worker builds one of those from its own inventory, and
detecting there would hand it the engine host's cards. The worker counts last:
what a batch job says it was granted still wins.

`GET /runs/metrics/names` is the distinct over a selection that `--list` and
the terminal's metric picker were approximating by reading the newest run that
had measured anything, which missed a name only an older run ever wrote.

`MetricSink` announces each batch it has written (`run_metric`, carrying the
names). Not a per-point event: one covers up to 500 points or two seconds of
them, and the rows stay the record. The terminal comparison fills in as the
first readings land instead of staying blank until reopened, and the browser
refetches the run and any comparison rather than the list behind them.

`retry --group` pages the list route by `before` instead of stopping at 500.

The terminal dashboard takes the terminal's colours (`ansi-dark`), and the web
UI can re-pair from Settings without disconnecting first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRQ9bmTvCbqCwXo9mxZzzV

backend/fluksio/sdk/cli.py

@@ -999,15 +999,41 @@ def cmd_runs(args: argparse.Namespace) -> int:
     return 0
 
 
-#: The most runs of one group a retry looks at, which is the list route's own
-#: ceiling. ponytail: a sweep larger than this needs paging, not a bigger number.
-MAX_GROUP = 500
+#: One page of the list route, which is its own ceiling. A sweep larger than
+#: this is read a page at a time rather than asked for in one.
+PAGE = 500
 
 #: What a retry leaves alone. Everything else in a group — error, cancelled,
 #: abandoned — is what "the ones that did not make it" means.
 KEPT = frozenset({"ok", "cached", "queued", "running"})
 
 
+def _group_runs(client: Client, group: str) -> Iterator[dict[str, Any]]:
+    """Every run of a sweep, a page at a time.
+
+    Rows come newest first and `before` is the cursor: handing back the last
+    row's own `created_at` reads the next page whatever landed meanwhile, where
+    an offset would shift under a run submitted between two pages.
+
+    # ponytail: `before` is exclusive, so two runs sharing a timestamp exactly
+    # across a page edge would lose one. Ids are minted per run; the history
+    # has never produced a tie.
+    """
+    before = ""
+    while True:
+        page = list(
+            client.runs(
+                limit=PAGE, group=group, **({"before": before} if before else {})
+            )
+        )
+        yield from page
+        if len(page) < PAGE:
+            return
+        before = str(page[-1].get("created_at") or "")
+        if not before:
+            return
+
+
 def cmd_retry(args: argparse.Namespace) -> int:
     """Run the same thing again, one run or a group's unfinished ones."""
     try:
@@ -1016,7 +1042,7 @@ def cmd_retry(args: argparse.Namespace) -> int:
             if args.group:
                 ids += [
                     str(row["id"])
-                    for row in client.runs(limit=MAX_GROUP, group=args.group)
+                    for row in _group_runs(client, args.group)
                     if str(row["status"]) not in KEPT
                 ]
             if not ids:
@@ -1207,39 +1233,15 @@ def _write_rows(rows: list[dict[str, Any]], fmt: str, out: str, hint: str = "")
     return 0
 
 
-#: How many runs `--list` reads before giving up on finding a metric name.
-#: The names belong to the flow's nodes rather than to a run, so the newest
-#: one that recorded any is the whole vocabulary — the rest are for a
-#: selection whose newest runs failed before they measured anything.
-# ponytail: the first run with names wins; a name only an older run recorded
-# is not listed. A `distinct` over the selection would be exact and is a route
-# of its own.
-LIST_SCAN = 10
-
-
-def metric_names(client: Client, ids: Iterable[str]) -> list[str]:
-    """The metric names these runs carry, since a name is flow-qualified.
-
-    `train_loss` is recorded as `train.train_loss`, and asking for the bare
-    one matches nothing — so this is the answer to "what would match".
-    """
-    for run_id in list(ids)[:LIST_SCAN]:
-        names = sorted({point["name"] for point in client.metrics(run_id)})
-        if names:
-            return names
-    return []
-
-
 def _list_names(client: Client, args: argparse.Namespace) -> int:
-    filters = _selection(args)
-    if "until" in filters:
-        # The history spells the same bound `before`, where it is also the
-        # cursor a page is taken from.
-        filters["before"] = filters.pop("until")
-    ids = args.run or [
-        row["id"] for row in client.runs(flow=args.flow, limit=LIST_SCAN, **filters)
-    ]
-    names = metric_names(client, ids)
+    """What `--list` prints: the names a selection recorded, from the engine.
+
+    One `distinct` over the whole selection rather than the newest run that
+    happened to measure anything, so a name only an older run ever wrote is
+    listed too. The selection is the export's own filters, unchanged — this
+    route takes `until` where the history route calls the same bound `before`.
+    """
+    names = client.metric_names(ids=args.run, flow=args.flow or "", **_selection(args))
     _say("\n".join(names) if names else "No metrics recorded by these runs.")
     return 0
 
@@ -1275,13 +1277,15 @@ def _export(
     )
     try:
         with _client_for(args, retries=0) as client:
-            if getattr(args, "list_names", False):
-                return _list_names(client, args)
             try:
+                if getattr(args, "list_names", False):
+                    return _list_names(client, args)
                 rows = fetch(client)
             except ApiError as exc:
                 if exc.status != 404:
                     raise
+                # `--list` asks a route of its own, and it arrived later than
+                # the exports — so an engine without either says the same thing.
                 return _fail(_too_old(client))
     except (SyncError, ApiError) as exc:
         return _fail(str(exc))

backend/fluksio/sdk/client.py

@@ -390,6 +390,23 @@ class Client:
             query["name"] = name
         return self._call("GET", f"/runs/{run_id}/metrics", params=query)
 
+    def metric_names(
+        self, ids: Iterable[str] = (), flow: str = "", **filters: Any
+    ) -> Any:
+        """The metric names a selection of runs recorded, distinct and exact.
+
+        Names are flow-qualified, so `train_loss` is recorded as
+        `train.train_loss` and asking for the bare one matches nothing — this
+        is what says which spellings exist.
+        """
+        query: dict[str, Any] = dict(filters)
+        named = [run_id for run_id in ids if run_id]
+        if named:
+            query["ids"] = ",".join(named)
+        if flow:
+            query["flow"] = flow
+        return self._call("GET", "/runs/metrics/names", params=query)
+
     def compare(self, ids: Iterable[str], metric: str, x: str = "") -> Any:
         """One metric across several runs, in the chart widget's series shape.