Close eight open SDK tasks: the pidfile, the log, cards, names and a live curve

Each was a loose end recorded under `### SDK` in the notepad.

`serve` takes its own pidfile down on SIGTERM. uvicorn restores the handler it
found and re-raises the signal it stopped on, so the default handler ended the
process without unwinding and the `finally` never ran — which is what a stop
sends, and what left `serve.pid` behind.

`serve.log` is cut back past 5 MB by the engine rather than by the screen that
started it, so an adopted engine is bounded too. Gated on its own stdout being
an appended regular file, which is what makes the cut safe: the kernel then
puts the next write at the new end.

Cards are counted from `/dev/nvidia[0-9]*`, so `FLOW_GPUS`/`--gpus` of 0 means
"work it out" the way `FLOW_CPUS` always has. The engine counts, not the
accountant — a remote worker builds one of those from its own inventory, and
detecting there would hand it the engine host's cards. The worker counts last:
what a batch job says it was granted still wins.

`GET /runs/metrics/names` is the distinct over a selection that `--list` and
the terminal's metric picker were approximating by reading the newest run that
had measured anything, which missed a name only an older run ever wrote.

`MetricSink` announces each batch it has written (`run_metric`, carrying the
names). Not a per-point event: one covers up to 500 points or two seconds of
them, and the rows stay the record. The terminal comparison fills in as the
first readings land instead of staying blank until reopened, and the browser
refetches the run and any comparison rather than the list behind them.

`retry --group` pages the list route by `before` instead of stopping at 500.

The terminal dashboard takes the terminal's colours (`ansi-dark`), and the web
UI can re-pair from Settings without disconnecting first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRQ9bmTvCbqCwXo9mxZzzV

docs/code/api.md

@@ -107,8 +107,10 @@ published to. Flows own the namespace; everything else is a client of it.
 | `GET` | `/runs/overview` | one row per flow that has runs, with how many are running or queued |
 | `GET` | `/runs/export/metrics?…&name=&stride=&format=` | every selected run's series as one long table: `run, name, step, ts, value` |
 | `GET` | `/runs/export/runs?…&params=&metrics=&format=` | one row per run: its inputs as columns, its final numbers, its status and provenance |
+| `GET` | `/runs/metrics/names?…` | every metric name the selected runs recorded, distinct; takes the export's own filters |
 | `GET` | `/runs/{id}` | one run in full: params, result, per-node record, artifacts |
 | `POST` | `/runs/{id}/cancel` | stop it |
+| `POST` | `/runs/{id}/retry` | run the same thing again, as a new run naming this one |
 | `GET` | `/runs/{id}/metrics?name=&stride=` | one metric's series, in step order; every series of the run without `name` |
 | `GET` | `/runs/series/compare?ids=a,b,c&metric=&x=` | that metric across several runs. `x` is what to plot against: nothing or `step`, `time` (seconds since each run's own first reading), or another metric's name, joined on the step the two share |
 

docs/code/cli.md

@@ -77,12 +77,14 @@ Another instance's Fluksio on the port is named, and the move happens as usual.
 | `--max-runs N` | 4 | batch runs driven at once (`FLOW_MAX_RUNS`) |
 | `--max-cascades N` | 4 | cascades in flight at once (`FLOW_MAX_CASCADES`) |
 | `--max-workers N` | 4 | python worker processes (`FLOW_MAX_WORKERS`) |
-| `--gpus N` | 0 | GPUs on this machine a node may be given (`FLOW_GPUS`) |
+| `--gpus N` | counted | GPUs on this machine a node may be given (`FLOW_GPUS`) |
 
-Cards are declared rather than detected, since asking a vendor's tooling would
-make one dependency two, so a machine with a GPU reports none until `--gpus` says
-otherwise, and a node asking for one is clamped to zero and runs alongside
-every other. `--gpus 1` is what serialises them.
+Cards are counted from NVIDIA's device nodes (`/dev/nvidia0`, `/dev/nvidia1`, …),
+which asks no vendor tool and so keeps the one dependency from becoming two.
+Anything they do not cover — another vendor, or a card this process cannot see
+— reports none until `--gpus` says otherwise, and a node asking for one is then
+clamped to zero and runs alongside every other. `--gpus 1` is what serialises
+them. Passing a number always wins over the count.
 
 `--enroll` with `--portal` is the one-command setup: it pairs before the engine
 starts, so the connection is dialled as part of coming up rather than needing a
@@ -147,6 +149,11 @@ which is what lets the screen be closed while the engine keeps running — a
 pipe with nobody reading it breaks the next line the engine writes, and a
 node's `print` is one of those. It also means the Logs tab has the output of
 an engine this screen only adopted, and the scrollback of the one before it.
+The engine cuts the file back to nothing once it passes 5 MB, so the bound
+holds whether or not a screen is open.
+
+The screen takes its colours from the terminal rather than painting its own,
+so it sits inside a light profile as readily as a dark one.
 
 An engine started elsewhere is adopted rather than duplicated, and can be
 stopped from here only when it is this instance's own: both the pidfile
@@ -502,7 +509,8 @@ taken leaf by leaf rather than as one blob.
 
 Metric names are flow-qualified (a node of `train` writing `train_loss` records
 `train.train_loss`) so `--list` prints the names the selected runs
-carry when the spelling is not obvious.
+carry when the spelling is not obvious. The engine answers it over the whole
+selection, so a name only an older run ever recorded is listed too.
 
 Both take `--flow`, `--run ID` (repeat it), `--group`, `--status`, `--since`,
 `--until` and `--local`, and both put the run id on every row: it is the join

docs/code/workers.md

@@ -36,7 +36,7 @@ driver in order to run a training step. An engine host already has it, and
 | `--parallel` | `1` | how many node calls it will take at once |
 | `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |
 | `--cpus` | what the job or the machine has | cores to advertise |
-| `--gpus` | what the job says, else none | GPUs to advertise; never probed |
+| `--gpus` | what the job says, else counted | GPUs to advertise |
 | `--ram-mb` | what the job or the machine has | memory to advertise, in MB |
 | `--max-idle` | never | stop after this many seconds with nothing running |
 
@@ -51,12 +51,15 @@ the engine schedules against it: a node asking for two cores and a GPU goes to
 a machine that has them free, not merely to one carrying the right label.
 
 Cores and memory are read off the machine, or off the batch job that started
-this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **GPUs are never
-probed.** Asking a vendor tool would make the one dependency two, so a GPU is
-something the job says it was given (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`,
-or `FLUKSIO_WORKER_GPUS`) or something you say with `--gpus`. A worker that
-reports nothing still attaches and is scheduled by its label alone, as every
-worker was before any of them reported anything.
+this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **What the job says
+it was given always wins for GPUs** (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`,
+or `FLUKSIO_WORKER_GPUS`): a node with eight cards may have granted this job
+one, and advertising eight would be a lie the scheduler acts on. With nothing
+said, NVIDIA's device nodes are counted, the same as the engine does for its
+own machine, and `--gpus` overrides either. No vendor tool is asked, which is
+what keeps the one dependency from becoming two. A worker that reports nothing
+still attaches and is scheduled by its label alone, as every worker was before
+any of them reported anything.
 
 The engine tells each call what it may use: thread caps, and the devices it
 may see. The worker starts a process per call, so it applies them at the only

docs/getting-started/data-science.md

@@ -298,10 +298,11 @@ nothing else is given that card while it runs.
 string whose contents depend on the version you have installed, so writing it
 for you would silently replace whatever you had put there.
 
-The engine has to be told how many cards it has (`fluksio serve --gpus 1`, or
-`FLOW_GPUS`) because detecting them would mean depending on a vendor's
-tooling. Until it is, a node asking for one is quietly given zero and runs
-alongside every other; the log says so the first time it happens.
+The engine counts NVIDIA's device nodes to find its cards, which asks no
+vendor tool. A card those do not cover has to be declared (`fluksio serve
+--gpus 1`, or `FLOW_GPUS`); until it is, a node asking for one is quietly
+given zero and runs alongside every other, and the log says so the first time
+it happens.
 
 Workers are kept warm on purpose, so a library that takes most of the card at
 import would hold it after the run finished. The workers that ran on a card
@@ -645,7 +646,7 @@ code digest beside it, so an exported file still says what produced its
 numbers. Numbers inside a record are columns of their own
 (`metric.final_metrics.train_loss`) and `--metrics` and `--params` take those
 dotted paths to narrow the table. Metric names are flow-qualified, so
-`--list` prints the ones a selection carries. `Client.export_metrics()` and
+`--list` prints every one a selection recorded. `Client.export_metrics()` and
 `Client.export_runs()` answer the same rows to a notebook, ready for
 `pandas.DataFrame`.
 

docs/reference/configuration.md

@@ -127,7 +127,7 @@ warning into a refusal to start.
 | `FLOW_MAX_RUNS` | `4` | batch runs driven at once. A different limit from the one above: a run drives a whole graph, and its nodes are bounded by `FLOW_MAX_WORKERS`. This is what a sweep queues behind |
 | `FLOW_NODE_TIMEOUT` | `0` | seconds a node may be silent, unless it sets its own; 0 is no limit |
 | `FLOW_CPUS` | `0` | cores nodes that declare `resources` may be given; 0 works it out as every core but two, which are what keeps the engine answering while the machine is busy |
-| `FLOW_GPUS` | `0` | GPUs on this machine, each held by one node at a time. Not detected — say how many there are |
+| `FLOW_GPUS` | `0` | GPUs on this machine, each held by one node at a time. 0 counts NVIDIA's device nodes; say a number for anything they miss |
 | `OBS_RETENTION_DAYS` | `30` | how long metrics, events and run records are kept |
 | `ARTIFACT_GC_INTERVAL_S` | `3600` | how often artifact bytes nothing refers to are swept away; 0 never sweeps |
 | `ARTIFACT_GC_GRACE_S` | `3600` | how long a freshly written artifact is spared, whatever refers to it |