Close eight open SDK tasks: the pidfile, the log, cards, names and a live curve

Each was a loose end recorded under `### SDK` in the notepad.

`serve` takes its own pidfile down on SIGTERM. uvicorn restores the handler it
found and re-raises the signal it stopped on, so the default handler ended the
process without unwinding and the `finally` never ran — which is what a stop
sends, and what left `serve.pid` behind.

`serve.log` is cut back past 5 MB by the engine rather than by the screen that
started it, so an adopted engine is bounded too. Gated on its own stdout being
an appended regular file, which is what makes the cut safe: the kernel then
puts the next write at the new end.

Cards are counted from `/dev/nvidia[0-9]*`, so `FLOW_GPUS`/`--gpus` of 0 means
"work it out" the way `FLOW_CPUS` always has. The engine counts, not the
accountant — a remote worker builds one of those from its own inventory, and
detecting there would hand it the engine host's cards. The worker counts last:
what a batch job says it was granted still wins.

`GET /runs/metrics/names` is the distinct over a selection that `--list` and
the terminal's metric picker were approximating by reading the newest run that
had measured anything, which missed a name only an older run ever wrote.

`MetricSink` announces each batch it has written (`run_metric`, carrying the
names). Not a per-point event: one covers up to 500 points or two seconds of
them, and the rows stay the record. The terminal comparison fills in as the
first readings land instead of staying blank until reopened, and the browser
refetches the run and any comparison rather than the list behind them.

`retry --group` pages the list route by `before` instead of stopping at 500.

The terminal dashboard takes the terminal's colours (`ansi-dark`), and the web
UI can re-pair from Settings without disconnecting first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRQ9bmTvCbqCwXo9mxZzzV
This commit is contained in:
2026-09-02 16:40:51 +02:00
co-authored by Claude Opus 5
parent 3e4224df53
commit 058f16ec1d
24 changed files with 686 additions and 169 deletions
+52 -7
View File
@@ -378,20 +378,18 @@ RUN_COLUMNS = (
)
def _selected(
session: Session,
def _selection(
flow: str | None,
status: str | None,
group: str | None,
ids: str,
since: datetime | None,
until: datetime | None,
) -> list[Run]:
"""The runs an export covers, newest first — the filters the list takes.
) -> Any:
"""The query behind a selection of runs, newest first.
Capped: the filters bound a *sensible* request and nothing bounded an
unfiltered one, so asking for everything read every row of the table into
memory before a byte was sent. A truncated export says so in a header.
Separate from reading it, because the names route asks the same question
as a subquery rather than for the rows.
"""
statement = select(Run).order_by(col(Run.created_at).desc())
if flow:
@@ -407,6 +405,25 @@ def _selected(
statement = statement.where(col(Run.created_at) >= _aware(since))
if until:
statement = statement.where(col(Run.created_at) < _aware(until))
return statement
def _selected(
session: Session,
flow: str | None,
status: str | None,
group: str | None,
ids: str,
since: datetime | None,
until: datetime | None,
) -> list[Run]:
"""The runs an export covers, newest first — the filters the list takes.
Capped: the filters bound a *sensible* request and nothing bounded an
unfiltered one, so asking for everything read every row of the table into
memory before a byte was sent. A truncated export says so in a header.
"""
statement = _selection(flow, status, group, ids, since, until)
return list(session.exec(statement.limit(EXPORT_CAP + 1)))
@@ -614,6 +631,34 @@ def export_runs(
return _stream(format, "runs", columns, chunks(), truncated)
@router.get("/metrics/names", response_model=list[str])
def read_metric_names(
session: SessionDep,
flow: str | None = None,
status: str | None = None,
group: str | None = None,
ids: str = "",
since: datetime | None = None,
until: datetime | None = None,
) -> Any:
"""Every metric name the selected runs recorded.
A name is flow-qualified — a node of `train` writing `train_loss` records
`train.train_loss` — so this is the answer to "what would match". Exact
over the whole selection: the client used to read the newest run that had
measured anything and take its names for the vocabulary, which missed a
name only an older run ever wrote. Declared before `/{run_id}`, or that
route would take "metrics" for an id.
"""
chosen = _selection(flow, status, group, ids, since, until).with_only_columns(
col(Run.id)
)
names = session.exec(
select(col(RunMetric.name)).where(col(RunMetric.run_id).in_(chosen)).distinct()
).all()
return sorted(names)
@router.get("/{run_id}", response_model=RunDetail)
def read_run(run_id: str, session: SessionDep) -> Any:
"""One run in full: what it was asked, what each node did, what it made."""