Close eight open SDK tasks: the pidfile, the log, cards, names and a live curve
Each was a loose end recorded under `### SDK` in the notepad. `serve` takes its own pidfile down on SIGTERM. uvicorn restores the handler it found and re-raises the signal it stopped on, so the default handler ended the process without unwinding and the `finally` never ran — which is what a stop sends, and what left `serve.pid` behind. `serve.log` is cut back past 5 MB by the engine rather than by the screen that started it, so an adopted engine is bounded too. Gated on its own stdout being an appended regular file, which is what makes the cut safe: the kernel then puts the next write at the new end. Cards are counted from `/dev/nvidia[0-9]*`, so `FLOW_GPUS`/`--gpus` of 0 means "work it out" the way `FLOW_CPUS` always has. The engine counts, not the accountant — a remote worker builds one of those from its own inventory, and detecting there would hand it the engine host's cards. The worker counts last: what a batch job says it was granted still wins. `GET /runs/metrics/names` is the distinct over a selection that `--list` and the terminal's metric picker were approximating by reading the newest run that had measured anything, which missed a name only an older run ever wrote. `MetricSink` announces each batch it has written (`run_metric`, carrying the names). Not a per-point event: one covers up to 500 points or two seconds of them, and the rows stay the record. The terminal comparison fills in as the first readings land instead of staying blank until reopened, and the browser refetches the run and any comparison rather than the list behind them. `retry --group` pages the list route by `before` instead of stopping at 500. The terminal dashboard takes the terminal's colours (`ansi-dark`), and the web UI can re-pair from Settings without disconnecting first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRQ9bmTvCbqCwXo9mxZzzV
This commit is contained in:
+42
-38
@@ -999,15 +999,41 @@ def cmd_runs(args: argparse.Namespace) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
#: The most runs of one group a retry looks at, which is the list route's own
|
||||
#: ceiling. ponytail: a sweep larger than this needs paging, not a bigger number.
|
||||
MAX_GROUP = 500
|
||||
#: One page of the list route, which is its own ceiling. A sweep larger than
|
||||
#: this is read a page at a time rather than asked for in one.
|
||||
PAGE = 500
|
||||
|
||||
#: What a retry leaves alone. Everything else in a group — error, cancelled,
|
||||
#: abandoned — is what "the ones that did not make it" means.
|
||||
KEPT = frozenset({"ok", "cached", "queued", "running"})
|
||||
|
||||
|
||||
def _group_runs(client: Client, group: str) -> Iterator[dict[str, Any]]:
|
||||
"""Every run of a sweep, a page at a time.
|
||||
|
||||
Rows come newest first and `before` is the cursor: handing back the last
|
||||
row's own `created_at` reads the next page whatever landed meanwhile, where
|
||||
an offset would shift under a run submitted between two pages.
|
||||
|
||||
# ponytail: `before` is exclusive, so two runs sharing a timestamp exactly
|
||||
# across a page edge would lose one. Ids are minted per run; the history
|
||||
# has never produced a tie.
|
||||
"""
|
||||
before = ""
|
||||
while True:
|
||||
page = list(
|
||||
client.runs(
|
||||
limit=PAGE, group=group, **({"before": before} if before else {})
|
||||
)
|
||||
)
|
||||
yield from page
|
||||
if len(page) < PAGE:
|
||||
return
|
||||
before = str(page[-1].get("created_at") or "")
|
||||
if not before:
|
||||
return
|
||||
|
||||
|
||||
def cmd_retry(args: argparse.Namespace) -> int:
|
||||
"""Run the same thing again, one run or a group's unfinished ones."""
|
||||
try:
|
||||
@@ -1016,7 +1042,7 @@ def cmd_retry(args: argparse.Namespace) -> int:
|
||||
if args.group:
|
||||
ids += [
|
||||
str(row["id"])
|
||||
for row in client.runs(limit=MAX_GROUP, group=args.group)
|
||||
for row in _group_runs(client, args.group)
|
||||
if str(row["status"]) not in KEPT
|
||||
]
|
||||
if not ids:
|
||||
@@ -1207,39 +1233,15 @@ def _write_rows(rows: list[dict[str, Any]], fmt: str, out: str, hint: str = "")
|
||||
return 0
|
||||
|
||||
|
||||
#: How many runs `--list` reads before giving up on finding a metric name.
|
||||
#: The names belong to the flow's nodes rather than to a run, so the newest
|
||||
#: one that recorded any is the whole vocabulary — the rest are for a
|
||||
#: selection whose newest runs failed before they measured anything.
|
||||
# ponytail: the first run with names wins; a name only an older run recorded
|
||||
# is not listed. A `distinct` over the selection would be exact and is a route
|
||||
# of its own.
|
||||
LIST_SCAN = 10
|
||||
|
||||
|
||||
def metric_names(client: Client, ids: Iterable[str]) -> list[str]:
|
||||
"""The metric names these runs carry, since a name is flow-qualified.
|
||||
|
||||
`train_loss` is recorded as `train.train_loss`, and asking for the bare
|
||||
one matches nothing — so this is the answer to "what would match".
|
||||
"""
|
||||
for run_id in list(ids)[:LIST_SCAN]:
|
||||
names = sorted({point["name"] for point in client.metrics(run_id)})
|
||||
if names:
|
||||
return names
|
||||
return []
|
||||
|
||||
|
||||
def _list_names(client: Client, args: argparse.Namespace) -> int:
|
||||
filters = _selection(args)
|
||||
if "until" in filters:
|
||||
# The history spells the same bound `before`, where it is also the
|
||||
# cursor a page is taken from.
|
||||
filters["before"] = filters.pop("until")
|
||||
ids = args.run or [
|
||||
row["id"] for row in client.runs(flow=args.flow, limit=LIST_SCAN, **filters)
|
||||
]
|
||||
names = metric_names(client, ids)
|
||||
"""What `--list` prints: the names a selection recorded, from the engine.
|
||||
|
||||
One `distinct` over the whole selection rather than the newest run that
|
||||
happened to measure anything, so a name only an older run ever wrote is
|
||||
listed too. The selection is the export's own filters, unchanged — this
|
||||
route takes `until` where the history route calls the same bound `before`.
|
||||
"""
|
||||
names = client.metric_names(ids=args.run, flow=args.flow or "", **_selection(args))
|
||||
_say("\n".join(names) if names else "No metrics recorded by these runs.")
|
||||
return 0
|
||||
|
||||
@@ -1275,13 +1277,15 @@ def _export(
|
||||
)
|
||||
try:
|
||||
with _client_for(args, retries=0) as client:
|
||||
if getattr(args, "list_names", False):
|
||||
return _list_names(client, args)
|
||||
try:
|
||||
if getattr(args, "list_names", False):
|
||||
return _list_names(client, args)
|
||||
rows = fetch(client)
|
||||
except ApiError as exc:
|
||||
if exc.status != 404:
|
||||
raise
|
||||
# `--list` asks a route of its own, and it arrived later than
|
||||
# the exports — so an engine without either says the same thing.
|
||||
return _fail(_too_old(client))
|
||||
except (SyncError, ApiError) as exc:
|
||||
return _fail(str(exc))
|
||||
|
||||
Reference in New Issue
Block a user