Follow a record into its fields, name the metrics, name the version
Docs / docs (push) Successful in 38s
Playwright Tests / test-playwright (1, 2) (push) Successful in 2m56s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m7s
pre-commit / pre-commit (push) Failing after 2m17s
Test Backend / test-backend (push) Successful in 2m54s
Compose Smoke Test / test-compose (push) Successful in 44s
Playwright Tests / merge-reports (push) Successful in 1m17s
Docs / docs (push) Successful in 38s
Playwright Tests / test-playwright (1, 2) (push) Successful in 2m56s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m7s
pre-commit / pre-commit (push) Failing after 2m17s
Test Backend / test-backend (push) Successful in 2m54s
Compose Smoke Test / test-compose (push) Successful in 44s
Playwright Tests / merge-reports (push) Successful in 1m17s
Three things the first export pass got wrong for a real study. **Dotted paths.** A node returns a record, not a scalar — the numbers arrive inside `final_metrics` — so `--metrics final_metrics.train_loss` yielded an empty column and `--metrics final_metrics` yielded the whole record in one cell. Both sides of the wide table now take dotted paths, and the defaults reach the same depth: every number a result carries is a column named by its path, and inputs are compared leaf by leaf, so two configurations differing in one field give that field as the axis rather than two blobs that are merely not equal. Lists stay whole — a curve belongs in the long table. **`--list`.** Metric names are flow-qualified, so `--name train_loss` matched nothing and said only that. `fluksio export metrics --list` prints the names the selection carries, and an empty export made with `--name` points at it. **A version to compare.** The CLI ships ahead of the engine and a stale one answered a flat 404 with nothing anywhere in the API to tell how old it was. The engine reports `version` on `/observability/summary`, `fluksio status` prints it, and a 404 from export now names both versions — or says "older" when the field itself predates the engine. Bumped to 0.1.5, which is what makes the number worth reading. Also formats `flow/metrics.py`, which had been committed unformatted and was the last `ruff format --check` failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9Hdrmf2cwNABCnE5x9UJa
This commit is contained in:
@@ -391,30 +391,62 @@ def _cell(value: Any) -> Any:
|
||||
return json.dumps(value)
|
||||
|
||||
|
||||
def _leaves(value: Any, prefix: str = "") -> Iterator[tuple[str, Any]]:
|
||||
"""Everything a record holds, by its dotted path.
|
||||
|
||||
A node returning a record rather than a scalar is the ordinary shape —
|
||||
the numbers arrive inside `final_metrics` — and a record in one cell is
|
||||
not a column anybody can compare. Lists are left whole: a curve belongs in
|
||||
the long table, not in a cell of this one.
|
||||
"""
|
||||
if isinstance(value, dict):
|
||||
for key, inner in value.items():
|
||||
yield from _leaves(inner, f"{prefix}.{key}" if prefix else str(key))
|
||||
else:
|
||||
yield prefix, value
|
||||
|
||||
|
||||
def _dig(record: dict[str, Any], path: str) -> Any:
|
||||
"""A dotted path into a record: `final_metrics.train_loss`.
|
||||
|
||||
A key with a dot in its own name is not reachable this way, which is the
|
||||
price of the spelling.
|
||||
"""
|
||||
value: Any = record
|
||||
for part in path.split("."):
|
||||
if not isinstance(value, dict) or part not in value:
|
||||
return None
|
||||
value = value[part]
|
||||
return value
|
||||
|
||||
|
||||
def _varying(runs: list[Run]) -> list[str]:
|
||||
"""The inputs that differ across these runs — the axis of a sweep.
|
||||
|
||||
What a reader comparing arms wants as columns. Under two runs nothing can
|
||||
differ, and a table of one run with none of its inputs in it is not worth
|
||||
reading, so all of them are kept.
|
||||
What a reader comparing arms wants as columns, compared leaf by leaf: two
|
||||
configurations differing in one field give that field as a column rather
|
||||
than two blobs that are not the same. Under two runs nothing can differ,
|
||||
and a table of one run with none of its inputs in it is not worth reading,
|
||||
so all of them are kept.
|
||||
"""
|
||||
keys = sorted({key for run in runs for key in run.params})
|
||||
keys = sorted({path for run in runs for path, _ in _leaves(run.params)})
|
||||
if len(runs) < 2:
|
||||
return keys
|
||||
return [
|
||||
key
|
||||
for key in keys
|
||||
if len({json.dumps(run.params.get(key), sort_keys=True) for run in runs}) > 1
|
||||
if len({json.dumps(_dig(run.params, key), sort_keys=True) for run in runs}) > 1
|
||||
]
|
||||
|
||||
|
||||
def _scored(runs: list[Run]) -> list[str]:
|
||||
"""A run's final numbers: every scalar its declared outputs carry."""
|
||||
"""A run's final numbers: every number its declared outputs carry, however
|
||||
deep it sits. A flag is not a number, and neither is a label."""
|
||||
return sorted(
|
||||
{
|
||||
key
|
||||
path
|
||||
for run in runs
|
||||
for key, value in run.result.items()
|
||||
for path, value in _leaves(run.result)
|
||||
if isinstance(value, (int, float)) and not isinstance(value, bool)
|
||||
}
|
||||
)
|
||||
@@ -526,6 +558,11 @@ def export_runs(
|
||||
by default the ones that vary across the selection, which is the sweep
|
||||
axis; ``params`` names them instead. ``metrics`` narrows the final numbers
|
||||
to a few of a run's declared outputs.
|
||||
|
||||
Both take dotted paths into a record a node returned:
|
||||
``metrics=final_metrics.train_loss,test_metrics.known.perfect`` selects
|
||||
three fields rather than two blobs, and the defaults reach the same
|
||||
depth.
|
||||
"""
|
||||
runs = _selected(session, flow, status, group, ids, since, until)
|
||||
inputs = [part for part in params.split(",") if part] or _varying(runs)
|
||||
@@ -539,8 +576,8 @@ def export_runs(
|
||||
def chunks() -> Iterator[list[dict[str, Any]]]:
|
||||
for run in runs:
|
||||
row = {column: _cell(getattr(run, column)) for column in RUN_COLUMNS}
|
||||
row.update((f"param.{k}", _cell(run.params.get(k))) for k in inputs)
|
||||
row.update((f"metric.{k}", _cell(run.result.get(k))) for k in scores)
|
||||
row.update((f"param.{k}", _cell(_dig(run.params, k))) for k in inputs)
|
||||
row.update((f"metric.{k}", _cell(_dig(run.result, k))) for k in scores)
|
||||
yield [row]
|
||||
|
||||
return _stream(format, "runs", columns, chunks())
|
||||
|
||||
Reference in New Issue
Block a user