Export runs and their curves as tables an analysis reads
Docs / docs (push) Successful in 35s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m11s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m17s
pre-commit / pre-commit (push) Failing after 2m44s
Test Backend / test-backend (push) Successful in 3m0s
Compose Smoke Test / test-compose (push) Successful in 41s
Playwright Tests / merge-reports (push) Successful in 8m14s

`fluksio export metrics` is the long table — a row per run, metric and step —
and `fluksio export runs` the wide one, a row per run with the inputs that
*vary* across the selection as columns beside its final numbers, status,
duration and the commit and digest of the code it ran. Both carry the run id
on every row, which is the join back to the run page and what makes an
exported file auditable. `Client.export_metrics`/`export_runs` answer the same
rows to a notebook.

The engine streams csv or jsonl from two routes declared above `/{run_id}`;
parquet is a client-side conversion behind the new `fluksio[parquet]` extra,
so nobody pays for pyarrow who does not want dtypes kept. The long export
reads each run through `_series`, so a cached node's curve comes with it, and
`--stride` thins each series rather than the concatenation of all of them.

Two things they needed on the way: `GET /runs` takes `?since=` and `?before=`,
so a long history pages by the last row's own timestamp instead of an offset
that shifts under it; and a read that reaches no engine now says so in half a
second rather than seven, because `runs`, `flavors`, `export` and an unwatched
`status` pass `retries=0`. Everything that submits keeps them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9Hdrmf2cwNABCnE5x9UJa
This commit is contained in:
2026-08-27 17:43:30 +02:00
co-authored by Claude Opus 5
parent 96cf1fc0c8
commit 51464941ac
12 changed files with 813 additions and 9 deletions
+15 -1
View File
@@ -104,8 +104,10 @@ published to. Flows own the namespace; everything else is a client of it.
|---|---|---|
| `POST` | `/runs/flows/{name}` | queue one run — `{"params": {...}, "seed": 7, "draft": false, "no_cache": false}`. `"cause"` says where it came from — `api` (the default), `cli` or `sdk` |
| `POST` | `/runs/flows/{name}/sweep` | queue many, sharing a `group_id` |
| `GET` | `/runs` | the queryable history: `?flow=`, `?status=`, `?group=`, `?digest=`, `?limit=`, `?offset=` |
| `GET` | `/runs` | the queryable history: `?flow=`, `?status=`, `?group=`, `?digest=`, `?since=`, `?before=`, `?limit=`, `?offset=` |
| `GET` | `/runs/overview` | one row per flow that has runs, with how many are running or queued |
| `GET` | `/runs/export/metrics?…&name=&stride=&format=` | every selected run's series as one long table: `run, name, step, ts, value` |
| `GET` | `/runs/export/runs?…&params=&metrics=&format=` | one row per run: its inputs as columns, its final numbers, its status and provenance |
| `GET` | `/runs/{id}` | one run in full: params, result, per-node record, artifacts |
| `POST` | `/runs/{id}/cancel` | stop it |
| `GET` | `/runs/{id}/metrics?name=&stride=` | one metric's series, in step order; every series of the run without `name` |
@@ -118,6 +120,18 @@ problem, before anything executes.
`?digest=` filters by the hash of the parameters, which is how you find "every
run that used exactly this configuration".
`?before=` is how a long history is paged: rows come newest first, so handing
back the last row's `created_at` reads the next page whatever landed
meanwhile — which `?offset=` cannot, since a run submitted between two pages
shifts every row down one. `?since=` is inclusive and bounds the other end.
Both exports stream `csv` (the default) or `jsonl`, and take the selection the
history takes plus `?ids=a,b,c`, `?since=` and `?until=`. `export/runs` puts
the inputs that *vary* across the selection in `param.` columns — the sweep
axis — unless `?params=` names them, and the run's numbers in `metric.`
columns. The run id is on every row of both, which is what makes an exported
file a join back to the run rather than a loose table.
`compare` answers in the same shape a chart widget draws, so three training
curves side by side is a widget binding rather than a screen of its own.
+30
View File
@@ -304,6 +304,36 @@ flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`,
`--no-cache` and `--local` mean what they do there. `--wait` blocks until all
of them are finished and exits non-zero if any failed.
### `fluksio export`
```sh
fluksio export metrics --flow train --name train_loss,val_loss --stride 10 -o curves.csv
fluksio export runs --flow train --status ok > arms.csv
```
The two tables an analysis reads. `export metrics` is the long one — a row per
run, metric and step — which is what a plotting library takes without
reshaping; `--name` keeps the metrics it lists and `--stride` keeps every Nth
point of *each* curve. `export runs` is the wide one: a row per run with its
inputs as columns, its final numbers, its status, its duration and the commit
and digest of the code it ran.
The inputs that become columns are the ones that **vary** across the selected
runs — the axis of the sweep, which is what a comparison is read along —
unless `--params lr,seed` names them. `--metrics` narrows the final numbers
the same way.
Both take `--flow`, `--run ID` (repeat it), `--group`, `--status`, `--since`,
`--until` and `--local`, and both put the run id on every row: it is the join
back to the run page and to what the run made.
`--format` is `csv` (the default), `jsonl` or `parquet`; output goes to stdout
unless `-o FILE` names somewhere. Parquet keeps the types and needs pyarrow —
`pip install 'fluksio[parquet]'` — and a file to write, since it is not a
stream. In a notebook, `Client.export_metrics()` and `Client.export_runs()`
answer the same rows as a list of dicts, which `pandas.DataFrame` takes
directly.
## What lives in the data directory
```text