Stage caching for batch runs, and an engine that lives in the command
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m5s
Playwright Tests / test-playwright (2, 2) (push) Failing after 20s
pre-commit / pre-commit (push) Failing after 2m33s
Test Backend / test-backend (push) Successful in 2m7s
Compose Smoke Test / test-compose (push) Failing after 20s
Playwright Tests / merge-reports (push) Failing after 1m3s
Publish / publish (push) Failing after 12s

A code node in a batch run is now fingerprinted by its source, its raw
settings and the values it reads — an artifact input counting as its digest,
which is what the content addressing was always for. A run that finds the key
restores what the earlier one returned and skips the node, recorded as
`cached`. The run history is the cache: `run_node.outputs` beside the
`cache_key` the schema already had, no second store. On for code nodes, never
for the built-in and connector types that have side effects; off per node with
`@node(cache=False)` and per run with `--no-cache`.

Emissions are not replayed on a hit, so a cached training node returns its
result without redrawing its curve. Recorded in NOTEPAD.md with the two other
deliberate limits.

`fluksio run --local` boots the real app in the command's own process and
drives it through its ASGI interface behind the ordinary client, so a run no
longer needs a `serve` terminal beside it — same data directory, same history,
and the cache carries between the two. It always waits, because the engine it
starts lives exactly as long as the command.

Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists,
`run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited
run rather than abandoning it, coloured statuses on a terminal, and `name`
made optional on the metrics endpoint so a follower can ask for every series.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-24 20:31:31 +02:00
co-authored by Claude Opus 5
parent 7a9502883a
commit 400d7d9c5c
23 changed files with 1147 additions and 58 deletions
+29 -4
View File
@@ -405,7 +405,24 @@ again.
`--lr` is typed by the flow's own inputs, so `0.003` arrives as a float. A
parameter you did not declare, or one of the wrong type, is refused before
anything executes. It answers immediately with a queued run — training is
measured in hours, so nothing waits for it unless you pass `--wait`.
measured in hours, so nothing waits for it unless you pass `--wait`, or
`--follow`, which waits and prints the numbers as the run reports them.
No engine has to be up for any of this: `fluksio run --local` boots the
engine inside the command instead, on the same `.fluksio` — the same database,
artifacts and history a served one would use. It costs a few seconds of
startup per invocation against the ~15 ms of submitting to an engine that is
already up, so it is for the run you want now rather than the loop you are
iterating in.
A second run of a flow you did not change mostly does not execute. Each of
your nodes is fingerprinted by its source, its settings and the values it
reads, and one an earlier run already answered is restored from that run
rather than run again — reported as `cached`, so a flow with nothing left to
do finishes as `(3/3 cached)`. Change `--lr` and only the nodes downstream of
it run. `--no-cache` turns that off for one run, `@node(..., cache=False)` for
one node; the caveat and the details are in
[Stage caching](../concepts/runs.md#stage-caching).
From Python, the flow you declared is also the handle to its runs:
@@ -436,8 +453,15 @@ was the learning rate on the run that got 94%?".
## Sweep it
A grid search and an ensemble are the same call — you build the list, Fluksio
runs them in parallel:
A grid search and an ensemble are the same submission, run in parallel:
```sh
fluksio sweep train --param lr=0.001,0.003,0.01 --wait
```
Every combination of the lists you give, so a second `--param` is a grid
rather than a second sweep. Where the set you want is not a product, build the
list yourself and post it:
```sh
curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \
@@ -578,7 +602,8 @@ is the same stack.
artifacts, sweeps, durability, what happens when your engine dies mid-training
- [Writing node code](../code/nodes.md) — generators, settings, what a node may
and may not do
- [The command line](../code/cli.md) — `login`, `sync`, `run`, `runs` in full
- [The command line](../code/cli.md) — `login`, `sync`, `run`, `runs`, `sweep`
in full
- [Remote workers](../code/workers.md) — send the training node to the GPU box
and keep the rest on your laptop
- [The flow editor](../interface/flow-editor.md) — once you have a portal, this