Stage caching for batch runs, and an engine that lives in the command

A code node in a batch run is now fingerprinted by its source, its raw
settings and the values it reads — an artifact input counting as its digest,
which is what the content addressing was always for. A run that finds the key
restores what the earlier one returned and skips the node, recorded as
`cached`. The run history is the cache: `run_node.outputs` beside the
`cache_key` the schema already had, no second store. On for code nodes, never
for the built-in and connector types that have side effects; off per node with
`@node(cache=False)` and per run with `--no-cache`.

Emissions are not replayed on a hit, so a cached training node returns its
result without redrawing its curve. Recorded in NOTEPAD.md with the two other
deliberate limits.

`fluksio run --local` boots the real app in the command's own process and
drives it through its ASGI interface behind the ordinary client, so a run no
longer needs a `serve` terminal beside it — same data directory, same history,
and the cache carries between the two. It always waits, because the engine it
starts lives exactly as long as the command.

Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists,
`run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited
run rather than abandoning it, coloured statuses on a terminal, and `name`
made optional on the metrics endpoint so a follower can ask for every series.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/code/api.md

@@ -102,12 +102,12 @@ published to. Flows own the namespace; everything else is a client of it.
 
 | Method | Path | What |
 |---|---|---|
-| `POST` | `/runs/flows/{name}` | queue one run — `{"params": {...}, "seed": 7, "draft": false}` |
+| `POST` | `/runs/flows/{name}` | queue one run — `{"params": {...}, "seed": 7, "draft": false, "no_cache": false}` |
 | `POST` | `/runs/flows/{name}/sweep` | queue many, sharing a `group_id` |
 | `GET` | `/runs` | the queryable history: `?flow=`, `?status=`, `?group=`, `?digest=`, `?limit=` |
 | `GET` | `/runs/{id}` | one run in full: params, result, per-node record, artifacts |
 | `POST` | `/runs/{id}/cancel` | stop it |
-| `GET` | `/runs/{id}/metrics?name=&stride=` | one metric's series, in step order |
+| `GET` | `/runs/{id}/metrics?name=&stride=` | one metric's series, in step order; every series of the run without `name` |
 | `GET` | `/runs/series/compare?ids=a,b,c&metric=` | that metric across several runs |
 
 Submitting answers immediately with a `queued` run. Wrong parameters — an

docs/code/cli.md

@@ -12,8 +12,8 @@ should only *run nodes* for an engine elsewhere. It has none of the engine in
 it. See [Remote workers](workers.md).
 
 The command is two things at once: `serve`, `enroll` and `worker` *are* an
-installation, while `login`, `sync`, `run` and `runs` talk to one that may be
-anywhere.
+installation, while `login`, `sync`, `run`, `runs` and `sweep` talk to one that
+may be anywhere.
 
 ## Where an installation lives
 
@@ -122,8 +122,9 @@ See [Remote workers](workers.md).
 
 ## Talking to an engine
 
-The four commands below are the client half: they run wherever you work, and
-address an engine over its API rather than being one.
+The commands below are the client half: they run wherever you work, and
+address an engine over its API rather than being one — except under `--local`,
+which boots one inside the command instead.
 
 ### `fluksio login`
 
@@ -171,13 +172,34 @@ fluksio run train --lr 0.05 --seed 7 [--wait]
 Syncs the working directory, then submits a run — so the command after an edit
 is this one and nothing else. Flags that are not its own are the flow's
 inputs, typed by what the flow declares them as. `--wait` blocks until the run
-finishes and exits non-zero if it failed.
+finishes and exits non-zero if it failed. `--follow` waits as well, and prints
+the numbers the run reports as they arrive:
+
+```text
+  train.loss[14] = 3.40295e-06
+```
+
+Ctrl-C while either is waiting cancels the run on the engine rather than only
+stopping the watching, and exits 130.
 
 `--no-sync` runs what is already on the engine. Worth it in a tight loop where
 you know nothing changed, since syncing retires the workers and the next call
 pays its imports again. A directory that declares no flows syncs nothing and
 says nothing — a flow drawn on the canvas is run the same way.
 
+`--no-cache` executes every node, including one an earlier run already
+answered. See [Stage caching](../concepts/runs.md#stage-caching).
+
+`--local` boots the engine inside this process instead of talking to a served
+one, so there is no `fluksio serve` terminal to keep open. It is the same
+installation either way — the same `.fluksio`, the same database, artifacts
+and run history — so a run made this way and a run made through a served
+engine cache against each other. It always waits, because the engine it starts
+lives exactly as long as the command. Starting one costs a few seconds of
+worker pool and module reconcile, against the ~15 ms of submitting to an
+engine that is already up: `--local` is for "I just want to run it", not for a
+loop you are iterating in.
+
 ### `fluksio runs`
 
 ```sh
@@ -185,7 +207,22 @@ fluksio runs [--flow train] [--limit 20]
 ```
 
 The runs an engine has recorded, newest first: id, status, flow, duration, the
-commit of the repository it came from, and its parameters.
+commit of the repository it came from, and its parameters. Statuses are
+coloured when a terminal is reading the output — `ok` green, `error` red,
+`cached` cyan. `--local` reads the same history from an in-process engine,
+without one having to be served.
+
+### `fluksio sweep`
+
+```sh
+fluksio sweep train --param lr=0.1,0.01 --param epochs=10,50 --wait
+```
+
+Every combination of the parameter lists, submitted as one group — four runs
+above, sharing a `group_id` and executing in parallel. Values are typed by the
+flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`,
+`--no-cache` and `--local` mean what they do there. `--wait` blocks until all
+of them are finished and exits non-zero if any failed.
 
 ## What lives in the data directory
 

docs/concepts/runs.md

@@ -66,6 +66,9 @@ curl -X POST $FLUKSIO/runs/flows/train_polymer_gnn/sweep \
 }'
 ```
 
+From a terminal that is `fluksio sweep train_polymer_gnn --param lr=0.1,0.3`,
+which builds the product of the lists you give it and posts the same call.
+
 They share a `group_id`, so `GET /api/v1/runs?group=…` is the sweep, and they
 execute in parallel. That is safe because **each run has a state backend of its
 own**: message names are global keys, so two runs of one flow would otherwise
@@ -126,7 +129,8 @@ def process():
 
 Every number a node emits is kept as the run's series, stepped by the count of
 emissions on that message. Read one back with
-`GET /api/v1/runs/{id}/metrics?name=<flow>.loss`, or compare runs:
+`GET /api/v1/runs/{id}/metrics?name=<flow>.loss` — or leave `name` off for
+every series the run kept — or compare runs:
 
 ```
 GET /api/v1/runs/series/compare?ids=<a>,<b>,<c>&metric=<flow>.loss
@@ -168,6 +172,40 @@ one preprocessed input stores it once, and a reference stays valid wherever
 the store is reachable from. Artifacts a run produced are listed on it and
 downloadable at `GET /api/v1/artifacts/{digest}`.
 
+## Stage caching
+
+A run mostly does not redo what an earlier one already did. Before a node
+executes it is fingerprinted — a sha256 over its source, its settings and the
+values it is about to read — and if some earlier run of that same fingerprint
+finished, what that one returned is restored into this run's state and the node
+is skipped. It is recorded with the status `cached` and a duration of zero, and
+its artifacts are listed on the new run as well, so they stay downloadable from
+either.
+
+The settings go into the key raw, so a secret contributes its `{"$secret":
+name}` reference and never its value. An artifact input counts as its content
+digest: the same bytes under a different filename are the same input. Each node
+on a run carries the `cache_key` it was looked up by.
+
+The run history *is* the cache; there is no second store. A node's returned
+outputs are kept on its run record as canonical JSON, up to 256000 characters —
+a node returning more than that is simply not cacheable that run. An entry
+whose artifact bytes have since left the store is a miss, not an error.
+
+Only `python` nodes are cached, and by default all of them are. A built-in node
+type or a connector node has side effects and no source to fingerprint, so
+neither is ever a candidate. Turn it off for one node with
+`@node(..., cache=False)` — the flow document carries it as `cache`, so the
+canvas and the API can change it too — or for one run with
+`fluksio run --no-cache`, `fluksio sweep --no-cache`, or `"no_cache": true` in
+the submission body.
+
+What a cached node does not bring back is what it emitted on the way. Its
+returned outputs are restored; the values it published mid-execution are not,
+because those were the story of an execution that is not happening this time.
+So a skipped training node contributes no loss curve to the new run — if you
+want the curve, that run has to actually train.
+
 ## Objects that cannot be serialized
 
 A live model, a `DataLoader`, a JAX-compiled function — these do not cross a

docs/getting-started/data-science.md

@@ -405,7 +405,24 @@ again.
 `--lr` is typed by the flow's own inputs, so `0.003` arrives as a float. A
 parameter you did not declare, or one of the wrong type, is refused before
 anything executes. It answers immediately with a queued run — training is
-measured in hours, so nothing waits for it unless you pass `--wait`.
+measured in hours, so nothing waits for it unless you pass `--wait`, or
+`--follow`, which waits and prints the numbers as the run reports them.
+
+No engine has to be up for any of this: `fluksio run --local` boots the
+engine inside the command instead, on the same `.fluksio` — the same database,
+artifacts and history a served one would use. It costs a few seconds of
+startup per invocation against the ~15 ms of submitting to an engine that is
+already up, so it is for the run you want now rather than the loop you are
+iterating in.
+
+A second run of a flow you did not change mostly does not execute. Each of
+your nodes is fingerprinted by its source, its settings and the values it
+reads, and one an earlier run already answered is restored from that run
+rather than run again — reported as `cached`, so a flow with nothing left to
+do finishes as `(3/3 cached)`. Change `--lr` and only the nodes downstream of
+it run. `--no-cache` turns that off for one run, `@node(..., cache=False)` for
+one node; the caveat and the details are in
+[Stage caching](../concepts/runs.md#stage-caching).
 
 From Python, the flow you declared is also the handle to its runs:
 
@@ -436,8 +453,15 @@ was the learning rate on the run that got 94%?".
 
 ## Sweep it
 
-A grid search and an ensemble are the same call — you build the list, Fluksio
-runs them in parallel:
+A grid search and an ensemble are the same submission, run in parallel:
+
+```sh
+fluksio sweep train --param lr=0.001,0.003,0.01 --wait
+```
+
+Every combination of the lists you give, so a second `--param` is a grid
+rather than a second sweep. Where the set you want is not a product, build the
+list yourself and post it:
 
 ```sh
 curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \
@@ -578,7 +602,8 @@ is the same stack.
   artifacts, sweeps, durability, what happens when your engine dies mid-training
 - [Writing node code](../code/nodes.md) — generators, settings, what a node may
   and may not do
-- [The command line](../code/cli.md) — `login`, `sync`, `run`, `runs` in full
+- [The command line](../code/cli.md) — `login`, `sync`, `run`, `runs`, `sweep`
+  in full
 - [Remote workers](../code/workers.md) — send the training node to the GPU box
   and keep the rest on your laptop
 - [The flow editor](../interface/flow-editor.md) — once you have a portal, this