Stage caching for batch runs, and an engine that lives in the command
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m5s
Playwright Tests / test-playwright (2, 2) (push) Failing after 20s
pre-commit / pre-commit (push) Failing after 2m33s
Test Backend / test-backend (push) Successful in 2m7s
Compose Smoke Test / test-compose (push) Failing after 20s
Playwright Tests / merge-reports (push) Failing after 1m3s
Publish / publish (push) Failing after 12s
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m5s
Playwright Tests / test-playwright (2, 2) (push) Failing after 20s
pre-commit / pre-commit (push) Failing after 2m33s
Test Backend / test-backend (push) Successful in 2m7s
Compose Smoke Test / test-compose (push) Failing after 20s
Playwright Tests / merge-reports (push) Failing after 1m3s
Publish / publish (push) Failing after 12s
A code node in a batch run is now fingerprinted by its source, its raw settings and the values it reads — an artifact input counting as its digest, which is what the content addressing was always for. A run that finds the key restores what the earlier one returned and skips the node, recorded as `cached`. The run history is the cache: `run_node.outputs` beside the `cache_key` the schema already had, no second store. On for code nodes, never for the built-in and connector types that have side effects; off per node with `@node(cache=False)` and per run with `--no-cache`. Emissions are not replayed on a hit, so a cached training node returns its result without redrawing its curve. Recorded in NOTEPAD.md with the two other deliberate limits. `fluksio run --local` boots the real app in the command's own process and drives it through its ASGI interface behind the ordinary client, so a run no longer needs a `serve` terminal beside it — same data directory, same history, and the cache carries between the two. It always waits, because the engine it starts lives exactly as long as the command. Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists, `run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited run rather than abandoning it, coloured statuses on a terminal, and `name` made optional on the metrics endpoint so a follower can ask for every series. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+39
-1
@@ -66,6 +66,9 @@ curl -X POST $FLUKSIO/runs/flows/train_polymer_gnn/sweep \
|
||||
}'
|
||||
```
|
||||
|
||||
From a terminal that is `fluksio sweep train_polymer_gnn --param lr=0.1,0.3`,
|
||||
which builds the product of the lists you give it and posts the same call.
|
||||
|
||||
They share a `group_id`, so `GET /api/v1/runs?group=…` is the sweep, and they
|
||||
execute in parallel. That is safe because **each run has a state backend of its
|
||||
own**: message names are global keys, so two runs of one flow would otherwise
|
||||
@@ -126,7 +129,8 @@ def process():
|
||||
|
||||
Every number a node emits is kept as the run's series, stepped by the count of
|
||||
emissions on that message. Read one back with
|
||||
`GET /api/v1/runs/{id}/metrics?name=<flow>.loss`, or compare runs:
|
||||
`GET /api/v1/runs/{id}/metrics?name=<flow>.loss` — or leave `name` off for
|
||||
every series the run kept — or compare runs:
|
||||
|
||||
```
|
||||
GET /api/v1/runs/series/compare?ids=<a>,<b>,<c>&metric=<flow>.loss
|
||||
@@ -168,6 +172,40 @@ one preprocessed input stores it once, and a reference stays valid wherever
|
||||
the store is reachable from. Artifacts a run produced are listed on it and
|
||||
downloadable at `GET /api/v1/artifacts/{digest}`.
|
||||
|
||||
## Stage caching
|
||||
|
||||
A run mostly does not redo what an earlier one already did. Before a node
|
||||
executes it is fingerprinted — a sha256 over its source, its settings and the
|
||||
values it is about to read — and if some earlier run of that same fingerprint
|
||||
finished, what that one returned is restored into this run's state and the node
|
||||
is skipped. It is recorded with the status `cached` and a duration of zero, and
|
||||
its artifacts are listed on the new run as well, so they stay downloadable from
|
||||
either.
|
||||
|
||||
The settings go into the key raw, so a secret contributes its `{"$secret":
|
||||
name}` reference and never its value. An artifact input counts as its content
|
||||
digest: the same bytes under a different filename are the same input. Each node
|
||||
on a run carries the `cache_key` it was looked up by.
|
||||
|
||||
The run history *is* the cache; there is no second store. A node's returned
|
||||
outputs are kept on its run record as canonical JSON, up to 256000 characters —
|
||||
a node returning more than that is simply not cacheable that run. An entry
|
||||
whose artifact bytes have since left the store is a miss, not an error.
|
||||
|
||||
Only `python` nodes are cached, and by default all of them are. A built-in node
|
||||
type or a connector node has side effects and no source to fingerprint, so
|
||||
neither is ever a candidate. Turn it off for one node with
|
||||
`@node(..., cache=False)` — the flow document carries it as `cache`, so the
|
||||
canvas and the API can change it too — or for one run with
|
||||
`fluksio run --no-cache`, `fluksio sweep --no-cache`, or `"no_cache": true` in
|
||||
the submission body.
|
||||
|
||||
What a cached node does not bring back is what it emitted on the way. Its
|
||||
returned outputs are restored; the values it published mid-execution are not,
|
||||
because those were the story of an execution that is not happening this time.
|
||||
So a skipped training node contributes no loss curve to the new run — if you
|
||||
want the curve, that run has to actually train.
|
||||
|
||||
## Objects that cannot be serialized
|
||||
|
||||
A live model, a `DataLoader`, a JAX-compiled function — these do not cross a
|
||||
|
||||
Reference in New Issue
Block a user