Stage caching for batch runs, and an engine that lives in the command
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m5s
Playwright Tests / test-playwright (2, 2) (push) Failing after 20s
pre-commit / pre-commit (push) Failing after 2m33s
Test Backend / test-backend (push) Successful in 2m7s
Compose Smoke Test / test-compose (push) Failing after 20s
Playwright Tests / merge-reports (push) Failing after 1m3s
Publish / publish (push) Failing after 12s
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m5s
Playwright Tests / test-playwright (2, 2) (push) Failing after 20s
pre-commit / pre-commit (push) Failing after 2m33s
Test Backend / test-backend (push) Successful in 2m7s
Compose Smoke Test / test-compose (push) Failing after 20s
Playwright Tests / merge-reports (push) Failing after 1m3s
Publish / publish (push) Failing after 12s
A code node in a batch run is now fingerprinted by its source, its raw settings and the values it reads — an artifact input counting as its digest, which is what the content addressing was always for. A run that finds the key restores what the earlier one returned and skips the node, recorded as `cached`. The run history is the cache: `run_node.outputs` beside the `cache_key` the schema already had, no second store. On for code nodes, never for the built-in and connector types that have side effects; off per node with `@node(cache=False)` and per run with `--no-cache`. Emissions are not replayed on a hit, so a cached training node returns its result without redrawing its curve. Recorded in NOTEPAD.md with the two other deliberate limits. `fluksio run --local` boots the real app in the command's own process and drives it through its ASGI interface behind the ordinary client, so a run no longer needs a `serve` terminal beside it — same data directory, same history, and the cache carries between the two. It always waits, because the engine it starts lives exactly as long as the command. Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists, `run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited run rather than abandoning it, coloured statuses on a terminal, and `name` made optional on the metrics endpoint so a follower can ask for every series. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+43
-6
@@ -12,8 +12,8 @@ should only *run nodes* for an engine elsewhere. It has none of the engine in
|
||||
it. See [Remote workers](workers.md).
|
||||
|
||||
The command is two things at once: `serve`, `enroll` and `worker` *are* an
|
||||
installation, while `login`, `sync`, `run` and `runs` talk to one that may be
|
||||
anywhere.
|
||||
installation, while `login`, `sync`, `run`, `runs` and `sweep` talk to one that
|
||||
may be anywhere.
|
||||
|
||||
## Where an installation lives
|
||||
|
||||
@@ -122,8 +122,9 @@ See [Remote workers](workers.md).
|
||||
|
||||
## Talking to an engine
|
||||
|
||||
The four commands below are the client half: they run wherever you work, and
|
||||
address an engine over its API rather than being one.
|
||||
The commands below are the client half: they run wherever you work, and
|
||||
address an engine over its API rather than being one — except under `--local`,
|
||||
which boots one inside the command instead.
|
||||
|
||||
### `fluksio login`
|
||||
|
||||
@@ -171,13 +172,34 @@ fluksio run train --lr 0.05 --seed 7 [--wait]
|
||||
Syncs the working directory, then submits a run — so the command after an edit
|
||||
is this one and nothing else. Flags that are not its own are the flow's
|
||||
inputs, typed by what the flow declares them as. `--wait` blocks until the run
|
||||
finishes and exits non-zero if it failed.
|
||||
finishes and exits non-zero if it failed. `--follow` waits as well, and prints
|
||||
the numbers the run reports as they arrive:
|
||||
|
||||
```text
|
||||
train.loss[14] = 3.40295e-06
|
||||
```
|
||||
|
||||
Ctrl-C while either is waiting cancels the run on the engine rather than only
|
||||
stopping the watching, and exits 130.
|
||||
|
||||
`--no-sync` runs what is already on the engine. Worth it in a tight loop where
|
||||
you know nothing changed, since syncing retires the workers and the next call
|
||||
pays its imports again. A directory that declares no flows syncs nothing and
|
||||
says nothing — a flow drawn on the canvas is run the same way.
|
||||
|
||||
`--no-cache` executes every node, including one an earlier run already
|
||||
answered. See [Stage caching](../concepts/runs.md#stage-caching).
|
||||
|
||||
`--local` boots the engine inside this process instead of talking to a served
|
||||
one, so there is no `fluksio serve` terminal to keep open. It is the same
|
||||
installation either way — the same `.fluksio`, the same database, artifacts
|
||||
and run history — so a run made this way and a run made through a served
|
||||
engine cache against each other. It always waits, because the engine it starts
|
||||
lives exactly as long as the command. Starting one costs a few seconds of
|
||||
worker pool and module reconcile, against the ~15 ms of submitting to an
|
||||
engine that is already up: `--local` is for "I just want to run it", not for a
|
||||
loop you are iterating in.
|
||||
|
||||
### `fluksio runs`
|
||||
|
||||
```sh
|
||||
@@ -185,7 +207,22 @@ fluksio runs [--flow train] [--limit 20]
|
||||
```
|
||||
|
||||
The runs an engine has recorded, newest first: id, status, flow, duration, the
|
||||
commit of the repository it came from, and its parameters.
|
||||
commit of the repository it came from, and its parameters. Statuses are
|
||||
coloured when a terminal is reading the output — `ok` green, `error` red,
|
||||
`cached` cyan. `--local` reads the same history from an in-process engine,
|
||||
without one having to be served.
|
||||
|
||||
### `fluksio sweep`
|
||||
|
||||
```sh
|
||||
fluksio sweep train --param lr=0.1,0.01 --param epochs=10,50 --wait
|
||||
```
|
||||
|
||||
Every combination of the parameter lists, submitted as one group — four runs
|
||||
above, sharing a `group_id` and executing in parallel. Values are typed by the
|
||||
flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`,
|
||||
`--no-cache` and `--local` mean what they do there. `--wait` blocks until all
|
||||
of them are finished and exits non-zero if any failed.
|
||||
|
||||
## What lives in the data directory
|
||||
|
||||
|
||||
Reference in New Issue
Block a user