Document the dashboard, the artifacts command and what moved with them
Docs / docs (push) Successful in 33s
Playwright Tests / test-playwright (1, 2) (push) Failing after 2m56s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m46s
pre-commit / pre-commit (push) Failing after 1m59s
Test Backend / test-backend (push) Failing after 2m23s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m18s
Docs / docs (push) Successful in 33s
Playwright Tests / test-playwright (1, 2) (push) Failing after 2m56s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m46s
pre-commit / pre-commit (push) Failing after 1m59s
Test Backend / test-backend (push) Failing after 2m23s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m18s
Covers this round of CLI work: the serve dashboard and its keys, the refusal to start a second engine for one installation, --plain, --gpus and the declared-not-detected card count, `fluksio artifacts`, the age column, `fluksio.logger` inside a node, the server extra, per-directory study module names, and an export whose columns no longer move with the selection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc
This commit is contained in:
+90
-20
@@ -7,13 +7,19 @@ pip install fluksio
|
||||
Installs the engine and the `fluksio` command. Python 3.12 or newer, Linux or
|
||||
macOS.
|
||||
|
||||
The MQTT and InfluxDB connectors, outbound mail and error reporting are
|
||||
`pip install 'fluksio[server]'` — a deployment talking to devices wants them,
|
||||
and a laptop waiting on them to install does not. Everything else, the engine
|
||||
and every python node included, is in the plain install; a node type whose
|
||||
library is missing says which extra to add when one is actually built.
|
||||
|
||||
There is a second, smaller distribution — `fluksio-worker` — for a machine that
|
||||
should only *run nodes* for an engine elsewhere. It has none of the engine in
|
||||
it. See [Remote workers](workers.md).
|
||||
|
||||
The command is two things at once: `serve`, `enroll` and `worker` *are* an
|
||||
installation, while `login`, `sync`, `run`, `runs`, `sweep` and `status` talk to
|
||||
one that may be anywhere.
|
||||
installation, while `login`, `sync`, `run`, `runs`, `artifacts`, `sweep` and
|
||||
`status` talk to one that may be anywhere.
|
||||
|
||||
## Where an installation lives
|
||||
|
||||
@@ -39,22 +45,41 @@ On the first start it creates an admin account and prints its password **once**.
|
||||
Nothing else has to be running: no database server, no message broker, no
|
||||
Docker.
|
||||
|
||||
At a terminal this opens a dashboard with the engine running under it; see
|
||||
[below](#the-dashboard). `--plain` prints the log stream instead, which is
|
||||
also what happens with no terminal — in a container, under systemd, or in CI.
|
||||
|
||||
The default port moves out of the way when something already has it — 8001,
|
||||
8002, and so on — and says which one it took; the URL written to
|
||||
`client.json` is the one it is actually on. A port you *asked* for is never
|
||||
moved off: `--port 9000` on a taken 9000 fails, because something else is
|
||||
there and you named it.
|
||||
|
||||
What it will *not* do is start a second engine for the same installation. If
|
||||
the port is held by an engine already serving this directory, it says so and
|
||||
stops — one SQLite database wants one engine. Another installation's Fluksio
|
||||
on that port is named, and the move happens as usual.
|
||||
|
||||
| Option | Default | What it does |
|
||||
|---|---|---|
|
||||
| `--data-dir PATH` | `./.fluksio` (or `$FLUKSIO_HOME`) | where this installation keeps everything |
|
||||
| `--host HOST` | `127.0.0.1` | what to bind |
|
||||
| `--port PORT` | `8000`, or the next free one | what to listen on |
|
||||
| `--plain` | off at a terminal | the log stream rather than the dashboard |
|
||||
| `--log-level LEVEL` | `info` | uvicorn's log level |
|
||||
| `--admin-email ADDR` | `admin@example.com` | the account created on first run |
|
||||
| `--admin-password PW` | generated | set it instead of having one generated |
|
||||
| `--enroll CODE` | — | pair with a portal as part of coming up |
|
||||
| `--portal URL` | — | the portal `--enroll` redeems at |
|
||||
| `--max-runs N` | 4 | batch runs driven at once (`FLOW_MAX_RUNS`) |
|
||||
| `--max-cascades N` | 4 | cascades in flight at once (`FLOW_MAX_CASCADES`) |
|
||||
| `--max-workers N` | 4 | python worker processes (`FLOW_MAX_WORKERS`) |
|
||||
| `--gpus N` | 0 | GPUs on this machine a node may be given (`FLOW_GPUS`) |
|
||||
|
||||
Cards are declared rather than detected — asking a vendor's tooling would make
|
||||
one dependency two — so a machine with a GPU reports none until `--gpus` says
|
||||
otherwise, and a node asking for one is clamped to zero and runs alongside
|
||||
every other. `--gpus 1` is what serialises them.
|
||||
|
||||
`--enroll` with `--portal` is the one-command setup: it pairs before the engine
|
||||
starts, so the connection is dialled as part of coming up rather than needing a
|
||||
@@ -88,6 +113,29 @@ Fluksio 0.1.0 — data in /home/you/.fluksio
|
||||
An enrolled installation says which portal it is on instead, and notes that the
|
||||
dashboard is served from there rather than here.
|
||||
|
||||
### The dashboard
|
||||
|
||||
At a terminal, `serve` draws the health overview, the recent runs, and the
|
||||
engine's own log in a pane below — the output above is in there, not replaced
|
||||
by it.
|
||||
|
||||
| Key | What it does |
|
||||
|---|---|
|
||||
| `q` | close the dashboard. **The engine keeps running**, and the pid is printed |
|
||||
| `s` | stop the engine, or start it again |
|
||||
| `r` | restart it |
|
||||
| `c` | cancel the run the cursor is on |
|
||||
| `e` | pair with a portal, without leaving the screen |
|
||||
|
||||
The engine is a child process rather than a thread, which is what makes those
|
||||
possible — and what makes `q` a way out of the screen rather than a way to
|
||||
stop the engine. Running `fluksio serve` again reattaches to it.
|
||||
|
||||
An engine started elsewhere is adopted rather than duplicated, and can be
|
||||
stopped from here only when it is this installation's own: both the pidfile
|
||||
beside the data and a token this directory's key signed have to agree. Another
|
||||
installation's engine is named and left alone.
|
||||
|
||||
## `fluksio enroll`
|
||||
|
||||
Pairs an existing installation with a portal.
|
||||
@@ -160,12 +208,14 @@ file path, because the shim has to import the same way.
|
||||
|
||||
A plain directory is walked all the way down, so one folder per study —
|
||||
`fluksio sync dev` over `dev/s1_baseline/study.py` — needs no naming. Hidden
|
||||
directories, `__pycache__`, `node_modules` and virtualenvs are left alone. Two
|
||||
files that would import under the same name are refused rather than
|
||||
silently collapsed into one: Python keeps one module per name, and a node's
|
||||
generated body imports by that name, so `dev/s1/study.py` and `dev/s2/study.py`
|
||||
need an `__init__.py` each — making them `s1.study` and `s2.study` — or
|
||||
different filenames.
|
||||
directories, `__pycache__`, `node_modules` and virtualenvs are left alone.
|
||||
|
||||
Each file is imported under the name its path spells beneath the directory
|
||||
being synced, so `dev/s1/study.py` and `dev/s2/study.py` are `s1.study` and
|
||||
`s2.study` and a `study.py` per study collides with nothing. No `__init__.py`
|
||||
is needed — the directories in between are namespace packages — which leaves a
|
||||
bare `from study import ...` in a test beside it working. A file at the top of
|
||||
what is synced keeps its plain name.
|
||||
|
||||
| Flag | What it does |
|
||||
|---|---|
|
||||
@@ -206,7 +256,9 @@ flow nothing changed in, so what the walk costs is importing the others.
|
||||
is not free, and `--no-sync` skips it entirely.
|
||||
|
||||
Flags that are not its own are the flow's inputs, typed by what the flow
|
||||
declares them as. `--wait` blocks until the run
|
||||
declares them as — so a name the flow has not got is refused by name, and
|
||||
`--param lr=0.002` is told that one value is `--lr 0.002` and several is a
|
||||
[sweep](#fluksio-sweep). `--wait` blocks until the run
|
||||
finishes and exits non-zero if it failed. `--follow` waits as well, and prints
|
||||
the numbers the run reports as they arrive:
|
||||
|
||||
@@ -309,10 +361,10 @@ engine is the command itself, so nothing changes under it.
|
||||
fluksio runs [--flow train] [--limit 20]
|
||||
```
|
||||
|
||||
The runs an engine has recorded, newest first: id, status, flow, duration, the
|
||||
commit of the repository it came from, and the inputs it was given. Statuses
|
||||
are coloured when a terminal is reading the output — `ok` green, `error` red,
|
||||
`cached` cyan.
|
||||
The runs an engine has recorded, newest first: id, status, flow, duration, how
|
||||
long ago it was submitted, the commit of the repository it came from, and the
|
||||
inputs it was given. Statuses are coloured when a terminal is reading the
|
||||
output — `ok` green, `error` red, `cached` cyan.
|
||||
|
||||
Only the inputs that *differ from what the flow declares* are shown, and they
|
||||
are clamped to what is left of the terminal's width — a run that took the
|
||||
@@ -359,18 +411,21 @@ point of *each* curve. `export runs` is the wide one: a row per run with its
|
||||
inputs as columns, its final numbers, its status, its duration and the commit
|
||||
and digest of the code it ran.
|
||||
|
||||
The inputs that become columns are the ones that **vary** across the selected
|
||||
runs — the axis of the sweep, which is what a comparison is read along —
|
||||
unless `--params lr,seed` names them. `--metrics` narrows the final numbers
|
||||
the same way.
|
||||
Every input the selected runs recorded becomes a column, so the schema does
|
||||
not move with the selection and a filter written against one export keeps
|
||||
working on the next; `--params lr,seed` narrows it to the axis a comparison is
|
||||
read along. `--metrics` narrows the final numbers the same way.
|
||||
|
||||
An input left out of a submit is recorded at the value the flow declares for
|
||||
it, so every row says what it was actually run with rather than leaving the
|
||||
cell blank.
|
||||
|
||||
A node usually returns a record rather than a scalar, so both sides take
|
||||
dotted paths into one: `--metrics final_metrics.train_loss,test_metrics.known.perfect`
|
||||
selects three fields rather than two blobs, and `--params model.ansatz` does
|
||||
the same for an input. The defaults reach the same depth — every number a
|
||||
result carries becomes a column wherever it sits, and inputs are compared
|
||||
leaf by leaf, so two configurations differing in one field give that field
|
||||
rather than two records that are merely not equal.
|
||||
result carries becomes a column wherever it sits, and a record's inputs are
|
||||
taken leaf by leaf rather than as one blob.
|
||||
|
||||
Metric names are flow-qualified — a node of `train` writing `train_loss`
|
||||
records `train.train_loss` — so `--list` prints the names the selected runs
|
||||
@@ -387,6 +442,21 @@ stream. In a notebook, `Client.export_metrics()` and `Client.export_runs()`
|
||||
answer the same rows as a list of dicts, which `pandas.DataFrame` takes
|
||||
directly.
|
||||
|
||||
### `fluksio artifacts`
|
||||
|
||||
```sh
|
||||
fluksio artifacts 1758042000123-9f2ab41c
|
||||
fluksio artifacts 1758042000123-9f2ab41c weights -o model.npz
|
||||
```
|
||||
|
||||
The files a run produced — what `fluksio.save_artifact(...)` wrote, and any
|
||||
artifact a node returned. Named alone it lists them: the message each left on,
|
||||
its size, its media type and the filename the node gave it. Name one and it is
|
||||
written here, under that filename unless `-o` says otherwise.
|
||||
|
||||
The message name is the one to pass, since it is what addresses the bytes;
|
||||
`--local` reads them from this directory without an engine served.
|
||||
|
||||
## What lives in the data directory
|
||||
|
||||
```text
|
||||
|
||||
Reference in New Issue
Block a user