diff --git a/docs/code/cli.md b/docs/code/cli.md index 03c6aa9..9f8b0dc 100644 --- a/docs/code/cli.md +++ b/docs/code/cli.md @@ -7,13 +7,19 @@ pip install fluksio Installs the engine and the `fluksio` command. Python 3.12 or newer, Linux or macOS. +The MQTT and InfluxDB connectors, outbound mail and error reporting are +`pip install 'fluksio[server]'` — a deployment talking to devices wants them, +and a laptop waiting on them to install does not. Everything else, the engine +and every python node included, is in the plain install; a node type whose +library is missing says which extra to add when one is actually built. + There is a second, smaller distribution — `fluksio-worker` — for a machine that should only *run nodes* for an engine elsewhere. It has none of the engine in it. See [Remote workers](workers.md). The command is two things at once: `serve`, `enroll` and `worker` *are* an -installation, while `login`, `sync`, `run`, `runs`, `sweep` and `status` talk to -one that may be anywhere. +installation, while `login`, `sync`, `run`, `runs`, `artifacts`, `sweep` and +`status` talk to one that may be anywhere. ## Where an installation lives @@ -39,22 +45,41 @@ On the first start it creates an admin account and prints its password **once**. Nothing else has to be running: no database server, no message broker, no Docker. +At a terminal this opens a dashboard with the engine running under it; see +[below](#the-dashboard). `--plain` prints the log stream instead, which is +also what happens with no terminal — in a container, under systemd, or in CI. + The default port moves out of the way when something already has it — 8001, 8002, and so on — and says which one it took; the URL written to `client.json` is the one it is actually on. A port you *asked* for is never moved off: `--port 9000` on a taken 9000 fails, because something else is there and you named it. +What it will *not* do is start a second engine for the same installation. If +the port is held by an engine already serving this directory, it says so and +stops — one SQLite database wants one engine. Another installation's Fluksio +on that port is named, and the move happens as usual. + | Option | Default | What it does | |---|---|---| | `--data-dir PATH` | `./.fluksio` (or `$FLUKSIO_HOME`) | where this installation keeps everything | | `--host HOST` | `127.0.0.1` | what to bind | | `--port PORT` | `8000`, or the next free one | what to listen on | +| `--plain` | off at a terminal | the log stream rather than the dashboard | | `--log-level LEVEL` | `info` | uvicorn's log level | | `--admin-email ADDR` | `admin@example.com` | the account created on first run | | `--admin-password PW` | generated | set it instead of having one generated | | `--enroll CODE` | — | pair with a portal as part of coming up | | `--portal URL` | — | the portal `--enroll` redeems at | +| `--max-runs N` | 4 | batch runs driven at once (`FLOW_MAX_RUNS`) | +| `--max-cascades N` | 4 | cascades in flight at once (`FLOW_MAX_CASCADES`) | +| `--max-workers N` | 4 | python worker processes (`FLOW_MAX_WORKERS`) | +| `--gpus N` | 0 | GPUs on this machine a node may be given (`FLOW_GPUS`) | + +Cards are declared rather than detected — asking a vendor's tooling would make +one dependency two — so a machine with a GPU reports none until `--gpus` says +otherwise, and a node asking for one is clamped to zero and runs alongside +every other. `--gpus 1` is what serialises them. `--enroll` with `--portal` is the one-command setup: it pairs before the engine starts, so the connection is dialled as part of coming up rather than needing a @@ -88,6 +113,29 @@ Fluksio 0.1.0 — data in /home/you/.fluksio An enrolled installation says which portal it is on instead, and notes that the dashboard is served from there rather than here. +### The dashboard + +At a terminal, `serve` draws the health overview, the recent runs, and the +engine's own log in a pane below — the output above is in there, not replaced +by it. + +| Key | What it does | +|---|---| +| `q` | close the dashboard. **The engine keeps running**, and the pid is printed | +| `s` | stop the engine, or start it again | +| `r` | restart it | +| `c` | cancel the run the cursor is on | +| `e` | pair with a portal, without leaving the screen | + +The engine is a child process rather than a thread, which is what makes those +possible — and what makes `q` a way out of the screen rather than a way to +stop the engine. Running `fluksio serve` again reattaches to it. + +An engine started elsewhere is adopted rather than duplicated, and can be +stopped from here only when it is this installation's own: both the pidfile +beside the data and a token this directory's key signed have to agree. Another +installation's engine is named and left alone. + ## `fluksio enroll` Pairs an existing installation with a portal. @@ -160,12 +208,14 @@ file path, because the shim has to import the same way. A plain directory is walked all the way down, so one folder per study — `fluksio sync dev` over `dev/s1_baseline/study.py` — needs no naming. Hidden -directories, `__pycache__`, `node_modules` and virtualenvs are left alone. Two -files that would import under the same name are refused rather than -silently collapsed into one: Python keeps one module per name, and a node's -generated body imports by that name, so `dev/s1/study.py` and `dev/s2/study.py` -need an `__init__.py` each — making them `s1.study` and `s2.study` — or -different filenames. +directories, `__pycache__`, `node_modules` and virtualenvs are left alone. + +Each file is imported under the name its path spells beneath the directory +being synced, so `dev/s1/study.py` and `dev/s2/study.py` are `s1.study` and +`s2.study` and a `study.py` per study collides with nothing. No `__init__.py` +is needed — the directories in between are namespace packages — which leaves a +bare `from study import ...` in a test beside it working. A file at the top of +what is synced keeps its plain name. | Flag | What it does | |---|---| @@ -206,7 +256,9 @@ flow nothing changed in, so what the walk costs is importing the others. is not free, and `--no-sync` skips it entirely. Flags that are not its own are the flow's inputs, typed by what the flow -declares them as. `--wait` blocks until the run +declares them as — so a name the flow has not got is refused by name, and +`--param lr=0.002` is told that one value is `--lr 0.002` and several is a +[sweep](#fluksio-sweep). `--wait` blocks until the run finishes and exits non-zero if it failed. `--follow` waits as well, and prints the numbers the run reports as they arrive: @@ -309,10 +361,10 @@ engine is the command itself, so nothing changes under it. fluksio runs [--flow train] [--limit 20] ``` -The runs an engine has recorded, newest first: id, status, flow, duration, the -commit of the repository it came from, and the inputs it was given. Statuses -are coloured when a terminal is reading the output — `ok` green, `error` red, -`cached` cyan. +The runs an engine has recorded, newest first: id, status, flow, duration, how +long ago it was submitted, the commit of the repository it came from, and the +inputs it was given. Statuses are coloured when a terminal is reading the +output — `ok` green, `error` red, `cached` cyan. Only the inputs that *differ from what the flow declares* are shown, and they are clamped to what is left of the terminal's width — a run that took the @@ -359,18 +411,21 @@ point of *each* curve. `export runs` is the wide one: a row per run with its inputs as columns, its final numbers, its status, its duration and the commit and digest of the code it ran. -The inputs that become columns are the ones that **vary** across the selected -runs — the axis of the sweep, which is what a comparison is read along — -unless `--params lr,seed` names them. `--metrics` narrows the final numbers -the same way. +Every input the selected runs recorded becomes a column, so the schema does +not move with the selection and a filter written against one export keeps +working on the next; `--params lr,seed` narrows it to the axis a comparison is +read along. `--metrics` narrows the final numbers the same way. + +An input left out of a submit is recorded at the value the flow declares for +it, so every row says what it was actually run with rather than leaving the +cell blank. A node usually returns a record rather than a scalar, so both sides take dotted paths into one: `--metrics final_metrics.train_loss,test_metrics.known.perfect` selects three fields rather than two blobs, and `--params model.ansatz` does the same for an input. The defaults reach the same depth — every number a -result carries becomes a column wherever it sits, and inputs are compared -leaf by leaf, so two configurations differing in one field give that field -rather than two records that are merely not equal. +result carries becomes a column wherever it sits, and a record's inputs are +taken leaf by leaf rather than as one blob. Metric names are flow-qualified — a node of `train` writing `train_loss` records `train.train_loss` — so `--list` prints the names the selected runs @@ -387,6 +442,21 @@ stream. In a notebook, `Client.export_metrics()` and `Client.export_runs()` answer the same rows as a list of dicts, which `pandas.DataFrame` takes directly. +### `fluksio artifacts` + +```sh +fluksio artifacts 1758042000123-9f2ab41c +fluksio artifacts 1758042000123-9f2ab41c weights -o model.npz +``` + +The files a run produced — what `fluksio.save_artifact(...)` wrote, and any +artifact a node returned. Named alone it lists them: the message each left on, +its size, its media type and the filename the node gave it. Name one and it is +written here, under that filename unless `-o` says otherwise. + +The message name is the one to pass, since it is what addresses the bytes; +`--local` reads them from this directory without an engine served. + ## What lives in the data directory ```text diff --git a/docs/code/nodes.md b/docs/code/nodes.md index 249151b..ab5bf12 100644 --- a/docs/code/nodes.md +++ b/docs/code/nodes.md @@ -110,6 +110,18 @@ Because the address is the content's hash, a sweep whose fifty configs share one preprocessed input stores it once, and a reference stays valid wherever the store is reachable from — including on another machine. +`fluksio artifacts RUN NAME` is how one comes back out at a terminal. + +### `fluksio.logger` + +A node's `print` is kept as that node's logs, and so is anything on +`fluksio.logger` — the same logger the SDK exports at top level, so code that +runs both inside a node and outside one says it the same way: + +```python +fluksio.logger.info("resuming from epoch %d", start) +``` + ### Media Say what the bytes are and the port can be typed for them: diff --git a/docs/getting-started/data-science.md b/docs/getting-started/data-science.md index d156971..80c97be 100644 --- a/docs/getting-started/data-science.md +++ b/docs/getting-started/data-science.md @@ -249,6 +249,11 @@ stops your own code from running. - **The declaration is checked against the function.** A port with no matching parameter, or a parameter that is neither port nor setting, is an error when the module is imported — not when the node is first called. +- **The decorator hands the function back untouched.** `fit(dataset, lr=0.05)` + is an ordinary call, and the module is an ordinary script — so a + `if __name__ == "__main__":` check beside the nodes runs. Syncing one + defined in a script *run as* `__main__` is what cannot work, since the + generated body would have no name to import it by. !!! note "Where a `yield` cannot reach" @@ -294,6 +299,16 @@ nothing else is given that card while it runs. string whose contents depend on the version you have installed, so writing it for you would silently replace whatever you had put there. +The engine has to be told how many cards it has — `fluksio serve --gpus 1`, or +`FLOW_GPUS` — because detecting them would mean depending on a vendor's +tooling. Until it is, a node asking for one is quietly given zero and runs +alongside every other; the log says so the first time it happens. + +Workers are kept warm on purpose, so a library that takes most of the card at +import would hold it after the run finished. The workers that ran on a card +are retired when the run ends, which gives the memory back; what it costs is +the next GPU run paying for its imports again. + Declaring nothing is the default and is right for most nodes — a poll, a threshold, a message on its way somewhere. Those share the engine's worker pool and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index ace2df9..323426e 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -134,10 +134,11 @@ warning into a refusal to start. | `ARTIFACT_GC_GRACE_S` | `3600` | how long a freshly written artifact is spared, whatever refers to it | The three concurrency limits are also flags on `fluksio serve` — `--max-workers`, -`--max-cascades`, `--max-runs` — which outrank the file, and the engine says -which numbers it started with in its first lines. Each is a pool size, so each -must be at least 1: a `0` is refused at startup by name rather than read as the -default. Leave one empty (or unset) to get the default. +`--max-cascades`, `--max-runs` — as is the card count, `--gpus`. The flags +outrank the file, and the engine says which numbers it started with in its +first lines. Each pool size must be at least 1 and the card count at least 0: +a number below that is refused as a flag error naming it, rather than read as +the default. Leave one empty (or unset) to get the default. An artifact is referred to by a run that recorded it or by a message currently holding it; anything else is what a camera published four hours ago, and the