Make the docs state things rather than argue them
Docs / docs (push) Successful in 37s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m35s
Playwright Tests / test-playwright (2, 2) (push) Failing after 17s
pre-commit / pre-commit (push) Failing after 2m8s
Test Backend / test-backend (push) Failing after 2m48s
Compose Smoke Test / test-compose (push) Failing after 13s
Playwright Tests / merge-reports (push) Failing after 2m25s
Docs / docs (push) Successful in 37s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m35s
Playwright Tests / test-playwright (2, 2) (push) Failing after 17s
pre-commit / pre-commit (push) Failing after 2m8s
Test Backend / test-backend (push) Failing after 2m48s
Compose Smoke Test / test-compose (push) Failing after 13s
Playwright Tests / merge-reports (push) Failing after 2m25s
The site read as a design journal: rationale paragraphs, hedges
("deliberately", "on purpose", "genuinely"), meta-commentary about the docs
themselves, and one em-dash every ten lines carrying an aside.
Roughly twenty rationale blocks are gone or reduced to what a reader needs
in order to use the thing. Em-dashes go from 507 to 135, and what is left is
structural rather than prose: list and definition separators, table cells,
and four inside code blocks that quote what the CLI actually prints.
Also: api.example.com becomes api.fluksio.com (the emails stay, since
bootstrap.py really defaults to admin@example.com and RFC 2606 reserves it);
the mqtt table gains the two settings it had drifted behind on and inject's
wording matches the engine; llms.txt lists the two connector pages that were
in the nav but not in it; and the two device/device_policy notes now agree.
Builds clean under `zensical build --strict`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YrQnKV3bnQd4K342y8tKj
This commit is contained in:
+12
-12
@@ -11,16 +11,16 @@ build, publish and run flows.
|
||||
MCP_ENABLED=true
|
||||
```
|
||||
|
||||
It is off by default, because switching it on opens client registration to
|
||||
whoever can reach the host. Turn it on deliberately.
|
||||
It is off by default: switching it on opens client registration to whoever can
|
||||
reach the host.
|
||||
|
||||
The endpoint is `https://api.${DOMAIN}/mcp` — or `http://api.localhost/mcp` on
|
||||
a local stack.
|
||||
The endpoint is `https://api.${DOMAIN}/mcp`, or `http://api.localhost/mcp` on a
|
||||
local stack.
|
||||
|
||||
## Connecting a client
|
||||
|
||||
```sh
|
||||
claude mcp add --transport http fluksio https://api.example.com/mcp
|
||||
claude mcp add --transport http fluksio https://api.fluksio.com/mcp
|
||||
```
|
||||
|
||||
The OAuth flow does the rest: the client registers itself, you approve it in
|
||||
@@ -30,7 +30,7 @@ the browser, and it exchanges the grant for a token. Nothing to paste.
|
||||
|
||||
An agent acts as **the person who approved it**, and only through the same REST
|
||||
API the dashboard uses. It cannot do anything you could not do in the browser,
|
||||
and it cannot do anything *you* cannot do — an agent approved by a non-superuser
|
||||
and it cannot do anything *you* cannot do. An agent approved by a non-superuser
|
||||
is not one either.
|
||||
|
||||
Its tokens are signed with a keypair of their own, kept beside the flow store.
|
||||
@@ -53,7 +53,7 @@ Read-only:
|
||||
| `get_message_history` | one message's recent numeric values |
|
||||
| `get_graph` | every flow as one graph |
|
||||
| `list_shared_nodes` | the shared library and its usages |
|
||||
| `list_secrets` | secret **names** — never values |
|
||||
| `list_secrets` | secret **names**, never values |
|
||||
| `get_modules` | the packages node code can import |
|
||||
| `get_health` | flows, nodes, queue and loop lag |
|
||||
| `get_metrics` | executions, errors and timings per minute |
|
||||
@@ -87,10 +87,10 @@ Operating:
|
||||
The tools do not reach into the engine. Each one calls the same REST endpoint
|
||||
the dashboard calls, over an in-process transport.
|
||||
|
||||
That keeps exactly one description of what a flow is and how it may be changed
|
||||
— the validation, the draft/publish split, the version check that stops two
|
||||
clients overwriting each other. An agent that saved a flow through a private
|
||||
back door would be an agent that could write one the canvas cannot open.
|
||||
That keeps exactly one description of what a flow is and how it may be changed:
|
||||
the validation, the draft/publish split, and the version check that stops two
|
||||
clients overwriting each other. An agent saving a flow through a private back
|
||||
door could write one the canvas cannot open.
|
||||
|
||||
The caller's token rides along on every hop, so the API sees the agent's own
|
||||
identity rather than a service account. The audit trail on Home names it.
|
||||
@@ -105,7 +105,7 @@ you what the canvas would have told you.
|
||||
## Limits
|
||||
|
||||
- MCP is not currently reachable through a [portal](../interface/portal.md)
|
||||
tunnel — the proxy forwards `/api/v1/` only. Connect an agent on the same
|
||||
tunnel: the proxy forwards `/api/v1/` only. Connect an agent on the same
|
||||
network as the instance.
|
||||
- Secrets are never readable, by an agent or by anyone else. `list_secrets`
|
||||
returns names.
|
||||
|
||||
+13
-14
@@ -1,7 +1,7 @@
|
||||
# The HTTP API
|
||||
|
||||
Everything the browser does, the API does first. The dashboard is a generated
|
||||
client of this schema, not a privileged path into the engine — so anything you
|
||||
client of this schema, not a privileged path into the engine, so anything you
|
||||
can click, you can script.
|
||||
|
||||
Base URL: `https://api.${DOMAIN}/api/v1`, or `http://127.0.0.1:8000/api/v1` for
|
||||
@@ -24,7 +24,7 @@ A bearer token, valid for eight days. `POST /login/test-token` checks one.
|
||||
Password recovery and reset are `POST /password-recovery/{email}` and
|
||||
`POST /reset-password/`.
|
||||
|
||||
Agents authenticate differently — see [Agents over MCP](agents.md).
|
||||
Agents authenticate differently; see [Agents over MCP](agents.md).
|
||||
|
||||
!!! tip "The interactive schema"
|
||||
|
||||
@@ -33,9 +33,8 @@ Agents authenticate differently — see [Agents over MCP](agents.md).
|
||||
`/api/v1/openapi.json`. That is the authoritative reference; this page is
|
||||
the tour.
|
||||
|
||||
It is closed in production on purpose: the schema enumerates every endpoint
|
||||
the instance serves, including the paths webhook nodes mounted at
|
||||
runtime.
|
||||
It is closed in production: the schema enumerates every endpoint the
|
||||
instance serves, including paths webhook nodes mounted at runtime.
|
||||
|
||||
## Flows
|
||||
|
||||
@@ -81,8 +80,8 @@ shared sources and which flows use each.
|
||||
| `POST` | `/flows/{name}/nodes/{id}/acknowledge` | clear the failure the canvas is marking |
|
||||
|
||||
`POST /flows/{name}/run` on a **batch** flow submits a run instead, because
|
||||
that is what running one means — the parameters, the series and the result are
|
||||
the point, and a call that quietly did something else would be a trap.
|
||||
that is what running one means: the parameters, the series and the result are
|
||||
the point.
|
||||
|
||||
## Messages
|
||||
|
||||
@@ -95,7 +94,7 @@ curl -s $FLUKSIO/messages/house.temperature/history -H "Authorization: Bearer $T
|
||||
|
||||
`GET /messages/` is the catalogue: every message any published flow declares,
|
||||
with its last value. Publishing puts a value into the graph exactly as a
|
||||
dashboard control does — which means only a message some flow *declares* can be
|
||||
dashboard control does, which means only a message some flow *declares* can be
|
||||
published to. Flows own the namespace; everything else is a client of it.
|
||||
|
||||
## Runs
|
||||
@@ -113,8 +112,8 @@ published to. Flows own the namespace; everything else is a client of it.
|
||||
| `GET` | `/runs/{id}/metrics?name=&stride=` | one metric's series, in step order; every series of the run without `name` |
|
||||
| `GET` | `/runs/series/compare?ids=a,b,c&metric=&x=` | that metric across several runs. `x` is what to plot against: nothing or `step`, `time` (seconds since each run's own first reading), or another metric's name, joined on the step the two share |
|
||||
|
||||
Submitting answers immediately with a `queued` run. Wrong parameters — an
|
||||
undeclared name, a value of the wrong type — come back as a 422 naming the
|
||||
Submitting answers immediately with a `queued` run. Wrong parameters (an
|
||||
undeclared name, a value of the wrong type) come back as a 422 naming the
|
||||
problem, before anything executes.
|
||||
|
||||
`?digest=` filters by the hash of the parameters, which is how you find "every
|
||||
@@ -122,13 +121,13 @@ run that used exactly this configuration".
|
||||
|
||||
`?before=` is how a long history is paged: rows come newest first, so handing
|
||||
back the last row's `created_at` reads the next page whatever landed
|
||||
meanwhile — which `?offset=` cannot, since a run submitted between two pages
|
||||
meanwhile, which `?offset=` cannot, since a run submitted between two pages
|
||||
shifts every row down one. `?since=` is inclusive and bounds the other end.
|
||||
|
||||
Both exports stream `csv` (the default) or `jsonl`, and take the selection the
|
||||
history takes plus `?ids=a,b,c`, `?since=` and `?until=`. `export/runs` puts
|
||||
the inputs that *vary* across the selection in `param.` columns — the sweep
|
||||
axis — unless `?params=` names them, and the run's numbers in `metric.`
|
||||
the inputs that *vary* across the selection in `param.` columns (the sweep
|
||||
axis) unless `?params=` names them, and the run's numbers in `metric.`
|
||||
columns. Both are dotted paths into whatever a node returned, so
|
||||
`?metrics=final_metrics.train_loss` selects a field of a record and the
|
||||
defaults reach every number inside one. The run id is on every row of both, which is what makes an exported
|
||||
@@ -239,4 +238,4 @@ stored version moved past the one you were editing. Re-read, merge, save again.
|
||||
|
||||
The frontend's TypeScript client is generated from the OpenAPI schema
|
||||
(`make generate-client`). Any OpenAPI generator will do the same for your
|
||||
language — point it at `/api/v1/openapi.json` on a non-production instance.
|
||||
language: point it at `/api/v1/openapi.json` on a non-production instance.
|
||||
|
||||
+62
-65
@@ -8,12 +8,12 @@ Installs the engine and the `fluksio` command. Python 3.12 or newer, Linux or
|
||||
macOS.
|
||||
|
||||
The MQTT and InfluxDB connectors, outbound mail and error reporting are
|
||||
`pip install 'fluksio[server]'` — a deployment talking to devices wants them,
|
||||
`pip install 'fluksio[server]'`: a deployment talking to devices wants them,
|
||||
and a laptop waiting on them to install does not. Everything else, the engine
|
||||
and every python node included, is in the plain install; a node type whose
|
||||
library is missing says which extra to add when one is actually built.
|
||||
|
||||
There is a second, smaller distribution — `fluksio-worker` — for a machine that
|
||||
There is a second, smaller distribution, `fluksio-worker`, for a machine that
|
||||
should only *run nodes* for an engine elsewhere. It has none of the engine in
|
||||
it. See [Remote workers](workers.md).
|
||||
|
||||
@@ -26,7 +26,7 @@ instance, while `login`, `sync`, `run`, `runs`, `artifacts`, `sweep` and
|
||||
`.fluksio` beside your code, found the way `.git` is: from the working
|
||||
directory, or any directory above it. Two repositories on one machine are
|
||||
therefore two engines, with their own flows, runs and token. `fluksio serve`
|
||||
makes one where there is none, and it ignores itself from within — a
|
||||
makes one where there is none, and it ignores itself from within with a
|
||||
`.gitignore` of `*`, so a database and a credential cannot be committed by
|
||||
accident.
|
||||
|
||||
@@ -47,17 +47,17 @@ Docker.
|
||||
|
||||
At a terminal this opens a dashboard with the engine running under it; see
|
||||
[below](#the-dashboard). `--plain` prints the log stream instead, which is
|
||||
also what happens with no terminal — in a container, under systemd, or in CI.
|
||||
also what happens with no terminal: in a container, under systemd, or in CI.
|
||||
|
||||
The default port moves out of the way when something already has it — 8001,
|
||||
8002, and so on — and says which one it took; the URL written to
|
||||
The default port moves out of the way when something already has it (8001,
|
||||
8002, and so on) and says which one it took; the URL written to
|
||||
`client.json` is the one it is actually on. A port you *asked* for is never
|
||||
moved off: `--port 9000` on a taken 9000 fails, because something else is
|
||||
there and you named it.
|
||||
|
||||
What it will *not* do is start a second engine for the same instance. If
|
||||
the port is held by an engine already serving this directory, it says so and
|
||||
stops — one SQLite database wants one engine. Another instance's Fluksio
|
||||
stops, since one SQLite database wants one engine. Another instance's Fluksio
|
||||
on that port is named, and the move happens as usual.
|
||||
|
||||
| Option | Default | What it does |
|
||||
@@ -76,8 +76,8 @@ on that port is named, and the move happens as usual.
|
||||
| `--max-workers N` | 4 | python worker processes (`FLOW_MAX_WORKERS`) |
|
||||
| `--gpus N` | 0 | GPUs on this machine a node may be given (`FLOW_GPUS`) |
|
||||
|
||||
Cards are declared rather than detected — asking a vendor's tooling would make
|
||||
one dependency two — so a machine with a GPU reports none until `--gpus` says
|
||||
Cards are declared rather than detected, since asking a vendor's tooling would
|
||||
make one dependency two, so a machine with a GPU reports none until `--gpus` says
|
||||
otherwise, and a node asking for one is clamped to zero and runs alongside
|
||||
every other. `--gpus 1` is what serialises them.
|
||||
|
||||
@@ -87,7 +87,7 @@ restart. It is skipped if the instance is already enrolled.
|
||||
|
||||
!!! warning "One process"
|
||||
|
||||
`fluksio serve` holds the flow engine. A second one is a *second engine* —
|
||||
`fluksio serve` holds the flow engine. A second one is a *second engine*:
|
||||
duplicated subscriptions, duplicated cron ticks, two webhooks answering the
|
||||
same path. Run one, and distribute work with
|
||||
[workers](workers.md) instead.
|
||||
@@ -95,7 +95,7 @@ restart. It is skipped if the instance is already enrolled.
|
||||
!!! note "`$HOME` on a cluster"
|
||||
|
||||
A login node's home directory is often NFS, where SQLite's write-ahead log
|
||||
does not work — the database would be locked or corrupt. `fluksio serve`
|
||||
does not work, since the database would be locked or corrupt. `fluksio serve`
|
||||
warns when it notices; point `--data-dir` at local disk.
|
||||
|
||||
### What it prints
|
||||
@@ -116,8 +116,8 @@ dashboard is served from there rather than here.
|
||||
### The dashboard
|
||||
|
||||
At a terminal, `serve` draws the health overview, the recent runs, and the
|
||||
engine's own log in a pane below — the output above is in there, not replaced
|
||||
by it.
|
||||
engine's own log in a pane below, so the output above is in there rather than
|
||||
replaced by it.
|
||||
|
||||
| Key | What it does |
|
||||
|---|---|
|
||||
@@ -131,7 +131,7 @@ by it.
|
||||
| `e` | pair with a portal, without leaving the screen |
|
||||
|
||||
The engine is a child process rather than a thread, which is what makes those
|
||||
possible — and what makes `q` a way out of the screen rather than a way to
|
||||
possible, and what makes `q` a way out of the screen rather than a way to
|
||||
stop the engine. Running `fluksio serve` again reattaches to it.
|
||||
|
||||
An engine started elsewhere is adopted rather than duplicated, and can be
|
||||
@@ -140,30 +140,27 @@ beside the data and a token this directory's key signed have to agree. Another
|
||||
instance's engine is named and left alone.
|
||||
|
||||
The screen subscribes to the engine's event bus over the same websocket a
|
||||
browser uses, so a run appears the moment it starts rather than at the next
|
||||
poll — which is what used to make a run that started and finished inside five
|
||||
seconds visible only in the history. It reconnects on its own, quietly: an
|
||||
engine stopped from this screen is a normal state, not an error to fill the
|
||||
log pane with.
|
||||
browser uses, so a run appears the moment it starts. It reconnects on its own,
|
||||
and an engine stopped from this screen is a normal state rather than an error.
|
||||
|
||||
#### Comparing runs
|
||||
|
||||
`enter` opens a comparison of the ticked runs: one metric's curve for each,
|
||||
over a table of what differs between them.
|
||||
|
||||
The curves are drawn in braille, five to a chart, in five distinct colours —
|
||||
a deliberate departure from the browser's chart palette, where the five series
|
||||
The curves are drawn in braille, five to a chart, in five distinct colours: a
|
||||
departure from the browser's chart palette, where the five series
|
||||
step one hue by lightness. Hue is what survives a terminal. Every curve is
|
||||
named in the legend either way.
|
||||
|
||||
Two pickers sit above the chart. The first is the metric, named as the run
|
||||
records it — `train.loss` rather than `loss`, since a name is qualified by the
|
||||
node that published it. The second is what it is plotted against: the step,
|
||||
records it (`train.loss` rather than `loss`, since a name is qualified by the
|
||||
node that published it). The second is what it is plotted against: the step,
|
||||
`time (s)` measured from each run's own first reading so runs started hours
|
||||
apart lie on top of each other, or another metric of the same runs.
|
||||
|
||||
The table below keeps only what actually differs — the inputs whose values are
|
||||
not the same in every run, and the seed when it varies — beside each run's
|
||||
The table below keeps only what actually differs (the inputs whose values are
|
||||
not the same in every run, and the seed when it varies) beside each run's
|
||||
status, duration and final numbers. A parameter every run shared is not a
|
||||
column worth scanning.
|
||||
|
||||
@@ -186,7 +183,7 @@ fluksio enroll ABCD-1234
|
||||
|
||||
Get the code from the portal under **Instances → Add instance**. It is
|
||||
single-use and expires in fifteen minutes. `--as` matters when the instance
|
||||
has several superusers — without it, enrolment refuses rather than guessing.
|
||||
has several superusers. Without it, enrolment refuses rather than guessing.
|
||||
|
||||
Afterwards, `fluksio serve` dials the portal as it comes up, and keeps dialling:
|
||||
a portal that restarts, a wifi that changes, a laptop that suspends and wakes
|
||||
@@ -198,11 +195,11 @@ minute. See [Accounts and the portal](../interface/portal.md).
|
||||
## `fluksio worker`
|
||||
|
||||
Runs nodes for an engine elsewhere. Everything after `worker` belongs to the
|
||||
agent's own parser — it is the same program `fluksio-worker` installs, so the
|
||||
agent's own parser: it is the same program `fluksio-worker` installs, so the
|
||||
two are interchangeable:
|
||||
|
||||
```sh
|
||||
fluksio worker --url wss://api.example.com/api/v1/workers/attach \
|
||||
fluksio worker --url wss://api.fluksio.com/api/v1/workers/attach \
|
||||
--token "$FLUKSIO_WORKER_TOKEN" --labels gpu
|
||||
```
|
||||
|
||||
@@ -211,21 +208,21 @@ See [Remote workers](workers.md).
|
||||
## Talking to an engine
|
||||
|
||||
The commands below are the client half: they run wherever you work, and
|
||||
address an engine over its API rather than being one — except under `--local`,
|
||||
address an engine over its API rather than being one, except under `--local`,
|
||||
which boots one inside the command instead.
|
||||
|
||||
### `fluksio login`
|
||||
|
||||
```sh
|
||||
fluksio login --url https://api.example.com
|
||||
fluksio login --url https://api.fluksio.com
|
||||
```
|
||||
|
||||
For an engine somewhere *else*. One you started yourself needs no login:
|
||||
`fluksio serve` writes the token as it comes up and says where it put it.
|
||||
|
||||
The token goes in this project's `.fluksio/client.json`, or with `--global` in
|
||||
`~/.fluksio/client.json`. Every command below reads it from there — nearest
|
||||
first, walking up from the working directory — or from `FLUKSIO_URL` and
|
||||
`~/.fluksio/client.json`. Every command below reads it from there, nearest
|
||||
first walking up from the working directory, or from `FLUKSIO_URL` and
|
||||
`FLUKSIO_TOKEN`, or from its own `--url` and `--token`. A token an older
|
||||
version wrote to `~/.config/fluksio/client.json` is still read.
|
||||
|
||||
@@ -240,14 +237,14 @@ each one with a generated import shim per node. A directory that is a package
|
||||
is walked; a dotted name is imported as it stands; nothing is loaded from a
|
||||
file path, because the shim has to import the same way.
|
||||
|
||||
A plain directory is walked all the way down, so one folder per study —
|
||||
`fluksio sync dev` over `dev/s1_baseline/study.py` — needs no naming. Hidden
|
||||
A plain directory is walked all the way down, so one folder per study
|
||||
(`fluksio sync dev` over `dev/s1_baseline/study.py`) needs no naming. Hidden
|
||||
directories, `__pycache__`, `node_modules` and virtualenvs are left alone.
|
||||
|
||||
Each file is imported under the name its path spells beneath the directory
|
||||
being synced, so `dev/s1/study.py` and `dev/s2/study.py` are `s1.study` and
|
||||
`s2.study` and a `study.py` per study collides with nothing. No `__init__.py`
|
||||
is needed — the directories in between are namespace packages — which leaves a
|
||||
is needed, since the directories in between are namespace packages, which leaves a
|
||||
bare `from study import ...` in a test beside it working. A file at the top of
|
||||
what is synced keeps its plain name.
|
||||
|
||||
@@ -258,18 +255,18 @@ what is synced keeps its plain name.
|
||||
| `--force` | overwrite a flow, or a node body, that was edited on the canvas |
|
||||
|
||||
Every sync retires the engine's workers, including one that had nothing to
|
||||
upload — a worker holds your package in memory, so an edit to it is invisible
|
||||
upload: a worker holds your package in memory, so an edit to it is invisible
|
||||
until the process goes.
|
||||
|
||||
It also records, per node, which of your modules that node's function imports
|
||||
its way to, and what they hash to. That is what the
|
||||
[stage cache](../concepts/runs.md#stage-caching) keys on, so editing a
|
||||
helper a node calls into is reported as that node changing — `train: updated
|
||||
(flow, fit)` — and re-runs it, while editing something the node never reaches
|
||||
helper a node calls into is reported as that node changing (`train: updated
|
||||
(flow, fit)`) and re-runs it, while editing something the node never reaches
|
||||
is left alone.
|
||||
|
||||
A sync that changed nothing says `unchanged`, which is the answer worth
|
||||
having. `— published` and `— draft` are said only when there was something to
|
||||
having. `published` and `draft` are said only when there was something to
|
||||
publish or a draft was genuinely left behind. An engine too old to store what
|
||||
a node's code reaches says so in a line naming both versions; until it is
|
||||
upgraded its cache is keyed on the whole repository, as it was before. See
|
||||
@@ -281,7 +278,7 @@ upgraded its cache is keyed on the whole repository, as it was before. See
|
||||
fluksio run train --lr 0.05 --seed 7 [--wait]
|
||||
```
|
||||
|
||||
Syncs the working directory *and everything under it*, then submits a run — so
|
||||
Syncs the working directory *and everything under it*, then submits a run, so
|
||||
the command after an edit is this one and nothing else, from the repository
|
||||
root as readily as from the study's own folder. A study that will not import
|
||||
is a warning rather than a stopped run; the upload is already a no-op for a
|
||||
@@ -290,7 +287,7 @@ flow nothing changed in, so what the walk costs is importing the others.
|
||||
is not free, and `--no-sync` skips it entirely.
|
||||
|
||||
Flags that are not its own are the flow's inputs, typed by what the flow
|
||||
declares them as — so a name the flow has not got is refused by name, and
|
||||
declares them as, so a name the flow has not got is refused by name, and
|
||||
`--param lr=0.002` is told that one value is `--lr 0.002` and several is a
|
||||
[sweep](#fluksio-sweep). `--wait` blocks until the run
|
||||
finishes and exits non-zero if it failed. `--follow` waits as well, and prints
|
||||
@@ -309,13 +306,13 @@ timeout.
|
||||
on the run, so what a result came from is answerable later. It goes into the
|
||||
digest that identifies a run's inputs, so two runs of one configuration with
|
||||
different seeds are different runs rather than a cache hit. And if the flow
|
||||
declares an input named `seed`, that is what fills it — so the number the run
|
||||
declares an input named `seed`, that is what fills it, so the number the run
|
||||
is labelled with is the number your code actually drew from, instead of merely
|
||||
looking like it. A flow that declares no such input still records it, and
|
||||
nothing reads it. Sweep over seeds with `--param seed=1,2,3`.
|
||||
|
||||
Any input takes what a previous run produced, named rather than typed out —
|
||||
a checkpoint, but equally a config object nobody wants to paste into a shell:
|
||||
Any input takes what a previous run produced, named rather than typed out: a
|
||||
checkpoint, but equally a config object nobody wants to paste into a shell:
|
||||
|
||||
```sh
|
||||
fluksio run evaluate --dataset @run:1758042000123-9f2ab41c.dataset
|
||||
@@ -327,7 +324,7 @@ fluksio run train --meta @run:1758042000123-9f2ab41c.dataset_meta
|
||||
type; a bare digest is the content itself, resolved into a reference. Both
|
||||
spellings are reserved on every input, `str` included, so an input that has to
|
||||
carry one of them literally cannot. Passing the value as JSON still works and
|
||||
is what a script that already holds one does — the same thing
|
||||
is what a script that already holds one does, the same thing
|
||||
`flow.submit(dataset=run.result["dataset"])` does from Python.
|
||||
|
||||
Run a flow with no parameters at a terminal and it asks for them, one line per
|
||||
@@ -346,15 +343,15 @@ the command, skips the questions, and `--defaults` skips them explicitly.
|
||||
`--no-sync` runs what is already on the engine. Worth it in a tight loop where
|
||||
you know nothing changed, since syncing retires the workers and the next call
|
||||
pays its imports again. A directory that declares no flows syncs nothing and
|
||||
says nothing — a flow drawn on the canvas is run the same way.
|
||||
says nothing: a flow drawn on the canvas is run the same way.
|
||||
|
||||
`--no-cache` executes every node, including one an earlier run already
|
||||
answered. See [Stage caching](../concepts/runs.md#stage-caching).
|
||||
|
||||
`--local` boots the engine inside this process instead of talking to a served
|
||||
one, so there is no `fluksio serve` terminal to keep open. It is the same
|
||||
instance either way — the same `.fluksio`, the same database, artifacts
|
||||
and run history — so a run made this way and a run made through a served
|
||||
instance either way (the same `.fluksio`, the same database, artifacts and run
|
||||
history) so a run made this way and a run made through a served
|
||||
engine cache against each other. It always waits, because the engine it starts
|
||||
lives exactly as long as the command. Starting one costs a few seconds of
|
||||
worker pool and module reconcile, against the ~15 ms of submitting to an
|
||||
@@ -378,11 +375,11 @@ engine that accounts for nothing.
|
||||
|
||||
The portal reads one of three ways. `no portal` means this instance was
|
||||
never enrolled. `portal hub.fluksio.com` means the link is up. `portal
|
||||
unreachable` names the error, and is the one worth acting on — the dashboard is
|
||||
unreachable` names the error, and is the one worth acting on: the dashboard is
|
||||
served from the other end, so nobody can reach it while that is showing.
|
||||
|
||||
`--watch` keeps it on screen and refreshes every five seconds until Ctrl-C —
|
||||
the cadence the dashboard polls at, since nothing here moves faster. It needs a
|
||||
`--watch` keeps it on screen and refreshes every five seconds until Ctrl-C, the
|
||||
cadence the dashboard polls at, since nothing here moves faster. It needs a
|
||||
terminal; without one, run it without `--watch` and the output pipes cleanly.
|
||||
|
||||
`--local` reads the flows and history out of this directory with no engine
|
||||
@@ -398,10 +395,10 @@ fluksio runs [--flow train] [--limit 20]
|
||||
The runs an engine has recorded, newest first: id, status, flow, duration, how
|
||||
long ago it was submitted, the commit of the repository it came from, and the
|
||||
inputs it was given. Statuses are coloured when a terminal is reading the
|
||||
output — `ok` green, `error` red, `cached` cyan.
|
||||
output: `ok` green, `error` red, `cached` cyan.
|
||||
|
||||
Only the inputs that *differ from what the flow declares* are shown, and they
|
||||
are clamped to what is left of the terminal's width — a run that took the
|
||||
are clamped to what is left of the terminal's width. A run that took the
|
||||
defaults lists none at all, and a flow taking a few kB of JSON does not push
|
||||
everything else off the line. `Client.runs()` and
|
||||
[`fluksio export runs`](#fluksio-export) are where the whole value is read.
|
||||
@@ -414,7 +411,7 @@ to be served.
|
||||
fluksio flavors
|
||||
```
|
||||
|
||||
The named sizes a node can ask for — `@node(resources="gpu-small")` — with the
|
||||
The named sizes a node can ask for, as in `@node(resources="gpu-small")`, with the
|
||||
cores, memory and cards each stands for. Editing them is the Workers screen or
|
||||
`POST /api/v1/flavors`; this is the read.
|
||||
|
||||
@@ -424,7 +421,7 @@ cores, memory and cards each stands for. Editing them is the Workers screen or
|
||||
fluksio sweep train --param lr=0.1,0.01 --param epochs=10,50 --wait
|
||||
```
|
||||
|
||||
Every combination of the parameter lists, submitted as one group — four runs
|
||||
Every combination of the parameter lists, submitted as one group: four runs
|
||||
above, sharing a `group_id` and executing in parallel. Values are typed by the
|
||||
flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`,
|
||||
`--no-cache` and `--local` mean what they do there. `--wait` blocks until all
|
||||
@@ -438,8 +435,8 @@ fluksio export metrics --flow train --name train.train_loss --stride 10 -o curve
|
||||
fluksio export runs --flow train --status ok > arms.csv
|
||||
```
|
||||
|
||||
The two tables an analysis reads. `export metrics` is the long one — a row per
|
||||
run, metric and step — which is what a plotting library takes without
|
||||
The two tables an analysis reads. `export metrics` is the long one, a row per
|
||||
run, metric and step, which is what a plotting library takes without
|
||||
reshaping; `--name` keeps the metrics it lists and `--stride` keeps every Nth
|
||||
point of *each* curve. `export runs` is the wide one: a row per run with its
|
||||
inputs as columns, its final numbers, its status, its duration and the commit
|
||||
@@ -457,12 +454,12 @@ cell blank.
|
||||
A node usually returns a record rather than a scalar, so both sides take
|
||||
dotted paths into one: `--metrics final_metrics.train_loss,test_metrics.known.perfect`
|
||||
selects three fields rather than two blobs, and `--params model.ansatz` does
|
||||
the same for an input. The defaults reach the same depth — every number a
|
||||
the same for an input. The defaults reach the same depth: every number a
|
||||
result carries becomes a column wherever it sits, and a record's inputs are
|
||||
taken leaf by leaf rather than as one blob.
|
||||
|
||||
Metric names are flow-qualified — a node of `train` writing `train_loss`
|
||||
records `train.train_loss` — so `--list` prints the names the selected runs
|
||||
Metric names are flow-qualified (a node of `train` writing `train_loss` records
|
||||
`train.train_loss`) so `--list` prints the names the selected runs
|
||||
carry when the spelling is not obvious.
|
||||
|
||||
Both take `--flow`, `--run ID` (repeat it), `--group`, `--status`, `--since`,
|
||||
@@ -470,8 +467,8 @@ Both take `--flow`, `--run ID` (repeat it), `--group`, `--status`, `--since`,
|
||||
back to the run page and to what the run made.
|
||||
|
||||
`--format` is `csv` (the default), `jsonl` or `parquet`; output goes to stdout
|
||||
unless `-o FILE` names somewhere. Parquet keeps the types and needs pyarrow —
|
||||
`pip install 'fluksio[parquet]'` — and a file to write, since it is not a
|
||||
unless `-o FILE` names somewhere. Parquet keeps the types, needs pyarrow
|
||||
(`pip install 'fluksio[parquet]'`) and needs a file to write, since it is not a
|
||||
stream. In a notebook, `Client.export_metrics()` and `Client.export_runs()`
|
||||
answer the same rows as a list of dicts, which `pandas.DataFrame` takes
|
||||
directly.
|
||||
@@ -483,7 +480,7 @@ fluksio artifacts 1758042000123-9f2ab41c
|
||||
fluksio artifacts 1758042000123-9f2ab41c weights -o model.npz
|
||||
```
|
||||
|
||||
The files a run produced — what `fluksio.save_artifact(...)` wrote, and any
|
||||
The files a run produced: what `fluksio.save_artifact(...)` wrote, and any
|
||||
artifact a node returned. Named alone it lists them: the message each left on,
|
||||
its size, its media type and the filename the node gave it. Name one and it is
|
||||
written here, under that filename unless `-o` says otherwise.
|
||||
@@ -496,9 +493,9 @@ The message name is the one to pass, since it is what addresses the bytes;
|
||||
```text
|
||||
.fluksio/ (or ~/.fluksio, with `--global`)
|
||||
├── client.json the token `serve` wrote, mode 600
|
||||
├── .gitignore `*` — a database and a credential, ignored from within
|
||||
├── .gitignore `*`, so a database and a credential are ignored from within
|
||||
├── fluksio.db SQLite: users, runs, metrics, observability, agents
|
||||
├── flows/ a git repository — one directory per flow
|
||||
├── flows/ a git repository, one directory per flow
|
||||
│ ├── house/
|
||||
│ │ ├── flow.json the published structure
|
||||
│ │ ├── nodes/*.py the published node code
|
||||
|
||||
@@ -36,8 +36,8 @@ uv add --editable ../../fluksio-connector-mydevice
|
||||
```
|
||||
|
||||
Editing your connector's **code** then takes effect on the next engine restart,
|
||||
with no reinstall. Changing its **metadata** — the entry point, the version,
|
||||
the package name — needs `uv sync` to regenerate the installed metadata before
|
||||
with no reinstall. Changing its **metadata** (the entry point, the version,
|
||||
the package name) needs `uv sync` to regenerate the installed metadata before
|
||||
a restart picks it up.
|
||||
|
||||
Two things to know about this:
|
||||
@@ -84,7 +84,7 @@ class MyDevice(ConnectorNode):
|
||||
|
||||
Then, on the canvas: place the node, set `poll_interval` to how often the
|
||||
device should be read, pick the API key from the secrets store, and add output
|
||||
ports named `temperature` and `humidity`. The port names are the wiring — any
|
||||
ports named `temperature` and `humidity`. The port names are the wiring: any
|
||||
node consuming those messages is downstream of this one.
|
||||
|
||||
## If the device is told rather than asked
|
||||
@@ -102,7 +102,7 @@ message drives the device. Set `idempotent = False` on the class: a write is a
|
||||
command, and a redelivery after a crash should not undo a newer one.
|
||||
|
||||
Keep the first version off the wire. A boolean setting the code checks before
|
||||
it sends — `artnet`'s `transmit` is the example — lets a flow be built and
|
||||
it sends (`artnet`'s `transmit` is the example) lets a flow be built and
|
||||
watched in the logs before anything physically moves.
|
||||
|
||||
## If the reading is bytes
|
||||
@@ -117,8 +117,8 @@ A camera or a microphone publishes a reference rather than the bytes:
|
||||
|
||||
Place the node with an `image`-typed output port and a Media widget draws each
|
||||
frame as it lands. `fluksio-connector-test-media` publishes test frames and
|
||||
tones this way, so the whole path can be wired up with no camera in the room —
|
||||
copy it if yours is a media device.
|
||||
tones this way, so the whole path can be wired up with no camera in the room.
|
||||
Copy it if yours is a media device.
|
||||
|
||||
## Try it without a device
|
||||
|
||||
|
||||
+30
-31
@@ -1,7 +1,7 @@
|
||||
# Writing node code
|
||||
|
||||
A Function node is a Python file. That is all it is — no base class, no
|
||||
decorator, no framework import unless you want one.
|
||||
A Function node is a Python file: no base class, no decorator, no framework
|
||||
import unless you want one.
|
||||
|
||||
```python
|
||||
def process(temperature, setpoint=21.0):
|
||||
@@ -22,18 +22,18 @@ See [Where a node's values come from](../concepts/values.md).
|
||||
|
||||
**The return value is a dict keyed by output ports.** Every value is checked
|
||||
against the port's declared type before it is published. A key that is not a
|
||||
declared port is an error, not a silent drop — nothing leaves a node except
|
||||
declared port is an error, not a silent drop. Nothing leaves a node except
|
||||
through a port it declared.
|
||||
|
||||
**Nothing else is importable from the engine.** Node code runs in a separate
|
||||
process, on a separate interpreter, with none of Fluksio's own modules on its
|
||||
path. What it can import is what the [Modules](../interface/operations.md)
|
||||
screen installed — which can include [your own project](#your-own-code-as-a-package),
|
||||
screen installed, which can include [your own project](#your-own-code-as-a-package),
|
||||
so a node need not be a self-contained file.
|
||||
|
||||
**A node is a pure function of its inputs.** No context object, no global
|
||||
store, no handle to reach for. A running total or a debounce timer has a
|
||||
specific shape — see [Keeping state in a flow](../concepts/state.md).
|
||||
specific shape; see [Keeping state in a flow](../concepts/state.md).
|
||||
|
||||
## Producing values over time
|
||||
|
||||
@@ -49,7 +49,7 @@ def process(lr, steps):
|
||||
return {"final_loss": loss}
|
||||
```
|
||||
|
||||
Whatever the generator `return`s at the end is the node's result — what
|
||||
Whatever the generator `return`s at the end is the node's result, which is what
|
||||
downstream nodes read. If you never `return`, the last thing you yield is the
|
||||
result instead.
|
||||
|
||||
@@ -60,15 +60,15 @@ Mark the port so the flow says what it does:
|
||||
```
|
||||
|
||||
Two consequences. In a [run](../concepts/runs.md), the whole series is kept as
|
||||
that run's metrics — this is why there is no `log_metric()` anywhere in the
|
||||
that run's metrics, which is why there is no `log_metric()` anywhere in the
|
||||
API. And **the node's timeout starts measuring silence rather than duration**:
|
||||
each emission resets the deadline, so a node yielding every few seconds can run
|
||||
for hours under a timeout of 300.
|
||||
|
||||
### `fluksio.emit`
|
||||
|
||||
Where a `yield` cannot reach — the value comes from inside somebody else's
|
||||
callback, and they call you rather than the other way round:
|
||||
Where a `yield` cannot reach, because the value comes from inside somebody
|
||||
else's callback and they call you rather than the other way round:
|
||||
|
||||
```python
|
||||
import fluksio
|
||||
@@ -103,19 +103,19 @@ def process(dataset):
|
||||
```
|
||||
|
||||
`save_artifact` takes bytes or a path, stores them by their SHA-256 digest, and
|
||||
returns a small reference — digest, size, media type, name — which is what an
|
||||
returns a small reference (digest, size, media type, name) which is what an
|
||||
`artifact`-typed port carries.
|
||||
|
||||
Because the address is the content's hash, a sweep whose fifty configs share
|
||||
one preprocessed input stores it once, and a reference stays valid wherever the
|
||||
store is reachable from — including on another machine.
|
||||
store is reachable from, including on another machine.
|
||||
|
||||
`fluksio artifacts RUN NAME` is how one comes back out at a terminal.
|
||||
|
||||
### `fluksio.logger`
|
||||
|
||||
A node's `print` is kept as that node's logs, and so is anything on
|
||||
`fluksio.logger` — the same logger the SDK exports at top level, so code that
|
||||
`fluksio.logger` is the same logger the SDK exports at top level, so code that
|
||||
runs both inside a node and outside one says it the same way:
|
||||
|
||||
```python
|
||||
@@ -152,7 +152,7 @@ each carries and what rates are realistic.
|
||||
|
||||
Only what a node *returns* is recorded against its run. Frames yielded
|
||||
along the way are replaced in state by the next one, and the artifact sweep
|
||||
removes bytes nothing refers to any more — which is what stops a camera
|
||||
removes bytes nothing refers to any more, which is what stops a camera
|
||||
filling the disk. If a particular frame matters, return it.
|
||||
|
||||
## Printing
|
||||
@@ -161,14 +161,13 @@ each carries and what rates are realistic.
|
||||
the flow editor's log panel and on the run's per-node record; the rest is
|
||||
dropped, so a node printing in a loop cannot fill anything up.
|
||||
|
||||
Use it to debug. Do not use it to record results — a number worth keeping is an
|
||||
output port, not a line of text.
|
||||
Use it to debug, not to record results. A number worth keeping is an output
|
||||
port, not a line of text.
|
||||
|
||||
## Errors
|
||||
|
||||
An exception fails that node's execution, not the flow. The message you see is
|
||||
one line from the frame in *your* code, not a stack through the engine — that
|
||||
is a deliberate choice about what is actionable.
|
||||
one line from the frame in *your* code, not a stack through the engine.
|
||||
|
||||
The node keeps its last error visible after it recovers, so a failure that
|
||||
fired an alert at 03:00 still says what it was at 09:00. It can also be
|
||||
@@ -178,13 +177,13 @@ acknowledged from the canvas.
|
||||
|
||||
`timeout` on a node is how many seconds its code may be *silent* before it is
|
||||
stopped. A yield or an `emit` resets the clock, and the first call's imports are
|
||||
not charged to it — a node importing torch is not being slow, it is loading.
|
||||
not charged to it: a node importing torch is not being slow, it is loading.
|
||||
|
||||
There is no timeout by default. Training runs for hours and a node that reports
|
||||
nothing is usually working, so the engine waits: what fails a call is the worker
|
||||
dying, which arrives at once rather than after a deadline. Set a timeout on the
|
||||
nodes where silence means stuck — an HTTP call that should answer in seconds,
|
||||
a loop that can spin — either per node or, for all of them,
|
||||
nodes where silence means stuck (an HTTP call that should answer in seconds, a
|
||||
loop that can spin) either per node or, for all of them,
|
||||
with `FLOW_NODE_TIMEOUT`. `timeout = 0` says explicitly that this node has none.
|
||||
|
||||
## Running a node somewhere else
|
||||
@@ -197,7 +196,7 @@ A node declares the label of the machine it needs:
|
||||
|
||||
`require` (the default) waits for a worker carrying that label; `prefer` runs
|
||||
locally when none is attached. A node bound to a device is compiled *on that
|
||||
machine* — a node importing `torch` is correct on the GPU box and a missing
|
||||
machine*: a node importing `torch` is correct on the GPU box and a missing
|
||||
module on the engine, so checking it here would fail something that is fine.
|
||||
|
||||
See [Remote workers](workers.md).
|
||||
@@ -205,8 +204,8 @@ See [Remote workers](workers.md).
|
||||
## Sharing code between flows
|
||||
|
||||
A node's source can be promoted to the shared library from its panel, and other
|
||||
flows can then use it by reference. One copy, one place to edit — and every
|
||||
flow using it runs the edit, which is the point and also the caution.
|
||||
flows can then use it by reference. There is one copy and one place to edit, and
|
||||
every flow using it runs the edit.
|
||||
|
||||
Shared sources live in `_lib/` in the flow repository, so they are versioned
|
||||
with everything else.
|
||||
@@ -214,7 +213,7 @@ with everything else.
|
||||
## Packages
|
||||
|
||||
Node code runs in a virtual environment of its own, on the instance's data
|
||||
volume — deliberately separate from the one Fluksio itself runs on.
|
||||
volume, separate from the one Fluksio itself runs on.
|
||||
|
||||
Declare what you import in [Modules](../interface/operations.md), or over the
|
||||
API:
|
||||
@@ -233,7 +232,7 @@ flows. An install takes effect immediately; nothing restarts.
|
||||
!!! tip "If Fluksio is installed in the venv you work in, skip this"
|
||||
|
||||
Node code then runs on that environment, so your project and everything it
|
||||
imports are already importable — see
|
||||
imports are already importable; see
|
||||
[Getting started: data science](../getting-started/data-science.md). What
|
||||
follows is for a Fluksio with a venv of its own, which is what a container
|
||||
always has.
|
||||
@@ -246,7 +245,7 @@ installable like any other dependency:
|
||||
numpy>=2
|
||||
```
|
||||
|
||||
A node body then imports it, and the logic stays where it already lives — in
|
||||
A node body then imports it, and the logic stays where it already lives: in
|
||||
your repository, under your own version control, importing its own siblings:
|
||||
|
||||
```python
|
||||
@@ -263,7 +262,7 @@ modules as it likes, and nothing was copied.
|
||||
!!! tip "You can have those three lines written for you"
|
||||
|
||||
Decorate `fit` with `@node(...)` where it is defined, say which nodes make
|
||||
a flow with `Flow(...)`, and `fluksio sync` generates the body above —
|
||||
a flow with `Flow(...)`, and `fluksio sync` generates the body above,
|
||||
along with the flow document, so there is nothing to PUT by hand. The
|
||||
declaration lives beside the function it describes and is checked against
|
||||
its signature. See
|
||||
@@ -271,7 +270,7 @@ modules as it likes, and nothing was copied.
|
||||
|
||||
!!! warning "Editable, but not live"
|
||||
|
||||
`-e` means edits reach the venv without reinstalling — but a node's process
|
||||
`-e` means edits reach the venv without reinstalling, but a node's process
|
||||
already holds the imported module in memory. The engine's workers are
|
||||
long-lived, so a change to your code is picked up when they are retired,
|
||||
which is what **Apply** does. Pressing it after an edit is the loop.
|
||||
@@ -279,20 +278,20 @@ modules as it likes, and nothing was copied.
|
||||
A [worker](workers.md) you attach yourself is the exception: it starts a
|
||||
process per call, so it reads your code fresh every run. If you are
|
||||
iterating on the code many times an hour, point one at your own
|
||||
interpreter — `fluksio-worker --python "$(which python)"` — and mark the
|
||||
interpreter (`fluksio-worker --python "$(which python)"`) and mark the
|
||||
node with its label.
|
||||
|
||||
!!! note "The path is a deployment detail"
|
||||
|
||||
It is resolved on whichever machine runs the node, and the manifest is
|
||||
committed to the flow repository — so an absolute path from your laptop
|
||||
committed to the flow repository, so an absolute path from your laptop
|
||||
means nothing inside a container or on a GPU box. Those need their own
|
||||
install of the same project; a VCS requirement
|
||||
(`myresearch @ git+ssh://…@a1b2c3d`) travels where a path does not.
|
||||
|
||||
## A worked example
|
||||
|
||||
The repository ships a small supervised fit as a seedable demo — three nodes,
|
||||
The repository ships a small supervised fit as a seedable demo: three nodes,
|
||||
a batch flow, streaming metrics, artifacts between stages, and a GPU-labelled
|
||||
node that falls back to the engine when no worker is attached. It is the
|
||||
shortest complete thing to read:
|
||||
|
||||
+28
-30
@@ -6,7 +6,7 @@ network as the other.
|
||||
|
||||
A **worker** is a process that runs the code of nodes marked for it. It dials
|
||||
*out* to the engine over one authenticated websocket, so nothing on that
|
||||
machine has to be reachable — and nothing has to expose the engine's state
|
||||
machine has to be reachable, and nothing has to expose the engine's state
|
||||
backend across hosts, which it never should.
|
||||
|
||||
## Install and attach
|
||||
@@ -15,13 +15,13 @@ backend across hosts, which it never should.
|
||||
pip install fluksio-worker
|
||||
|
||||
fluksio-worker \
|
||||
--url wss://api.example.com/api/v1/workers/attach \
|
||||
--url wss://api.fluksio.com/api/v1/workers/attach \
|
||||
--token "$FLUKSIO_WORKER_TOKEN" \
|
||||
--labels gpu,cuda12 \
|
||||
--python /opt/torch-venv/bin/python
|
||||
```
|
||||
|
||||
`fluksio-worker` is its own distribution — the agent, the node runner, and
|
||||
`fluksio-worker` is its own distribution: the agent, the node runner, and
|
||||
`websockets`. Nothing of the engine, so a GPU box does not install a database
|
||||
driver in order to run a training step. An engine host already has it, and
|
||||
`fluksio worker …` is the same program.
|
||||
@@ -40,13 +40,13 @@ driver in order to run a training step. An engine host already has it, and
|
||||
| `--ram-mb` | what the job or the machine has | memory to advertise, in MB |
|
||||
| `--max-idle` | never | stop after this many seconds with nothing running |
|
||||
|
||||
`--python` is the important one. It is how this machine keeps its own wheels —
|
||||
the CUDA build, the vendor SDK, the thing that will not install anywhere else —
|
||||
`--python` is the important one. It is how this machine keeps its own wheels
|
||||
(the CUDA build, the vendor SDK, the thing that will not install anywhere else)
|
||||
without the engine ever installing them or knowing about them.
|
||||
|
||||
### What it says it has
|
||||
|
||||
A worker reports its inventory when it attaches — cores, GPUs and memory — and
|
||||
A worker reports its inventory when it attaches (cores, GPUs and memory) and
|
||||
the engine schedules against it: a node asking for two cores and a GPU goes to
|
||||
a machine that has them free, not merely to one carrying the right label.
|
||||
|
||||
@@ -58,12 +58,12 @@ or `FLUKSIO_WORKER_GPUS`) or something you say with `--gpus`. A worker that
|
||||
reports nothing still attaches and is scheduled by its label alone, as every
|
||||
worker was before any of them reported anything.
|
||||
|
||||
The engine tells each call what it may use — thread caps, and the devices it
|
||||
The engine tells each call what it may use: thread caps, and the devices it
|
||||
may see. The worker starts a process per call, so it applies them at the only
|
||||
moment a numerical library still reads them: before the import.
|
||||
|
||||
`--max-idle` is for a worker something else started for one job — a batch
|
||||
scheduler, say. It exits when nothing has run for that long, so the allocation
|
||||
`--max-idle` is for a worker something else started for one job, a batch
|
||||
scheduler say. It exits when nothing has run for that long, so the allocation
|
||||
goes back rather than idling until its walltime.
|
||||
|
||||
## Mint the token
|
||||
@@ -75,15 +75,15 @@ curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"name": "gpu-dev"}'
|
||||
```
|
||||
|
||||
Shown once, valid for a year — a worker is a machine somebody sets up and
|
||||
Shown once, valid for a year, since a worker is a machine somebody sets up and
|
||||
leaves running. It is signed with the same keypair agent tokens use, so
|
||||
rotating that key revokes every worker along with them.
|
||||
|
||||
??? note "A host where pip is not an option"
|
||||
|
||||
The two files work copied into one directory and run with `python agent.py
|
||||
…`. The engine serves the runner itself at `GET /api/v1/workers/runtime` —
|
||||
it is the same module its own local workers run, deliberately standard
|
||||
…`. The engine serves the runner itself at `GET /api/v1/workers/runtime`.
|
||||
It is the same module its own local workers run, deliberately standard
|
||||
library only.
|
||||
|
||||
## Send a node to it
|
||||
@@ -107,10 +107,10 @@ A node declares the label of the machine it needs:
|
||||
`prefer` is what makes a flow work before the GPU box exists. `require` is what
|
||||
you want once it does.
|
||||
|
||||
!!! note "Set from the API"
|
||||
!!! note "Not in the panel yet"
|
||||
|
||||
`device` and `device_policy` are not yet fields in the node panel. Set them
|
||||
with `PUT /flows/{name}`.
|
||||
`device` and `device_policy`, which machine a node's code runs on, are set
|
||||
through the API rather than the panel, with `PUT /flows/{name}`.
|
||||
|
||||
## What follows from this
|
||||
|
||||
@@ -120,15 +120,15 @@ you want once it does.
|
||||
`torch` is correct on the GPU box and a missing module on the engine, so
|
||||
checking it here would fail something that is fine.
|
||||
- **`import fluksio` inside a node is the worker's own reporter.** `emit`,
|
||||
`save_artifact`, `load_artifact` — installed before your code runs, so an
|
||||
`save_artifact`, `load_artifact`, installed before your code runs, so an
|
||||
installed `fluksio` package on that box never shadows it.
|
||||
- **Cancelling a run kills what it is executing**, there or here, and leaves
|
||||
other runs of the same node alone.
|
||||
- **If the worker disappears mid-call**, the run fails in seconds with
|
||||
`worker went away mid-call` rather than waiting out its timeout.
|
||||
- **A worker sends a heartbeat every ten seconds while it executes**, so a long
|
||||
node is distinguishable from a dead socket. Ninety seconds of nothing at all —
|
||||
not even a heartbeat — fails the call as gone. A heartbeat says the *agent* is
|
||||
node is distinguishable from a dead socket. Ninety seconds of nothing at all,
|
||||
not even a heartbeat, fails the call as gone. A heartbeat says the *agent* is
|
||||
alive and nothing about the node, so it never satisfies a node's own timeout:
|
||||
one set to thirty seconds fires after thirty seconds of the node reporting
|
||||
nothing, wherever it runs.
|
||||
@@ -154,8 +154,8 @@ environment, and what it says it has: cores, GPUs and memory.
|
||||
## Upgrading
|
||||
|
||||
The engine and the worker speak a version-matched protocol, and a worker
|
||||
announcing anything else is refused rather than half-understood. Protocol 2 —
|
||||
the one that carries inventory — is `fluksio-worker` 0.2.0. An older agent is
|
||||
announcing anything else is refused rather than half-understood. Protocol 2,
|
||||
the one that carries inventory, is `fluksio-worker` 0.2.0. An older agent is
|
||||
told so on the socket and stops, rather than retrying against an engine that
|
||||
will never accept it; `pip install -U fluksio-worker` on that host is the whole
|
||||
upgrade. Nothing changed in the runner served at `GET /api/v1/workers/runtime`,
|
||||
@@ -163,7 +163,7 @@ so a host that copies its two files copies the same one as before.
|
||||
|
||||
## Machines from a batch scheduler
|
||||
|
||||
A cluster is not a machine that attaches and stays — it is a queue somebody else
|
||||
A cluster is not a machine that attaches and stays; it is a queue somebody else
|
||||
owns. So Fluksio does not submit *nodes* to Slurm. It submits a job whose payload
|
||||
is an ordinary worker dialling back in, and from there everything works the way
|
||||
it already does: the same protocol, the same artifacts, the same cancellation.
|
||||
@@ -176,7 +176,7 @@ Write the clusters into `provisioners.json` beside the flows:
|
||||
"name": "hpc",
|
||||
"login": "me@login.cluster",
|
||||
"ssh_key": "/secrets/hpc_ed25519",
|
||||
"engine_url": "wss://api.example.com/api/v1/workers/attach",
|
||||
"engine_url": "wss://api.fluksio.com/api/v1/workers/attach",
|
||||
"max_idle_s": 300,
|
||||
"provision_timeout_s": 900,
|
||||
"profiles": [{
|
||||
@@ -192,12 +192,10 @@ Write the clusters into `provisioners.json` beside the flows:
|
||||
A **profile** is what the scheduler is asked for, where a flavor is what a node
|
||||
asks for. They are separate on purpose, and agree when you set them up to.
|
||||
|
||||
When a node needs a machine nothing attached can give, and a profile would fit,
|
||||
the engine `sbatch`es one over ssh — the system `ssh`, so nothing new is
|
||||
installed — and the run waits meanwhile, saying so. `prerun` owns the
|
||||
environment: a `module load`, a venv with `fluksio-worker` already in it. There
|
||||
is deliberately no `pip install` in the generated script, because what is
|
||||
installed on a cluster is somebody's decision and not this program's.
|
||||
When a node needs a machine nothing attached can give, and a profile fits, the
|
||||
engine `sbatch`es one over the system `ssh`, and the run waits meanwhile, saying
|
||||
so. `prerun` owns the environment: a `module load`, or a venv with
|
||||
`fluksio-worker` already in it. The generated script runs no `pip install`.
|
||||
|
||||
One outstanding request per profile, however often it is asked for. A job that
|
||||
never attaches within `provision_timeout_s` is `scancel`led, as is anything
|
||||
@@ -219,8 +217,8 @@ queue does, which is what a cluster that queues overnight needs, and
|
||||
## What a worker is not
|
||||
|
||||
It is not a second engine. Subscriptions, schedules, webhooks, the dashboards
|
||||
and the run queue all stay in one process — that is what keeps a message having
|
||||
one definition and a cron tick happening once. A worker executes node bodies.
|
||||
and the run queue all stay in one process, which keeps a message having one
|
||||
definition and a cron tick happening once. A worker executes node bodies.
|
||||
|
||||
Running two engines against one data directory is not supported. Distribute
|
||||
work with workers.
|
||||
|
||||
Reference in New Issue
Block a user