Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Publish the documentation site: docs.fluksio.com

Browse Source 
A zensical site under docs/, served by a new `docs` compose service behind
Traefik, built with --strict in CI. Same pattern the sibling n3xd workspace
uses.

Getting started splits the way the landing page does — one path is
`pip install fluksio` and a training script, the other is a Docker stack and
an afternoon in the browser — because the two audiences will not spend the same
amount of time. Everything after that is shared: the concepts, the web
interface (app and portal), the CLI and the API, and a reference for node types,
payload types and configuration.

The three flow guides move here from the docs submodule rather than being
copied, so there is one version of them.

Styling mirrors DESIGN-GUIDELINES.md: the app's token palette remapped onto
Material's variables in both schemes, Inter, the 16px panel radius, and the one
terracotta accent spent on the facility lane of the audience split.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M7Xv3cJEW5c8AXxn2hoojV
This commit is contained in:
stroblme
2026-08-22 05:55:34 +02:00
co-authored by Claude Opus 5
parent c3cbbbc962
commit d12c81c8a0
34 changed files with 4167 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/code/agents.md
+111
View File
@@ -0,0 +1,111 @@
# Agents over MCP
The engine exposes the flow API to agents over the
[Model Context Protocol](https://modelcontextprotocol.io). Point Claude Code,
Claude Desktop, Codex or anything else that speaks MCP at it, and it can read,
build, publish and run flows.
## Switching it on
```sh
MCP_ENABLED=true
```
It is off by default, because switching it on opens client registration to
whoever can reach the host. Turn it on deliberately.
The endpoint is `https://api.${DOMAIN}/mcp` — or `http://api.localhost/mcp` on
a local stack.
## Connecting a client
```sh
claude mcp add --transport http fluksio https://api.example.com/mcp
```
The OAuth flow does the rest: the client registers itself, you approve it in
the browser, and it exchanges the grant for a token. Nothing to paste.
## What an agent may do
An agent acts as **the person who approved it**, and only through the same REST
API the dashboard uses. It cannot do anything you could not do in the browser,
and it cannot do anything *you* cannot do — an agent approved by a non-superuser
is not one either.
Its tokens are signed with a keypair of their own, kept beside the flow store.
Deleting that file revokes every agent's access without logging anyone out of
the dashboard.
Registered clients are managed under **Admin**.
## The tools
Read-only:
| Tool | Returns |
|---|---|
| `list_flows` | every flow with its node count and state |
| `get_flow` | one flow's definition, node status and issues |
| `get_node_source` | a node's Python |
| `list_node_types` | every type and its parameter schema |
| `get_flow_state` | the current value of each of a flow's messages |
| `get_message_history` | one message's recent numeric values |
| `get_graph` | every flow as one graph |
| `list_shared_nodes` | the shared library and its usages |
| `list_secrets` | secret **names** — never values |
| `get_modules` | the packages node code can import |
| `get_health` | flows, nodes, queue and loop lag |
| `get_metrics` | executions, errors and timings per minute |
| `list_failures` | what went wrong recently |
| `list_runs` | recent cascades |
Editing:
| Tool | Does |
|---|---|
| `save_flow` | write a draft |
| `save_node_source` | write a node's code into the draft |
| `validate_flow` | the issues, without saving |
| `publish_flow` | put the draft live (takes the version it saw) |
| `discard_draft` | throw the draft away |
| `delete_flow` | delete it |
| `apply_modules` | replace the package manifest |
Operating:
| Tool | Does |
|---|---|
| `run_flow` | run every node once from the values given |
| `trigger_node` | feed one node and run what is downstream |
| `start_flow` / `stop_flow` | activate or tear down |
| `pause_flow` / `resume_flow` | hold messages, or let them through |
| `cancel_node` | kill a node's code while it runs |
## Why it is shaped this way
The tools do not reach into the engine. Each one calls the same REST endpoint
the dashboard calls, over an in-process transport.
That keeps exactly one description of what a flow is and how it may be changed
— the validation, the draft/publish split, the version check that stops two
clients overwriting each other. An agent that saved a flow through a private
back door would be an agent that could write one the canvas cannot open.
The caller's token rides along on every hop, so the API sees the agent's own
identity rather than a service account. The audit trail on Home names it.
## A note on working this way
The pattern that works is the same one that works with a person: let the agent
read the flow and its issues, let it write a **draft**, and publish yourself
once you have looked. `validate_flow` before `publish_flow` is cheap and tells
you what the canvas would have told you.
## Limits
- MCP is not currently reachable through a [portal](../interface/portal.md)
tunnel — the proxy forwards `/api/v1/` only. Connect an agent on the same
network as the installation.
- Secrets are never readable, by an agent or by anyone else. `list_secrets`
returns names.
docs/code/api.md
+223
View File
@@ -0,0 +1,223 @@
# The HTTP API
Everything the browser does, the API does first. The dashboard is a generated
client of this schema, not a privileged path into the engine — so anything you
can click, you can script.
Base URL: `https://api.${DOMAIN}/api/v1`, or `http://127.0.0.1:8000/api/v1` for
a `fluksio serve` installation.
```sh
export FLUKSIO=http://127.0.0.1:8000/api/v1
```
## Authenticating
```sh
export TOKEN=$(curl -s -X POST $FLUKSIO/login/access-token \
-d "username=admin@example.com&password=..." | jq -r .access_token)
curl -s $FLUKSIO/users/me -H "Authorization: Bearer $TOKEN"
```
A bearer token, valid for eight days. `POST /login/test-token` checks one.
Password recovery and reset are `POST /password-recovery/{email}` and
`POST /reset-password/`.
Agents authenticate differently — see [Agents over MCP](agents.md).
!!! tip "The interactive schema"
When `ENVIRONMENT` is not `production`, the full OpenAPI schema is at
`/docs` (Swagger) and `/redoc`, and the raw document at
`/api/v1/openapi.json`. That is the authoritative reference; this page is
the tour.
It is closed in production on purpose: the schema enumerates every endpoint
the installation serves, including the paths webhook nodes mounted at
runtime.
## Flows
| Method | Path | What |
|---|---|---|
| `GET` | `/flows/` | every flow, with node counts, draft and running state |
| `GET` | `/flows/{name}` | one flow — the draft if there is one — with node status and issues |
| `PUT` | `/flows/{name}` | save a draft (`version` must match, or 409) |
| `POST` | `/flows/{name}/publish` | put the draft live |
| `POST` | `/flows/{name}/discard-draft` | throw the draft away |
| `POST` | `/flows/{name}/rename` | rename it |
| `DELETE` | `/flows/{name}` | delete it |
| `GET` | `/flows/node-types` | every node type and its parameter schema |
| `GET` | `/flows/graph` | every flow as one graph — what Home draws |
Reading a flow returns the *draft* when one exists, along with a preview of how
it would run: node statuses and validation issues for the version you are
editing, not the one running underneath it.
### Node source
```sh
curl -s $FLUKSIO/flows/house/nodes/decide/source -H "Authorization: Bearer $TOKEN"
curl -X PUT $FLUKSIO/flows/house/nodes/decide/source -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"code": "def process(x):\n return {\"y\": x * 2}\n"}'
```
`POST .../share` promotes a node's source to the shared library;
`POST .../unshare` gives it a private copy back. `GET /flows/library` lists the
shared sources and which flows use each.
### Running and controlling
| Method | Path | What |
|---|---|---|
| `POST` | `/flows/{name}/start` · `/stop` | activate or tear down its subscriptions and schedules |
| `POST` | `/flows/{name}/pause` · `/resume` | hold messages instead of running them |
| `POST` | `/flows/{name}/step` | release exactly one held message |
| `POST` | `/flows/{name}/validate` | the issues, without saving |
| `POST` | `/flows/{name}/run` | run every node once, from the values you pass |
| `POST` | `/flows/{name}/nodes/{id}/trigger` | feed values into one node |
| `POST` | `/flows/{name}/nodes/{id}/cancel` | kill the worker running it right now |
| `POST` | `/flows/{name}/nodes/{id}/acknowledge` | clear the failure the canvas is marking |
`POST /flows/{name}/run` on a **batch** flow submits a run instead, because
that is what running one means — the parameters, the series and the result are
the point, and a call that quietly did something else would be a trap.
## Messages
```sh
curl -s $FLUKSIO/messages/ -H "Authorization: Bearer $TOKEN"
curl -X POST $FLUKSIO/messages/house.setpoint -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"value": 22.5}'
curl -s $FLUKSIO/messages/house.temperature/history -H "Authorization: Bearer $TOKEN"
```
`GET /messages/` is the catalogue: every message any published flow declares,
with its last value. Publishing puts a value into the graph exactly as a
dashboard control does — which means only a message some flow *declares* can be
published to. Flows own the namespace; everything else is a client of it.
## Runs
| Method | Path | What |
|---|---|---|
| `POST` | `/runs/flows/{name}` | queue one run — `{"params": {...}, "seed": 7, "draft": false}` |
| `POST` | `/runs/flows/{name}/sweep` | queue many, sharing a `group_id` |
| `GET` | `/runs` | the queryable history: `?flow=`, `?status=`, `?group=`, `?digest=`, `?limit=` |
| `GET` | `/runs/{id}` | one run in full: params, result, per-node record, artifacts |
| `POST` | `/runs/{id}/cancel` | stop it |
| `GET` | `/runs/{id}/metrics?name=&stride=` | one metric's series, in step order |
| `GET` | `/runs/series/compare?ids=a,b,c&metric=` | that metric across several runs |
Submitting answers immediately with a `queued` run. Wrong parameters — an
undeclared name, a value of the wrong type — come back as a 422 naming the
problem, before anything executes.
`?digest=` filters by the hash of the parameters, which is how you find "every
run that used exactly this configuration".
`compare` answers in the same shape a chart widget draws, so three training
curves side by side is a widget binding rather than a screen of its own.
See [Runs: pipelines that finish](../concepts/runs.md).
## Artifacts
```sh
curl -X PUT $FLUKSIO/artifacts -H "Authorization: Bearer $TOKEN" \
--data-binary @model.pt
curl -s $FLUKSIO/artifacts/sha256:abc... -H "Authorization: Bearer $TOKEN" -o model.pt
```
Content-addressed, so uploading the same bytes twice stores them once. Node
code normally reaches these through `fluksio.save_artifact` /
`fluksio.load_artifact` rather than here.
## Dashboards and panels
| Method | Path | What |
|---|---|---|
| `GET` | `/dashboards/` | every dashboard, without its contents |
| `GET` | `/dashboards/{name}?draft=true` | the published document, or the editor's copy |
| `POST` | `/dashboards/{name}` | create one |
| `PUT` | `/dashboards/{name}` | save a draft |
| `POST` | `/dashboards/{name}/publish` · `/discard` · `/rename` | |
| `GET` `PUT` | `/panels/` | which device shows which dashboards |
| `POST` | `/panels/pair` | start a pairing |
| `GET` | `/panels/pair/{code}` | what is holding a code |
## Secrets, modules, alerts
```sh
curl -s $FLUKSIO/secrets/ -H "Authorization: Bearer $TOKEN" # names only
curl -X PUT $FLUKSIO/secrets/influx-token -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"value": "..."}'
curl -s $FLUKSIO/modules/ -H "Authorization: Bearer $TOKEN"
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"requirements": "numpy>=2\n"}'
curl -s $FLUKSIO/alerts/config -H "Authorization: Bearer $TOKEN"
curl -X POST $FLUKSIO/alerts/test/my-phone -H "Authorization: Bearer $TOKEN"
```
Secrets are write-only over the API: you can list names and set values, never
read one back.
## Observability
| Path | What |
|---|---|
| `/observability/summary` | engine health — always 200, degraded or not |
| `/observability/timeseries` | executions and failures over a window |
| `/observability/flows` | per-flow rollups with a 60-slice trend |
| `/observability/runs` | recent cascades, with `?flow=`, `?since=`, `?until=` |
| `/observability/events?kind=failure\|audit` | what went wrong, or who changed what |
| `/observability/dead-letter` | work the engine gave up on |
`GET /utils/health/` is the deep health check the container probe uses: it
fails when the event loop is wedged or the state backend is gone, not just when
the process is dead.
## Workers
| Method | Path | What |
|---|---|---|
| `GET` | `/workers` | what is attached, its labels and how busy it is |
| `POST` | `/workers/tokens` | mint a worker credential (superuser; shown once) |
| `GET` | `/workers/runtime` | the node runner's source, for a host without pip |
| `WS` | `/workers/attach` | where a worker dials in |
See [Remote workers](workers.md).
## Live events
```text
ws://127.0.0.1:8000/api/v1/flows/ws?token=<access token>
```
The websocket authenticates from its query string. It sends a snapshot on
connect and then every engine event: node executions, values published, health
changes, run started and finished. This is what the canvas and the dashboards
draw from.
## Errors
| Code | Means |
|---|---|
| `400` | the request was malformed, or the node refused it with a message |
| `401` / `403` | not signed in, or not allowed |
| `404` | no such flow, dashboard, run or message |
| `409` | someone else saved first — the body carries `current_version` |
| `422` | a parameter, port or binding did not typecheck |
| `503` | that subsystem is not available on this installation |
A 409 on a save or a publish is not an error to retry blindly: it means the
stored version moved past the one you were editing. Re-read, merge, save again.
## Generating a client
The frontend's TypeScript client is generated from the OpenAPI schema
(`make generate-client`). Any OpenAPI generator will do the same for your
language — point it at `/api/v1/openapi.json` on a non-production installation.
docs/code/cli.md
+153
View File
@@ -0,0 +1,153 @@
# The `fluksio` command
```sh
pip install fluksio
```
Installs the engine and the `fluksio` command. Python 3.10 or newer, Linux or
macOS.
There is a second, smaller distribution — `fluksio-worker` — for a machine that
should only *run nodes* for an engine elsewhere. It has none of the engine in
it. See [Remote workers](workers.md).
## `fluksio serve`
Runs the engine.
```sh
fluksio serve
```
On the first start it creates an admin account and prints its password **once**.
Nothing else has to be running: no database server, no message broker, no
Docker.
| Option | Default | What it does |
|---|---|---|
| `--data-dir PATH` | `~/.fluksio` (or `$FLUKSIO_HOME`) | where this installation keeps everything |
| `--host HOST` | `127.0.0.1` | what to bind |
| `--port PORT` | `8000` | what to listen on |
| `--log-level LEVEL` | `info` | uvicorn's log level |
| `--admin-email ADDR` | `admin@example.com` | the account created on first run |
| `--admin-password PW` | generated | set it instead of having one generated |
| `--enroll CODE` | — | pair with a portal as part of coming up |
| `--portal URL` | — | the portal `--enroll` redeems at |
`--enroll` with `--portal` is the one-command setup: it pairs before the engine
starts, so the connection is dialled as part of coming up rather than needing a
restart. It is skipped if the installation is already enrolled.
!!! warning "One process"
`fluksio serve` holds the flow engine. A second one is a *second engine* —
duplicated subscriptions, duplicated cron ticks, two webhooks answering the
same path. Run one, and distribute work with
[workers](workers.md) instead.
!!! note "`$HOME` on a cluster"
A login node's home directory is often NFS, where SQLite's write-ahead log
does not work — the database would be locked or corrupt. `fluksio serve`
warns when it notices; point `--data-dir` at local disk.
### What it prints
```text
Created the admin account admin@example.com
password: k3Qm-8vTpLdX
Shown once. Change it from the dashboard.
Fluksio 0.1.0 — data in /home/you/.fluksio
API http://127.0.0.1:8000/api/v1
No portal. Pair this installation with:
fluksio enroll <code> --portal https://hub.example.com
```
An enrolled installation says which portal it is on instead, and notes that the
dashboard is served from there rather than here.
## `fluksio enroll`
Pairs an existing installation with a portal.
```sh
fluksio enroll ABCD-1234 --portal https://hub.fluksio.com
```
| Option | What it does |
|---|---|
| `--portal URL` | **required** — the portal the code was minted on |
| `--as EMAIL` | the local account a portal session arrives as |
| `--data-dir PATH` | which installation, if not the default |
Get the code from the portal under **Installations → Add installation**. It is
single-use and expires in fifteen minutes. `--as` matters when the installation
has several superusers — without it, enrolment refuses rather than guessing.
Afterwards, `fluksio serve` dials the portal as it comes up. See
[Accounts and the portal](../interface/portal.md).
## `fluksio worker`
Runs nodes for an engine elsewhere. Everything after `worker` belongs to the
agent's own parser — it is the same program `fluksio-worker` installs, so the
two are interchangeable:
```sh
fluksio worker --url wss://api.example.com/api/v1/workers/attach \
--token "$FLUKSIO_WORKER_TOKEN" --labels gpu
```
See [Remote workers](workers.md).
## What lives in the data directory
```text
~/.fluksio/
├── fluksio.db SQLite: users, runs, metrics, observability, agents
├── flows/ a git repository — one directory per flow
│ ├── house/
│ │ ├── flow.json the published structure
│ │ ├── nodes/*.py the published node code
│ │ ├── flow.draft.json unpublished edits, if any
│ │ └── nodes.draft/*.py
│ ├── _lib/ shared node sources
│ ├── _dashboards/ dashboards, drafts and all
│ └── requirements.txt what the Modules screen installs
├── artifacts/ content-addressed bytes, two levels deep
├── user-venv/ the interpreter your node code runs on
├── secrets.enc encrypted credentials, deliberately outside flows/
├── alerts.json alert channels and rules
├── panels.json wall-panel pairings
├── oauth-key.pem signs agent tokens
├── cloud.json the portal enrolment, if there is one
├── secret_key signs sessions and derives the secrets key
└── env optional settings file
```
Two things follow from this layout and are worth internalising:
**`flows/` is a real git repository.** `git log` is the history of every change
anyone made to any flow. A run records the commit it ran at, so `git show` on
that hash is literally the code that produced the number.
**Backing up the data directory backs up the installation.** Everything else is
rebuildable. Copy it while the engine is stopped, or use SQLite's online backup
for the database if it is not.
## Settings
Settings come from the environment, or from an `env` file in the data
directory. The ones you are most likely to touch:
| Variable | Default | What it does |
|---|---|---|
| `DATA_DIR` | `~/.fluksio` via the CLI | everything below it derives from this |
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links |
| `ENVIRONMENT` | `local` | `production` closes the interactive API schema |
| `MCP_ENABLED` | `false` | opens the [agent endpoint](agents.md) |
| `SECRET_KEY` | generated once | signs sessions, derives the secrets key |
The full list is in [Configuration](../reference/configuration.md).
docs/code/nodes.md
+202
View File
@@ -0,0 +1,202 @@
# Writing node code
A Function node is a Python file. That is all it is — no base class, no
decorator, no framework import unless you want one.
```python
def process(temperature, setpoint=21.0):
"""Ask for heat when the room is below the comfort point."""
return {"heat": temperature < setpoint}
```
## The rules
**One function called `process`.** If the file defines exactly one public
function under another name, that one is used instead. Two, and the node
refuses to load rather than guessing.
**Arguments come from ports and settings, by name.** `temperature` above is an
input port; `setpoint` is a setting typed into the node's panel. Both arrive as
keyword arguments, which is why a setting may not share a name with a port.
See [Where a node's values come from](../concepts/values.md).
**The return value is a dict keyed by output ports.** Every value is checked
against the port's declared type before it is published. A key that is not a
declared port is an error, not a silent drop — nothing leaves a node except
through a port it declared.
**Nothing else is importable from the engine.** Node code runs in a separate
process, on a separate interpreter, with none of Fluksio's own modules on its
path. What it can import is what the [Modules](../interface/operations.md)
screen installed.
**A node is a pure function of its inputs.** No context object, no global
store, no handle to reach for. A running total or a debounce timer has a
specific shape — see [Keeping state in a flow](../concepts/state.md).
## Producing values over time
A node that produces values *during* its execution is a generator. Every
`yield` is a dict keyed by output port, published the instant it happens:
```python
def process(lr, steps):
loss = 1.0
for _ in range(steps):
loss = train_one_step(lr)
yield {"loss": loss} # published now
return {"final_loss": loss}
```
Whatever the generator `return`s at the end is the node's result — what
downstream nodes read. If you never `return`, the last thing you yield is the
result instead.
Mark the port so the flow says what it does:
```json
{"name": "loss", "dtype": "float", "stream": true}
```
Two consequences. In a [run](../concepts/runs.md), the whole series is kept as
that run's metrics — this is why there is no `log_metric()` anywhere in the
API. And **the node's timeout starts measuring silence rather than duration**:
each emission resets the deadline, so a node yielding every few seconds can run
for hours under a timeout of 300.
### `fluksio.emit`
Where a `yield` cannot reach — the value comes from inside somebody else's
callback, and they call you rather than the other way round:
```python
import fluksio
def process():
model.fit(callbacks=[LambdaCallback(
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
return {"weights": ...}
```
Same ports, same type checking, same publication. Prefer `yield` where you can
reach it; `emit` where you cannot.
## Bytes: artifacts
Messages are JSON, which is what lets the same value pass through Redis, the
work queue and the worker protocol unchanged. A checkpoint is not that.
```python
import fluksio
def process(dataset):
path = fluksio.load_artifact(dataset) # → a local path to read
...
return {
"weights": fluksio.save_artifact("model.pt", media_type="application/octet-stream"),
"score": 0.94,
}
```
`save_artifact` takes bytes or a path, stores them by their SHA-256 digest, and
returns a small reference — digest, size, media type, name — which is what an
`artifact`-typed port carries.
Because the address is the content's hash, a sweep whose fifty configs share
one preprocessed input stores it once, and a reference stays valid wherever the
store is reachable from — including on another machine.
## Printing
`print` works and is captured. The first 16 KB per call is kept and shown in
the flow editor's log panel and on the run's per-node record; the rest is
dropped, so a node printing in a loop cannot fill anything up.
Use it to debug. Do not use it to record results — a number worth keeping is an
output port, not a line of text.
## Errors
An exception fails that node's execution, not the flow. The message you see is
one line from the frame in *your* code, not a stack through the engine — that
is a deliberate choice about what is actionable.
The node keeps its last error visible after it recovers, so a failure that
fired an alert at 03:00 still says what it was at 09:00. It can also be
acknowledged from the canvas.
## Timeouts
`timeout` on a node is how many seconds its code may run before it is stopped.
The default is 30, and it covers the *first* call's imports, which can be much
slower than the body — a node importing torch is not being slow, it is loading.
Above 60 seconds, a live flow may deliver the same work again while the node is
still running. In a batch run, which never redelivers, it is an idle timeout
instead: silence this long is a kill.
## Running a node somewhere else
A node declares the label of the machine it needs:
```json
{"id": "train", "device": "gpu", "device_policy": "require", "timeout": 7200}
```
`require` (the default) waits for a worker carrying that label; `prefer` runs
locally when none is attached. A node bound to a device is compiled *on that
machine* — a node importing `torch` is correct on the GPU box and a missing
module on the engine, so checking it here would fail something that is fine.
See [Remote workers](workers.md).
## Sharing code between flows
A node's source can be promoted to the shared library from its panel, and other
flows can then use it by reference. One copy, one place to edit — and every
flow using it runs the edit, which is the point and also the caution.
Shared sources live in `_lib/` in the flow repository, so they are versioned
with everything else.
## Packages
Node code runs in a virtual environment of its own, on the installation's data
volume — deliberately separate from the one Fluksio itself runs on.
Declare what you import in [Modules](../interface/operations.md), or over the
API:
```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"requirements": "numpy>=2\npandas\n"}'
```
It is a pip manifest installed with `uv pip sync`, versioned alongside your
flows. An install takes effect immediately; nothing restarts.
## A worked example
The repository ships a small supervised fit as a seedable demo — three nodes,
a batch flow, streaming metrics, artifacts between stages, and a GPU-labelled
node that falls back to the engine when no worker is attached. It is the
shortest complete thing to read:
```text
prepare ──dataset(artifact)──▶ train ──weights(artifact)──▶ evaluate
│
└── loss (streaming float) ──▶ chart
```
`make seed-demo` builds it against a running stack.
## See also
- [Where a node's values come from](../concepts/values.md)
- [Keeping state in a flow](../concepts/state.md)
- [Runs: pipelines that finish](../concepts/runs.md)
- [Node types](../reference/node-types.md) — the ones you do not have to write
docs/code/workers.md
+137
View File
@@ -0,0 +1,137 @@
# Remote workers
The engine runs where the automations are. The GPU is somewhere else, the
Raspberry Pi with the relays is in a shed, and neither of them is on the same
network as the other.
A **worker** is a process that runs the code of nodes marked for it. It dials
*out* to the engine over one authenticated websocket, so nothing on that
machine has to be reachable — and nothing has to expose the engine's state
backend across hosts, which it never should.
## Install and attach
```sh
pip install fluksio-worker
fluksio-worker \
--url wss://api.example.com/api/v1/workers/attach \
--token "$FLUKSIO_WORKER_TOKEN" \
--labels gpu,cuda12 \
--python /opt/torch-venv/bin/python
```
`fluksio-worker` is its own distribution — the agent, the node runner, and
`websockets`. Nothing of the engine, so a GPU box does not install a database
driver in order to run a training step. An engine host already has it, and
`fluksio worker …` is the same program.
| Option | Default | What it does |
|---|---|---|
| `--url` | **required** | `wss://…/api/v1/workers/attach` |
| `--token` | `$FLUKSIO_WORKER_TOKEN` | the credential, minted on the engine |
| `--name` | this host's name | how it shows up in the worker list |
| `--labels` | none | comma-separated; what a node's `device` matches |
| `--python` | this interpreter | the interpreter node code runs on |
| `--parallel` | `1` | how many node calls it will take at once |
| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |
`--python` is the important one. It is how this machine keeps its own wheels —
the CUDA build, the vendor SDK, the thing that will not install anywhere else —
without the engine ever installing them or knowing about them.
## Mint the token
On the engine, as a superuser:
```sh
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"name": "gpu-dev"}'
```
Shown once, valid for a year — a worker is a machine somebody sets up and
leaves running. It is signed with the same keypair agent tokens use, so
rotating that key revokes every worker along with them.
??? note "A host where pip is not an option"
The two files work copied into one directory and run with `python agent.py
…`. The engine serves the runner itself at `GET /api/v1/workers/runtime` —
it is the same module its own local workers run, deliberately standard
library only.
## Send a node to it
A node declares the label of the machine it needs:
```json
{
"id": "train",
"device": "gpu",
"device_policy": "require",
"timeout": 7200
}
```
| `device_policy` | Behaviour when nothing carrying the label is attached |
|---|---|
| `require` (default) | the run stays `queued` and says what it is waiting for |
| `prefer` | it runs on the engine instead |
`prefer` is what makes a flow work before the GPU box exists. `require` is what
you want once it does.
!!! note "Set from the API"
`device` and `device_policy` are not yet fields in the node panel. Set them
with `PUT /flows/{name}`.
## What follows from this
- **The node's source travels with every call.** Nothing has to be deployed to
the worker, and changing a node's code takes effect on the next execution.
- **A node bound to a device is compiled on that machine.** A node importing
`torch` is correct on the GPU box and a missing module on the engine, so
checking it here would fail something that is fine.
- **`import fluksio` inside a node is the worker's own reporter.** `emit`,
`save_artifact`, `load_artifact` — installed before your code runs, so an
installed `fluksio` package on that box never shadows it.
- **Cancelling a run kills what it is executing**, there or here, and leaves
other runs of the same node alone.
- **If the worker disappears mid-call**, the run fails in seconds with
`worker went away mid-call` rather than waiting out its timeout.
- **A worker sends a heartbeat while it executes**, so a long node is
distinguishable from a dead socket. Ninety seconds of silence is gone.
## Artifacts across machines
An artifact reference names content by its hash, not a location, so it stays
valid wherever the store is reachable from. A worker that shares the engine's
filesystem writes to it directly; one that does not fetches and uploads over
HTTP, using the artifact endpoint beside the socket it already has. Either way
your node code is the same two calls.
## Seeing what is attached
```sh
curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq
```
Name, labels, how many calls it will take at once, how many are in flight, when
it attached, when it was last seen, its Python version, and a digest of its
environment.
## What a worker is not
It is not a second engine. Subscriptions, schedules, webhooks, the dashboards
and the run queue all stay in one process — that is what keeps a message having
one definition and a cron tick happening once. A worker executes node bodies.
Running two engines against one data directory is not supported. Distribute
work with workers.
## See also
- [Runs: pipelines that finish](../concepts/runs.md#running-a-node-somewhere-else)
- [Writing node code](nodes.md)
- [Getting started: data science](../getting-started/data-science.md)