docs
Docs / docs (push) Successful in 26s
Playwright Tests / test-playwright (1, 2) (push) Failing after 4m7s
Playwright Tests / test-playwright (2, 2) (push) Failing after 2m30s
pre-commit / pre-commit (push) Failing after 2m42s
Test Backend / test-backend (push) Failing after 3m3s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m51s

This commit is contained in:
2026-10-13 10:02:20 +02:00
parent 8d97b8ffaa
commit b2aa932b6d
24 changed files with 2210 additions and 4310 deletions
+45 -123
View File
@@ -1,145 +1,67 @@
# Agents over MCP
The engine exposes the flow API to agents over the
[Model Context Protocol](https://modelcontextprotocol.io). Point Claude Code,
Claude Desktop, Codex or anything else that speaks MCP at it, and it can read,
build, publish and run flows.
Fluksio exposes flow tools over MCP. An agent can inspect, edit, publish,
and run flows with the permissions of the user who authorizes it.
## Switching it on
## Connect an agent
```sh
MCP_ENABLED=true
```
Set `MCP_ENABLED=true` in the engine's environment and restart it. The Python
package defaults to off; the Docker template enables it.
It is off by default: switching it on opens client registration to whoever can
reach the host.
Configure your MCP client with an HTTP endpoint:
The endpoint is `https://api.${DOMAIN}/mcp`, or `http://api.localhost/mcp` on a
local stack.
- Local Docker stack: `http://api.localhost/mcp`
- Deployment: `https://api.<your-domain>/mcp`
## Connecting a client
The client registers, opens browser authorization, and receives an OAuth
token after you approve access. The issuer and browser redirects use
`DOMAIN`, `ENVIRONMENT`, and `FRONTEND_HOST`, so configure those for the
addresses your client and browser can reach.
```sh
claude mcp add --transport http fluksio https://api.${DOMAIN}/mcp
```
Manage registered clients under **Admin → Agents**. Revoking a client
invalidates its access. MCP uses a separate signing key at
`OAUTH_PRIVATE_KEY_FILE`; rotating that key invalidates all MCP tokens.
The OAuth flow does the rest: the client registers itself, you approve it in
the browser, and it exchanges the grant for a token. Nothing to paste.
MCP is not forwarded by the [portal tunnel](../interface/portal.md).
Connect directly to the instance's API host.
## What an agent may do
## Available tools
An agent acts as **the person who approved it**, and only through the same REST
API the dashboard uses. It cannot do anything you could not do in the browser,
and it cannot do anything *you* cannot do. An agent approved by a non-superuser
is not one either.
Its tokens are signed with a keypair of their own, kept beside the flow store.
Deleting that file revokes every agent's access without logging anyone out of
the dashboard.
Registered clients are managed under **Admin**.
## The tools
Read-only:
| Tool | Returns |
| Task | Tools |
|---|---|
| `list_flows` | every flow with its node count and state |
| `get_flow` | one flow's definition, node status and issues |
| `get_node_source` | a node's Python |
| `list_node_types` | every type and its parameter schema |
| `get_flow_state` | the current value of each of a flow's messages |
| `get_message_history` | one message's recent numeric values |
| `get_graph` | every flow as one graph |
| `list_shared_nodes` | the shared library and its usages |
| `list_secrets` | secret **names**, never values |
| `get_modules` | the packages node code can import |
| `get_health` | flows, nodes, queue and loop lag |
| `get_metrics` | executions, errors and timings per minute |
| `list_failures` | what went wrong recently |
| `list_runs` | recent cascades |
| Inspect flows | `list_flows`, `get_flow`, `get_node_source`, `list_node_types`, `get_graph`, `list_shared_nodes` |
| Inspect values | `get_flow_state`, `get_message_history` |
| Inspect the instance | `list_secrets` (names only), `get_modules`, `get_health`, `get_metrics`, `list_failures`, `list_runs` (live cascades) |
| Edit flows | `save_flow`, `save_node_source`, `validate_flow`, `publish_flow`, `discard_draft`, `delete_flow` |
| Operate flows | `run_flow`, `trigger_node`, `start_flow`, `stop_flow`, `pause_flow`, `resume_flow`, `cancel_node` |
| Install node packages | `apply_modules` |
Editing:
Tools call the same API as the browser. Drafts, validation, version checks,
and superuser restrictions apply. Secrets cannot be read back.
| Tool | Does |
|---|---|
| `save_flow` | write a draft |
| `save_node_source` | write a node's code into the draft |
| `validate_flow` | the issues, without saving |
| `publish_flow` | put the draft live (takes the version it saw) |
| `discard_draft` | throw the draft away |
| `delete_flow` | delete it |
| `apply_modules` | replace the package manifest |
Operating:
| Tool | Does |
|---|---|
| `run_flow` | run every node once from the values given |
| `trigger_node` | feed one node and run what is downstream |
| `start_flow` / `stop_flow` | activate or tear down |
| `pause_flow` / `resume_flow` | hold messages, or let them through |
| `cancel_node` | kill a node's code while it runs |
## Why it is shaped this way
The tools do not reach into the engine. Each one calls the same REST endpoint
the dashboard calls, over an in-process transport.
That keeps exactly one description of what a flow is and how it may be changed:
the validation, the draft/publish split, and the version check that stops two
clients overwriting each other. An agent saving a flow through a private back
door could write one the canvas cannot open.
The caller's token rides along on every hop, so the API sees the agent's own
identity rather than a service account. The audit trail on Home names it.
A useful workflow: ask the agent to inspect the flow, save a draft, and
validate it. Review the draft before asking it to publish. The agent has
a publish tool, so make your intent explicit.
## The assistant
The same tools drive the chat inside the dashboard. An operator sets a
provider under **Settings → Assistant** (anything OpenAI-compatible: OpenAI,
Anthropic, OpenRouter, a local Ollama), and an **Assistant** entry appears in
the sidebar for everyone. The panel opens on any screen and keeps its place
from screen to screen; on a flow, it knows which one you are looking at.
A superuser configures an OpenAI-compatible provider under
**Settings → Assistant**, with a base URL, API key, and model. This enables
the dashboard's Assistant entry for signed-in users.
The model calls the tools above, in-process, as **you**: every call carries
your own token, so the chat can do nothing you could not do yourself, and its
edits show up in the editor as drafts the moment they are saved. It is told to
publish only when asked, but it has the tool, so read what it did before
saying so. The trace under each answer lists every call it made.
The conversation is one per person, kept on the instance beside the
configuration, and the bin in the panel's header starts it over. `MCP_ENABLED`
is not needed for it: the endpoint is for agents outside, the chat for the
one inside.
The assistant uses the same tools with your own permissions. Draft edits
appear in the editor, and each answer includes a tool trace. Conversations
are stored per user on the instance; the bin button clears yours.
`MCP_ENABLED` is only needed for external clients, so the built-in assistant
works without it.
## Control
Where a live flow has a [`wren` node](../reference/node-types.md#wren) with
its `prompt` and `actions` wired, the chat gains a **Control** tab, and Home a
chat button at the top right of the brain graph (in the header, on a phone)
that opens it. No provider is needed for this one: the model is the node.
A live [Wren node](../reference/node-types.md#wren) with both `prompt` and
`actions` wired adds a **Control** tab to the chat. It uses the node's local
model and needs no assistant provider.
A command is published to the node's `prompt` message as a value from the
chat, and the answer is the next value of its `actions`: each call, the
message it went to, whether it was applied, and how sure the model was. A
failing node answers with its error, and one that says nothing for fifteen
seconds is reported as such. With several nodes, a picker in the header
chooses. The conversation is kept in the browser only, and a reload clears
it.
## A note on working this way
The pattern that works is the same one that works with a person: let the agent
read the flow and its issues, let it write a **draft**, and publish yourself
once you have looked. `validate_flow` before `publish_flow` is cheap and tells
you what the canvas would have told you.
## Limits
- MCP is not currently reachable through a [portal](../interface/portal.md)
tunnel: the proxy forwards `/api/v1/` only. Connect an agent on the same
network as the instance.
- Secrets are never readable, by an agent or by anyone else. `list_secrets`
returns names.
Choose a node, type a command, and inspect the reported actions: target
message, value, confidence, and whether each action was applied.
The node's `apply` and `min_confidence` settings decide whether it writes
to devices. Control conversations stay in the browser and clear on reload.
+168 -201
View File
@@ -1,274 +1,241 @@
# The HTTP API
Everything the browser does, the API does first. The dashboard is a generated
client of this schema, not a privileged path into the engine, so anything you
can click, you can script.
The API covers the same operations as the browser. Use it for scripts,
integrations, or your own client.
Base URL: `https://api.${DOMAIN}/api/v1`, or `http://127.0.0.1:8000/api/v1` for
a `fluksio serve` instance.
Set the base URL for your instance:
```sh
export FLUKSIO=http://127.0.0.1:8000/api/v1
```
Use the port printed by `fluksio serve`. For Docker, use
`http://api.localhost/api/v1`, `http://<host-ip>:8080/api/v1` on a LAN stack,
or `https://api.<your-domain>/api/v1` for a deployment.
All paths below are relative to this base.
## Authenticating
```sh
export TOKEN=$(curl -s -X POST $FLUKSIO/login/access-token \
-d "username=admin@example.com&password=..." | jq -r .access_token)
`fluksio serve` signs the local CLI in automatically. To reuse its token
from a project instance, with `jq` installed:
curl -s $FLUKSIO/users/me -H "Authorization: Bearer $TOKEN"
```sh
export TOKEN=$(jq -r .token .fluksio/client.json)
```
A bearer token, valid for eight days. `POST /login/test-token` checks one.
Password recovery and reset are `POST /password-recovery/{email}` and
Adjust the path for `--global` or `--data-dir`. For a remote instance,
use its login endpoint with the administrator credentials:
```sh
export TOKEN=$(curl -s -X POST "$FLUKSIO/login/access-token" \
--data-urlencode "username=admin@example.com" \
--data-urlencode "password=<your-password>" | jq -r .access_token)
curl "$FLUKSIO/users/me" -H "Authorization: Bearer $TOKEN"
```
Session tokens last eight days by default. `POST /login/test-token` checks
a token. Password recovery uses `POST /password-recovery/{email}` and
`POST /reset-password/`.
Agents authenticate differently; see [Agents over MCP](agents.md).
[Agents over MCP](agents.md) use separate OAuth tokens.
!!! tip "The interactive schema"
!!! tip "Interactive API reference"
When `ENVIRONMENT` is not `production`, the full OpenAPI schema is at
`/docs` (Swagger) and `/redoc`, and the raw document at
`/api/v1/openapi.json`, for a browser on the engine's own machine (or any,
with `API_DOCS_ENABLED=true`). That is the authoritative reference; this
page is the tour.
It is closed in production: the schema enumerates every endpoint the
instance serves, including paths webhook nodes mounted at runtime.
Outside production, open `/docs` or `/redoc` on the API host.
The schema is at `/api/v1/openapi.json`. Access is limited to loopback
unless `API_DOCS_ENABLED=true`; the Docker development stack enables it.
Production disables these endpoints.
## Flows
| Method | Path | What |
| Method | Path | Action |
|---|---|---|
| `GET` | `/flows/` | every flow, with node counts, draft and running state |
| `GET` | `/flows/{name}` | one flow — the draft if there is one — with node status and issues |
| `PUT` | `/flows/{name}` | save a draft (`version` must match, or 409) |
| `POST` | `/flows/{name}/publish` | put the draft live |
| `POST` | `/flows/{name}/discard-draft` | throw the draft away |
| `POST` | `/flows/{name}/rename` | rename it |
| `DELETE` | `/flows/{name}` | delete it. Its runs stay — a run is the record of what ran, and only `DELETE /runs` clears one. Refused with 409 while a run of it is `running` or `queued` |
| `GET` | `/flows/node-types` | every node type and its parameter schema |
| `GET` | `/flows/graph` | every flow as one graph — what Home draws |
| `GET` | `/flows/` | list flows and their state |
| `GET` | `/flows/{name}` | read the draft when present, with status and issues |
| `PUT` | `/flows/{name}` | save a draft; send the current `version` |
| `POST` | `/flows/{name}/publish` | publish; send `{"version": <current-version>}` |
| `POST` | `/flows/{name}/discard-draft` | discard unpublished edits |
| `POST` | `/flows/{name}/rename` | rename |
| `DELETE` | `/flows/{name}` | delete; refused while a run is queued or running |
| `GET` | `/flows/node-types` | list types and parameter schemas |
| `GET` | `/flows/graph` | read the graph shown on Home |
Reading a flow returns the *draft* when one exists, along with a preview of how
it would run: node statuses and validation issues for the version you are
editing, not the one running underneath it.
Deleting a flow keeps its run history. Delete runs separately.
### Node source
```sh
curl -s $FLUKSIO/flows/house/nodes/decide/source -H "Authorization: Bearer $TOKEN"
curl -X PUT $FLUKSIO/flows/house/nodes/decide/source -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"code": "def process(x):\n return {\"y\": x * 2}\n"}'
curl "$FLUKSIO/flows/house/nodes/decide/source" \
-H "Authorization: Bearer $TOKEN"
curl -X PUT "$FLUKSIO/flows/house/nodes/decide/source" \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"code": "def process(x):\n return {\"y\": x * 2}\n"}'
```
Writing a source is a superuser's, and so is any `PUT /flows/{name}` that
adds a Function or Command node, retypes or repoints one, changes a Command
node's settings, or changes any node's `resources.env`: each of them is code
the engine runs. Anyone else gets a 403 and keeps the rest — wiring, titles,
a Function node's settings and every other node.
Writing code requires a superuser. So does adding, retyping, or repointing
Function and Command nodes, changing Command settings, or changing
`resources.env`. Other users can edit wiring, titles, and ordinary settings.
`POST .../share` promotes a node's source to the shared library;
`POST .../unshare` gives it a private copy back. `GET /flows/library` lists the
shared sources and which flows use each.
Ordinary source edits stay in the draft until publishing. Shared library
source changes affect its users immediately.
`POST /flows/{name}/nodes/{id}/share` promotes a source to the library;
`/unshare` makes it private again. `GET /flows/library` lists shared sources.
### Running and controlling
| Method | Path | What |
| Method | Path | Action |
|---|---|---|
| `POST` | `/flows/{name}/start` · `/stop` | activate or tear down its subscriptions and schedules |
| `POST` | `/flows/{name}/pause` · `/resume` | hold messages instead of running them |
| `POST` | `/flows/{name}/step` | release exactly one held message |
| `POST` | `/flows/{name}/validate` | the issues, without saving |
| `POST` | `/flows/{name}/run` | run every node once, from the values you pass |
| `POST` | `/flows/{name}/nodes/{id}/trigger` | feed values into one node |
| `POST` | `/flows/{name}/nodes/{id}/cancel` | kill the worker running it right now |
| `POST` | `/flows/{name}/nodes/{id}/acknowledge` | clear the failure the canvas is marking |
| `POST` | `/flows/{name}/start`, `/flows/{name}/stop` | activate or stop a live flow |
| `POST` | `/flows/{name}/pause`, `/flows/{name}/resume` | hold or release incoming work |
| `POST` | `/flows/{name}/step` | release one held message |
| `POST` | `/flows/{name}/validate` | validate the current draft |
| `POST` | `/flows/{name}/run` | run the current draft once with `{"inputs": {...}}` |
| `POST` | `/flows/{name}/nodes/{id}/trigger` | feed a node with `{"values": {...}}` |
| `POST` | `/flows/{name}/nodes/{id}/cancel` | stop active node code |
| `POST` | `/flows/{name}/nodes/{id}/acknowledge` | clear its failure marker |
`POST /flows/{name}/run` on a **batch** flow submits a run instead, because
that is what running one means: the parameters, the series and the result are
the point.
A preview run can have real side effects. On a batch flow, `/run` submits a
recorded run; use the run endpoints below for parameters and tracking.
## Messages
```sh
curl -s $FLUKSIO/messages/ -H "Authorization: Bearer $TOKEN"
curl -X POST $FLUKSIO/messages/house.setpoint -H "Authorization: Bearer $TOKEN" \
curl "$FLUKSIO/messages/" -H "Authorization: Bearer $TOKEN"
curl -X POST "$FLUKSIO/messages/house.setpoint" \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"value": 22.5}'
curl -s $FLUKSIO/messages/house.temperature/history -H "Authorization: Bearer $TOKEN"
curl "$FLUKSIO/messages/house.temperature/history" \
-H "Authorization: Bearer $TOKEN"
```
`GET /messages/` is the catalogue: every message any published flow declares,
with its last value. Publishing puts a value into the graph exactly as a
dashboard control does, which means only a message some flow *declares* can be
published to. Flows own the namespace; everything else is a client of it.
The catalogue contains messages declared by published flows and their last
values. You can publish only to declared messages, just as dashboard controls do.
## Runs
| Method | Path | What |
| Method | Path | Action |
|---|---|---|
| `POST` | `/runs/flows/{name}` | queue one run — `{"params": {...}, "seed": 7, "draft": false, "no_cache": false}`. `"cause"` says where it came from — `api` (the default), `cli` or `sdk` |
| `POST` | `/runs/flows/{name}/sweep` | queue many, sharing a `group_id` |
| `POST` | `/runs/import` | record runs executed somewhere else, up to 200 per request. Answers `[{"external_id", "id", "created"}]` |
| `GET` | `/runs` | the queryable history: `?flow=`, `?status=`, `?group=`, `?digest=`, `?since=`, `?before=`, `?limit=`, `?offset=` |
| `DELETE` | `/runs?flow=` | every run of one flow, and everything hanging off them. `flow` is required. Refused with 409 while one of them is still going |
| `GET` | `/runs/overview` | one row per flow that has runs, with how many are running or queued |
| `GET` | `/runs/export/metrics?…&name=&stride=&format=` | every selected run's series as one long table: `run, name, step, ts, value` |
| `GET` | `/runs/export/runs?…&params=&metrics=&format=` | one row per run: its inputs as columns, its final numbers, its status and provenance |
| `GET` | `/runs/metrics/names?…` | every metric name the selected runs recorded, distinct; takes the export's own filters |
| `GET` | `/runs/{id}` | one run in full: params, result, per-node record, artifacts |
| `POST` | `/runs/{id}/cancel` | stop it |
| `DELETE` | `/runs/{id}` | forget it, with its nodes, series and artifact rows. Cancel a live one first |
| `POST` | `/runs/{id}/retry` | run the same thing again, as a new run naming this one |
| `GET` | `/runs/{id}/metrics?name=&stride=` | one metric's series, in step order; every series of the run without `name` |
| `GET` | `/runs/series/compare?ids=a,b,c&metric=&x=` | that metric across several runs. `x` is what to plot against: nothing or `step`, `time` (seconds since each run's own first reading), or another metric's name, joined on the step the two share |
| `POST` | `/runs/flows/{name}` | queue a run with `{"params": {...}, "seed": 7}` |
| `POST` | `/runs/flows/{name}/sweep` | queue runs with a shared group |
| `POST` | `/runs/import` | import up to 200 completed runs |
| `GET` | `/runs` | query history |
| `GET` | `/runs/overview` | summarize runs per flow |
| `GET` | `/runs/{id}` | read parameters, results, nodes, and artifacts |
| `POST` | `/runs/{id}/cancel`, `/runs/{id}/retry` | cancel, or create a retry |
| `DELETE` | `/runs/{id}` | delete a finished run and its records |
| `DELETE` | `/runs?flow={name}` | delete a flow's runs; refused while any are active |
| `GET` | `/runs/{id}/metrics?name=&stride=` | read metric series |
| `GET` | `/runs/metrics/names` | list metric names across a selection |
| `GET` | `/runs/series/compare?ids=a,b,c&metric=&x=` | compare curves by step, time, or another metric |
| `GET` | `/runs/export/metrics`, `/runs/export/runs` | stream CSV or JSONL |
Submitting answers immediately with a `queued` run. Wrong parameters (an
undeclared name, a value of the wrong type) come back as a 422 naming the
problem, before anything executes.
Submissions return a queued run immediately. Invalid or undeclared parameters
return `422` before execution. Optional `draft` and `no_cache` flags select
unpublished code or skip stage caching.
`/runs/import` takes `{"runs": [...]}`. Each entry carries `flow`,
`external_id`, `params`, `seed`, `status` (`ok`, `error` or `cancelled`),
`created_at`, `started_at`, `finished_at`, `result`, `metrics` (`name`,
`step`, `ts`, `value`), `artifacts` (`name`, `filename`, `media_type`,
`content_base64`), `labels` and `group_id`. The parameters are completed and
checked as a submit's are, the run is stored with the cause `import` and an
empty commit and code digest, and nothing is queued. The request is checked
whole before anything is written, and an `external_id` already imported
answers with that run and `"created": false`.
History filters include `flow`, `status`, `group`, `digest`, `since`,
`before`, `limit`, and `offset`. Rows arrive newest first; use the last
row's `created_at` as `before` for stable pagination.
`?digest=` filters by the hash of the parameters, which is how you find "every
run that used exactly this configuration".
Exports also accept `ids` and `until`, with `format=csv` or `jsonl`.
The metrics export is a long table; the runs export is one row per run.
Use `params` and `metrics` to choose columns, including dotted paths into
record outputs.
`?before=` is how a long history is paged: rows come newest first, so handing
back the last row's `created_at` reads the next page whatever landed
meanwhile, which `?offset=` cannot, since a run submitted between two pages
shifts every row down one. `?since=` is inclusive and bounds the other end.
Both exports stream `csv` (the default) or `jsonl`, and take the selection the
history takes plus `?ids=a,b,c`, `?since=` and `?until=`. `export/runs` puts
the inputs that *vary* across the selection in `param.` columns (the sweep
axis) unless `?params=` names them, and the run's numbers in `metric.`
columns. Both are dotted paths into whatever a node returned, so
`?metrics=final_metrics.train_loss` selects a field of a record and the
defaults reach every number inside one. A key with a dot in its own name, such
as `-1.0`, cannot be addressed that way and gets no column. The run id is on every row of both, which is what makes an exported
file a join back to the run rather than a loose table.
`compare` answers in the same shape a chart widget draws, so three training
curves side by side is a widget binding rather than a screen of its own.
See [Runs: pipelines that finish](../concepts/runs.md).
Import takes `{"runs": [...]}`, records completed results without executing
code, and deduplicates by `external_id`. See
[Runs](../concepts/runs.md) for examples, artifacts, caching, and exports.
## Artifacts
```sh
curl -X PUT $FLUKSIO/artifacts -H "Authorization: Bearer $TOKEN" \
--data-binary @model.pt
curl -s $FLUKSIO/artifacts/sha256:abc... -H "Authorization: Bearer $TOKEN" -o model.pt
curl -X PUT "$FLUKSIO/artifacts?name=model.pt" \
-H "Authorization: Bearer $TOKEN" --data-binary @model.pt
curl "$FLUKSIO/artifacts/<digest>" \
-H "Authorization: Bearer $TOKEN" -o model.pt
```
Content-addressed, so uploading the same bytes twice stores them once. Node
code normally reaches these through `fluksio.save_artifact` /
`fluksio.load_artifact` rather than here.
The upload returns a reference with the content digest. Identical bytes are
stored once. Node code uses `save_artifact` and `load_artifact` instead.
## Dashboards and panels
| Method | Path | What |
| Method | Path | Action |
|---|---|---|
| `GET` | `/dashboards/` | every dashboard, without its contents |
| `GET` | `/dashboards/{name}?draft=true` | the published document, or the editor's copy |
| `POST` | `/dashboards/{name}` | create one |
| `POST` | `/dashboards/from-flow/{flow}` | build `{flow}_results` from a batch flow's declared ports — a chart per streamed metric, a stat per output. 409 if it already exists |
| `GET` | `/dashboards/` | list dashboards |
| `GET` | `/dashboards/{name}?draft=true` | read a draft; omit the query for published content |
| `POST` | `/dashboards/{name}` | create |
| `POST` | `/dashboards/from-flow/{flow}` | create a batch flow's results dashboard |
| `PUT` | `/dashboards/{name}` | save a draft |
| `POST` | `/dashboards/{name}/publish` · `/discard` · `/rename` | |
| `GET` `PUT` | `/panels/` | which device shows which dashboards |
| `POST` | `/panels/pair` | start a pairing |
| `GET` | `/panels/pair/{code}` | what is holding a code |
| `POST` | `/panels/{id}/unpair` | drop this panel's credential, keep the panel |
| `POST` | `/dashboards/{name}/publish`, `/dashboards/{name}/discard`, `/dashboards/{name}/rename` | publish, discard, or rename |
| `GET`, `PUT` | `/panels/` | read or update panel assignments |
| `POST` | `/panels/pair` | start device pairing |
| `GET` | `/panels/pair/{code}` | poll pairing with its `secret` |
| `POST` | `/panels/{id}/pair` | approve the pairing |
| `POST` | `/panels/{id}/unpair` | revoke the panel credential |
## Secrets, modules, alerts
## Secrets, modules, and alerts
```sh
curl -s $FLUKSIO/secrets/ -H "Authorization: Bearer $TOKEN" # names only
curl -X PUT $FLUKSIO/secrets/influx-token -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"value": "..."}'
curl -s $FLUKSIO/modules/ -H "Authorization: Bearer $TOKEN"
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"requirements": "numpy>=2\n"}'
curl -s $FLUKSIO/alerts/config -H "Authorization: Bearer $TOKEN"
curl -X POST $FLUKSIO/alerts/test/my-phone -H "Authorization: Bearer $TOKEN"
```
Secrets are write-only over the API: you can list names and set values, never
read one back. Secrets and `POST /modules/apply` are a superuser's, and so is
what a connector keeps across restarts: `GET` or `DELETE
/connectors/{node_type}/state`, or `DELETE /connectors/{node_type}/state/{key}`
for one key (see [the connector contract](../reference/connector-contract.md#keeping-state-across-restarts)).
## Observability
| Path | What |
|---|---|
| `/observability/summary` | engine health — always 200, degraded or not — and the `version` it is running |
| `/observability/timeseries` | executions and failures over a window |
| `/observability/flows` | per-flow rollups with a 60-slice trend |
| `/observability/runs` | recent cascades, with `?flow=`, `?since=`, `?until=` |
| `/observability/events?kind=failure\|audit` | what went wrong, or who changed what; narrows by `?flow=`, `?run=`, `?since=`, `?until=` |
| `/observability/dead-letter` | work the engine gave up on |
`GET /utils/health/` is the deep health check the container probe uses: it
fails when the event loop is wedged or the state backend is gone, not just when
the process is dead.
## Workers
| Method | Path | What |
| Method | Path | Action |
|---|---|---|
| `GET` | `/workers` | what is attached, its labels and how busy it is |
| `POST` | `/workers/tokens` | mint a worker credential with its name and labels (superuser; shown once) |
| `GET` | `/workers/tokens` | the worker credentials minted, without the tokens (superuser) |
| `POST` | `/workers/tokens/{jti}/revoke` | refuse one from now on and drop a worker attached with it (superuser) |
| `GET` | `/workers/runtime` | the node runner's source, for a host without pip |
| `WS` | `/workers/attach` | where a worker dials in, with `Authorization: Bearer <worker token>` |
| `GET` | `/secrets/` | list names only |
| `PUT` | `/secrets/{name}` | set `{"value": "..."}` |
| `GET` | `/modules/` | read installed packages and the manifest |
| `POST` | `/modules/apply` | replace the manifest with `{"requirements": "numpy>=2\n"}` |
| `GET`, `PUT` | `/alerts/config` | read or update alert configuration |
| `POST` | `/alerts/test/{channel}` | test a channel |
See [Remote workers](workers.md).
Secrets and package changes require a superuser. Applying a manifest replaces
the whole managed environment's package list. Adopted environments are read-only
here; install their packages yourself.
Connector state keys can also be listed and cleared by a superuser; see
[the connector contract](../reference/connector-contract.md#keeping-state-across-restarts).
## Observability and workers
| Method | Path | Action |
|---|---|---|
| `GET` | `/observability/summary` | health and engine version; always returns `200` |
| `GET` | `/observability/timeseries`, `/observability/flows` | execution metrics or flow rollups |
| `GET` | `/observability/runs`, `/observability/events`, `/observability/dead-letter` | cascades, events, or abandoned work |
| `GET` | `/utils/health/` | deep health check; fails on loop or state-backend problems |
| `GET` | `/workers`, `/workers/resources` | attached workers and resource availability |
| `GET`, `POST` | `/workers/tokens` | list or mint worker credentials; superuser only |
| `POST` | `/workers/tokens/{jti}/revoke` | revoke a worker |
| `GET` | `/workers/runtime` | read node-runner source |
| `WS` | `/workers/attach` | attach with a worker token |
| `GET`, `POST` | `/flavors/` | list or create resource sizes |
| `PATCH`, `DELETE` | `/flavors/{name}` | update or delete a size |
Changing flavors requires a superuser. See [Remote workers](workers.md).
## Live events
```text
ws://127.0.0.1:8000/api/v1/flows/ws
```
Connect to `ws://127.0.0.1:8000/api/v1/flows/ws` with an
`Authorization: Bearer <access-token>` header.
A client that can set headers sends `Authorization: Bearer <access token>` on
the handshake. A browser cannot, so it first calls `POST /flows/ws-ticket` and
opens `/flows/ws?ticket=<ticket>`: a ticket is good for one connect, within 30
seconds, so the URL that access logs keep is worth nothing afterwards.
`?token=` is still accepted for one more release. The socket sends a snapshot
on connect and then every engine event: node executions, values published,
health changes, run started and finished. This is what the canvas and the
dashboards draw from.
Browsers first call `POST /flows/ws-ticket`, then connect with
`?ticket=<ticket>`. Tickets are single-use and expire after 30 seconds.
The socket sends a snapshot followed by node, value, health, and run events.
## Errors
## Errors and client generation
| Code | Means |
| Code | Meaning |
|---|---|
| `400` | the request was malformed, or the node refused it with a message |
| `401` / `403` | not signed in, or not allowed |
| `404` | no such flow, dashboard, run or message |
| `409` | someone else saved first — the body carries `current_version` |
| `422` | a parameter, port or binding did not typecheck |
| `503` | that subsystem is not available on this instance |
| `400` | malformed request or refused operation |
| `401` / `403` | missing authentication or permission |
| `404` | resource not found |
| `409` | version conflict, duplicate, or resource still in use |
| `413` | artifact upload exceeds the configured limit |
| `422` | invalid parameters, ports, or bindings |
| `503` | subsystem unavailable |
A 409 on a save or a publish is not an error to retry blindly: it means the
stored version moved past the one you were editing. Re-read, merge, save again.
For a version conflict, read `current_version`, reload, merge your edits, and
save again. Other `409` responses describe their own conflict.
## Generating a client
The frontend's TypeScript client is generated from the OpenAPI schema
(`make generate-client`). Any OpenAPI generator will do the same for your
language: point it at `/api/v1/openapi.json` on a non-production instance,
from the machine it runs on.
`make generate-client` regenerates the frontend SDK. To generate a client in
another language, use the instance's `/api/v1/openapi.json` schema.
+221 -468
View File
@@ -1,467 +1,261 @@
# The `fluksio` command
Install on Linux or macOS with Python 3.12 or newer:
```sh
pip install fluksio
```
Installs the engine and the `fluksio` command. Python 3.12 or newer, Linux or
macOS.
The MQTT and InfluxDB connectors, outbound mail and error reporting are
`pip install 'fluksio[server]'`: a deployment talking to devices wants them,
and a laptop waiting on them to install does not. Everything else, the engine
and every python node included, is in the plain install; a node type whose
library is missing says which extra to add when one is actually built.
There is a second, smaller distribution, `fluksio-worker`, for a machine that
should only *run nodes* for an engine elsewhere. It has none of the engine in
it. See [Remote workers](workers.md).
The command is two things at once: `serve`, `enroll` and `worker` *are* an
instance, while `login`, `sync`, `run`, `runs`, `artifacts`, `sweep` and
`status` talk to one that may be anywhere.
## Where an instance lives
`.fluksio` beside your code, found the way `.git` is: from the working
directory, or any directory above it. Two repositories on one machine are
therefore two engines, with their own flows, runs and token. `fluksio serve`
makes one where there is none, and it ignores itself from within with a
`.gitignore` of `*`, so a database and a credential cannot be committed by
accident.
`--global` uses `~/.fluksio` instead, shared by every directory. `--data-dir`
(or `FLUKSIO_HOME`) names any directory outright and wins over both.
## `fluksio serve`
Runs the engine.
```sh
fluksio serve
```
On the first start it creates an admin account and prints its password **once**.
Nothing else has to be running: no database server, no message broker, no
Docker.
For the browser dashboard, sign up at [fluksio.com](https://fluksio.com), add
an instance, then run `fluksio enroll <code>` in the same directory.
At a terminal this opens a dashboard with the engine running under it; see
[below](#the-dashboard). `--plain` prints the log stream instead, which is
also what happens with no terminal: in a container, under systemd, or in CI.
Use `fluksio --help` or `fluksio COMMAND --help` for command options.
`pip install 'fluksio[server]'` adds MQTT, InfluxDB, outbound mail and error
reporting. A machine that only executes remote nodes can install
[`fluksio-worker`](workers.md).
The default port moves out of the way when something already has it (8001,
8002, and so on) and says which one it took; the URL written to
`client.json` is the one it is actually on. A port you *asked* for is never
moved off: `--port 9000` on a taken 9000 fails, because something else is
there and you named it. It fails before `client.json` is written, so every
client keeps talking to the engine that is actually up.
## Where an instance lives
What it will *not* do is start a second engine for the same instance. Before
anything else, `serve` looks for an engine already serving this directory —
the pidfile beside the data says where, and a token this directory's key
signed says whether what answers there is ours — and stops if it finds one,
whatever port the second was asked for. One SQLite database wants one engine,
and the second would repoint `client.json` at a port that dies with it.
Another instance's Fluksio on the port is named, and the move happens as usual.
By default, Fluksio finds the nearest `.fluksio/` directory by walking up from
your working directory. `serve` creates one in the current directory if needed.
Separate project directories can therefore hold separate instances.
| Option | Default | What it does |
|---|---|---|
| `--data-dir PATH` | `./.fluksio` (or `$FLUKSIO_HOME`) | where this instance keeps everything |
| `--host HOST` | `127.0.0.1` | what to bind |
| `--port PORT` | `8000`, or the next free one | what to listen on |
| `--plain` | off at a terminal | the log stream rather than the dashboard |
| `--log-level LEVEL` | `info` | uvicorn's log level |
| `--admin-email ADDR` | `admin@example.com` | the account created on first run |
| `--admin-password PW` | generated | set it instead of having one generated |
| `--enroll CODE` | — | pair with a portal as part of coming up |
| `--portal URL` | — | the portal `--enroll` redeems at |
| `--max-runs N` | 4 | batch runs driven at once (`FLOW_MAX_RUNS`) |
| `--max-cascades N` | 4 | cascades in flight at once (`FLOW_MAX_CASCADES`) |
| `--max-workers N` | 4 | python worker processes (`FLOW_MAX_WORKERS`) |
| `--gpus N` | counted | GPUs on this machine a node may be given (`FLOW_GPUS`) |
| `--ram MB` | worked out | memory on this machine a node may be given (`FLOW_RAM`) |
| `--worker-max-rss MB` | 0 (never) | retire a warm worker holding more when its node returns (`FLOW_WORKER_MAX_RSS_MB`) |
`--global` selects `~/.fluksio`. `--data-dir PATH` or `FLUKSIO_HOME`
selects a directory explicitly. The data directory contains a `.gitignore`
that excludes its database, credentials and other contents from your project.
Cards are counted from NVIDIA's device nodes (`/dev/nvidia0`, `/dev/nvidia1`, …),
which asks no vendor tool and so keeps the one dependency from becoming two.
Anything they do not cover — another vendor, or a card this process cannot see
— reports none until `--gpus` says otherwise, and a node asking for one is then
clamped to zero and runs alongside every other. `--gpus 1` is what serialises
them. Passing a number always wins over the count.
## `fluksio serve`
`--enroll` with `--portal` is the one-command setup: it pairs before the engine
starts, so the connection is dialled as part of coming up rather than needing a
restart. It is skipped if the instance is already enrolled.
Starts the API and flow engine. No external database, broker or Docker is
needed for the default setup.
!!! warning "One process"
`fluksio serve` holds the flow engine. A second one is a *second engine*:
duplicated subscriptions, duplicated cron ticks, two webhooks answering the
same path. Run one, and distribute work with
[workers](workers.md) instead.
!!! note "`$HOME` on a cluster"
A login node's home directory is often NFS, where SQLite's write-ahead log
does not work, since the database would be locked or corrupt. `fluksio serve`
warns when it notices; point `--data-dir` at local disk.
### What it prints
```text
Created the admin account admin@example.com
password: example-password
Shown once. Change it from the dashboard.
Fluksio 0.1.0 — data in /home/you/.fluksio
API http://127.0.0.1:8000/api/v1
No portal. Pair this instance with:
fluksio enroll <code>
```sh
fluksio serve
fluksio serve --plain --port 9000
```
An enrolled instance says which portal it is on instead, and notes that the
dashboard is served from there rather than here.
The first start prints an admin password once. Local CLI commands receive a
saved token automatically; `login` is for connecting to another engine.
The default port is 8000, advancing to the next free port if necessary.
An explicit `--port` fails when occupied. The startup banner and saved client
configuration show the selected URL.
Run one engine per instance. At a terminal, a second `serve` reattaches to
that instance's running engine; plain mode refuses to start a duplicate.
| Option | Default | Purpose |
|---|---|---|
| `--data-dir PATH` | Nearest `.fluksio`, or a new one here | Instance storage |
| `--global` | Off | Use `~/.fluksio` |
| `--host HOST` | `127.0.0.1` | Bind address |
| `--port PORT` | `8000`, or next free | Listening port |
| `--plain` | Off at a terminal | Print logs instead of opening the terminal dashboard |
| `--log-level LEVEL` | `info` | Uvicorn log level |
| `--admin-email ADDR` | `admin@example.com` | First admin account |
| `--admin-password PW` | Generated | First admin password |
| `--enroll CODE` | Unset | Enroll before starting, if not already paired |
| `--portal URL` | `https://hub.fluksio.com` | Portal used by `--enroll` |
| `--max-runs N` | `4` | Concurrent batch runs |
| `--max-cascades N` | `4` | Concurrent live cascades |
| `--max-workers N` | `4` | Local Python worker processes |
| `--gpus N` | Auto-detected | Available local GPUs |
| `--ram MB` | Auto-detected | Available local memory |
| `--worker-max-rss MB` | `0` (disabled) | Retire a worker above this resident memory after its call |
Resource defaults can also come from the corresponding `FLOW_*` settings;
see [Configuration](../reference/configuration.md). GPU detection uses
NVIDIA device nodes. Set `--gpus` when your hardware is not detected.
Use local disk for the data directory on a cluster: SQLite's write-ahead log
does not support NFS.
### The dashboard
At a terminal, `serve` opens three tabs: **Overview** is the health block and
the flows, **Runs** is the history, and **Logs** is the engine's own output.
At a terminal, `serve` opens **Overview**, **Runs** and **Logs**:
| Key | What it does |
| Key | Action |
|---|---|
| `1` `2` `3` | the Overview, the Runs and the Logs |
| `q` | close the dashboard. **The engine keeps running**, and the pid is printed |
| `s` | stop the engine, or start it again |
| `r` | restart it |
| `e` | pair with a portal, without leaving the screen |
| `1`, `2`, `3` | Switch tabs |
| `q` | Close the dashboard; the engine keeps running |
| `s` | Stop or start the engine |
| `r` | Restart the engine |
| `e` | Enroll with a portal |
On the Runs tab the toolbar carries what a run is for:
On **Runs**, `space` selects runs, `enter` opens a chart or comparison,
`c` cancels a run, and `a` lists or downloads artifacts. Compare curves by
step, elapsed time or another metric.
| Key | What it does |
|---|---|
| `space` | tick the run under the cursor into a comparison |
| `enter` | compare the ticked runs, or draw the one under the cursor |
| `c` | cancel the run the cursor is on |
| `a` | list what the run left behind, and fetch it |
The engine is a child process rather than a thread, which is what makes those
possible, and what makes `q` a way out of the screen rather than a way to
stop the engine. Running `fluksio serve` again reattaches to it.
Its output goes to `serve.log` in the data directory rather than down a pipe,
which is what lets the screen be closed while the engine keeps running — a
pipe with nobody reading it breaks the next line the engine writes, and a
node's `print` is one of those. It also means the Logs tab has the output of
an engine this screen only adopted, and the scrollback of the one before it.
The engine cuts the file back to nothing once it passes 5 MB, so the bound
holds whether or not a screen is open.
The screen takes its colours from the terminal rather than painting its own,
so it sits inside a light profile as readily as a dark one.
An engine started elsewhere is adopted rather than duplicated, and can be
stopped from here only when it is this instance's own: both the pidfile
beside the data and a token this directory's key signed have to agree. Another
instance's engine is named and left alone.
The screen subscribes to the engine's event bus over the same websocket a
browser uses, so a run appears the moment it starts. It reconnects on its own,
and an engine stopped from this screen is a normal state rather than an error.
#### Comparing runs
`enter` opens a comparison of the ticked runs: one metric's curve for each,
over a table of what differs between them.
The curves are drawn in braille, five to a chart, in five distinct colours: a
departure from the browser's chart palette, where the five series
step one hue by lightness. Hue is what survives a terminal. Every curve is
named in the legend either way.
Two pickers sit above the chart. The first is the metric, named as the run
records it (`train.loss` rather than `loss`, since a name is qualified by the
node that published it). The second is what it is plotted against: the step,
`time (s)` measured from each run's own first reading so runs started hours
apart lie on top of each other, or another metric of the same runs.
The table below keeps only what actually differs (the inputs whose values are
not the same in every run, and the seed when it varies) beside each run's
status, duration and final numbers. A parameter every run shared is not a
column worth scanning.
A run that is still going has its curve re-read once a second, and the whole
comparison refreshes when the engine says a run finished.
Run `fluksio serve` again to reopen the dashboard. Its engine writes to
`serve.log` in the data directory; the log is truncated after 5 MB.
Without a terminal, or with `--plain`, the command prints logs directly.
## `fluksio enroll`
Pairs an existing instance with a portal.
Connects this instance to the portal:
```sh
fluksio enroll ABCD-1234
```
| Option | What it does |
Get a code from **Instances → Add instance** in the portal. Run the command
from the instance's project directory, or select it explicitly.
| Option | Purpose |
|---|---|
| `--portal URL` | a portal of your own, instead of `https://hub.fluksio.com` |
| `--as EMAIL` | the local account a portal session arrives as |
| `--data-dir PATH` | which instance, if not the one this directory is in |
| `--portal URL` | Use another portal instead of `https://hub.fluksio.com` |
| `--as EMAIL` | Choose the local superuser represented by portal sessions |
| `--data-dir PATH` | Select instance storage |
| `--global` | Select `~/.fluksio` |
Get the code from the portal under **Instances → Add instance**. It is
single-use and expires in fifteen minutes. `--as` matters when the instance
has several superusers. Without it, enrolment refuses rather than guessing.
With multiple local superusers, `--as` is required. A new code replaces an
existing enrollment only after the portal accepts it. A running engine picks
up the change and reconnects automatically.
An instance that is already paired is re-paired: a new claim code replaces the
connection, at the same portal or another one, and a running engine drops the
old tunnel within a few seconds. The code is redeemed before anything is
written, so one the portal rejects leaves the connection as it was. Moving to
a different portal clears the local accounts' mappings to the old one, since
nothing in a mapping says which portal issued it — the people who had access
are admitted again from the new portal.
Afterwards, `fluksio serve` dials the portal as it comes up, and keeps dialling:
a portal that restarts, a wifi that changes, a laptop that suspends and wakes
somewhere else all end the same connection, and the link is put back up without
anybody noticing. A connection that stood up and then dropped is retried at
once; one that never stood up waits a little longer each time, up to half a
minute. See [Accounts and the portal](../interface/portal.md).
The instance keeps an outbound connection to the portal and reconnects after
network interruptions. See [Accounts and the portal](../interface/portal.md).
## `fluksio worker`
Runs nodes for an engine elsewhere. Everything after `worker` belongs to the
agent's own parser: it is the same program `fluksio-worker` installs, so the
two are interchangeable:
Starts the same agent as the standalone `fluksio-worker` command:
```sh
fluksio worker --url wss://api.${DOMAIN}/api/v1/workers/attach --labels gpu
FLUKSIO_WORKER_TOKEN=your-token fluksio worker \
--url wss://api.example.com/api/v1/workers/attach --labels gpu
```
See [Remote workers](workers.md).
Create the worker token on the engine first. See [Remote workers](workers.md)
for token creation, interpreter selection and labels.
## Talking to an engine
The commands below are the client half: they run wherever you work, and
address an engine over its API rather than being one, except under `--local`,
which boots one inside the command instead.
Client commands use the nearest `.fluksio/client.json`, or the global
configuration when there is no project instance. Legacy client configuration
is a fallback.
`FLUKSIO_URL` and `FLUKSIO_TOKEN` override saved values; command options
`--url` and `--token` override both.
### `fluksio login`
```sh
fluksio login --url https://api.${DOMAIN}
fluksio login --url https://api.example.com
```
For an engine somewhere *else*. One you started yourself needs no login:
`fluksio serve` writes the token as it comes up and says where it put it.
The token goes in this project's `.fluksio/client.json`, or with `--global` in
`~/.fluksio/client.json`. Every command below reads it from there, nearest
first walking up from the working directory, or from `FLUKSIO_URL` and
`FLUKSIO_TOKEN`, or from its own `--url` and `--token`. A token an older
version wrote to `~/.config/fluksio/client.json` is still read.
Prompts for credentials and saves an API token for a remote engine.
`--global` saves it in `~/.fluksio` instead of this project. Your locally
started engine already saves a token.
### `fluksio sync`
```sh
fluksio sync [PATH_OR_MODULE ...] # default: the current directory
fluksio sync study
fluksio sync examples/myresearch
fluksio sync --dry-run study
```
Imports what you name, collects the flows the decorators declared, and uploads
each one with a generated import shim per node. A directory that is a package
is walked; a dotted name is imported as it stands; nothing is loaded from a
file path, because the shim has to import the same way.
Imports modules, packages or directories and uploads the declared flows.
With no target, it searches the current directory. Directory searches skip
hidden folders, virtual environments, `__pycache__` and `node_modules`.
File paths are not targets: use `study`, not `study.py`.
A plain directory is walked all the way down, so one folder per study
(`fluksio sync dev` over `dev/s1_baseline/study.py`) needs no naming. Hidden
directories, `__pycache__`, `node_modules` and virtualenvs are left alone.
The generated node bodies import your functions. Keep the project available
to each machine running them; sync does not copy your package to a remote
worker. Uploading source requires a superuser token.
Each file is imported under the name its path spells beneath the directory
being synced, so `dev/s1/study.py` and `dev/s2/study.py` are `s1.study` and
`s2.study` and a `study.py` per study collides with nothing. No `__init__.py`
is needed, since the directories in between are namespace packages, which leaves a
bare `from study import ...` in a test beside it working. A file at the top of
what is synced keeps its plain name.
| Flag | What it does |
| Flag | Purpose |
|---|---|
| `--dry-run` | print the flow documents and shims, upload nothing |
| `--no-publish` | leave the upload as a draft |
| `--force` | overwrite a flow, or a node body, that was edited on the canvas |
| `--dry-run` | Print flow documents and node bodies without uploading |
| `--no-publish` | Save changes as drafts |
| `--force` | Overwrite canvas edits that conflict with your local definitions |
Every sync retires the engine's workers, including one that had nothing to
upload: a worker holds your package in memory, so an edit to it is invisible
until the process goes.
It also records, per node, which of your modules that node's function imports
its way to, and what they hash to. That is what the
[stage cache](../concepts/runs.md#stage-caching) keys on, so editing a
helper a node calls into is reported as that node changing (`train: updated
(flow, fit)`) and re-runs it, while editing something the node never reaches
is left alone.
Uploading node code takes a superuser's token; anyone else's sync is refused.
A sync that changed nothing says `unchanged`, which is the answer worth
having. `published` and `draft` are said only when there was something to
publish or a draft was genuinely left behind. An engine too old to store what
a node's code reaches says so in a line naming both versions; until it is
upgraded its cache is keyed on the whole repository, as it was before. See
[Getting started: data science](../getting-started/data-science.md).
Sync records source digests for [stage caching](../concepts/runs.md#stage-caching)
and refreshes local workers, so edited imports are picked up. An unchanged
flow reports `unchanged`. See the
[data science guide](../getting-started/data-science.md) for a complete example.
### `fluksio run`
```sh
fluksio run train --lr 0.05 --seed 7 [--wait]
fluksio run train --lr 0.05 --epochs 20 --follow
```
Syncs the flow, then submits a run, so the command after an edit is this one
and nothing else, from the repository root as readily as from the study's own
folder. What it imports is the module that declared the flow, as the last
`sync` recorded it, under the same module name. Importing a module runs it, so
the other scripts in the repository are left alone.
Syncs local code, then submits a run. Flags such as `--lr` name the flow's
declared inputs and are converted to their declared types. Unknown inputs are
rejected. Use `--param name=a,b` with `sweep` for multiple values.
A flow the engine has no such record for (never synced, drawn on the canvas,
or on an engine older than the record) syncs the working directory *and
everything under it* instead, skipping whatever the repository's `.gitignore`
covers. A study there that will not import is a warning rather than a stopped
run. `--sync dev/s1_baseline` (repeatable) names what to sync outright, and
`--no-sync` skips it entirely.
After the first sync, `run` imports the module that declared the flow. If no
declaration is recorded, it searches the working directory and skips paths
covered by `.gitignore`. Import errors in that fallback search produce
warnings. Use `--sync PATH` to choose a target explicitly.
Flags that are not its own are the flow's inputs, typed by what the flow
declares them as, so a name the flow has not got is refused by name, and
`--param lr=0.002` is told that one value is `--lr 0.002` and several is a
[sweep](#fluksio-sweep). `--wait` blocks until the run
finishes and exits non-zero if it failed. `--follow` waits as well, and prints
the numbers the run reports as they arrive:
| Flag | Purpose |
|---|---|
| `--wait` | Wait for completion; exit non-zero if the run fails |
| `--follow` | Wait and print emitted metrics |
| `--timeout SECONDS` | Stop waiting after this long; leave the run active |
| `--seed N` | Record the seed and fill a declared input named `seed` |
| `--defaults` | Use declared input defaults without prompting |
| `--sync PATH` | Choose what to sync; repeat for multiple targets |
| `--no-sync` | Run the version already on the engine |
| `--no-cache` | Execute every node |
| `--local` | Start an engine in this command and wait for the result |
```text
train.loss[14] = 3.40295e-06
```
With no parameters at a terminal, the command prompts for each input.
Press Enter to keep its default. Scripted commands do not prompt.
Ctrl-C while waiting cancels the run and exits with status 130.
Ctrl-C while either is waiting cancels the run on the engine rather than only
stopping the watching, and exits 130. `--timeout SECONDS` gives up *waiting*
after that long and leaves the run going; it is nothing to do with a node's own
timeout.
A seed does not initialize your libraries' random number generators. Declare
a `seed` input and pass it to your code if it controls randomness.
`--seed N` is the experiment's seed, and it does three things. It is recorded
on the run, so what a result came from is answerable later. It goes into the
digest that identifies a run's inputs, so two runs of one configuration with
different seeds are different runs rather than a cache hit. And if the flow
declares an input named `seed`, that is what fills it, so the number the run
is labelled with is the number your code actually drew from, instead of merely
looking like it. A flow that declares no such input still records it, and
nothing reads it. Sweep over seeds with `--param seed=1,2,3`.
Any input takes what a previous run produced, named rather than typed out: a
checkpoint, but equally a config object nobody wants to paste into a shell:
Inputs can reference earlier results:
```sh
fluksio run evaluate --dataset @run:1758042000123-9f2ab41c.dataset
fluksio run evaluate --dataset sha256:6dd1f0…
fluksio run train --meta @run:1758042000123-9f2ab41c.dataset_meta
fluksio run evaluate --dataset @run:<run-id>.dataset
fluksio run evaluate --dataset sha256:<digest>
```
`@run:<id>.<output>` is whatever that run's output was, whole and with its own
type; a bare digest is the content itself, resolved into a reference. Both
spellings are reserved on every input, `str` included, so an input that has to
carry one of them literally cannot. Passing the value as JSON still works and
is what a script that already holds one does, the same thing
`flow.submit(dataset=run.result["dataset"])` does from Python.
`@run:<id>.<output>` resolves a run output; `sha256:<digest>` resolves
stored bytes into an artifact reference.
Run a flow with no parameters at a terminal and it asks for them, one line per
declared input, with the declared value in brackets:
```text
lr (float) [0.05]: 0.01
epochs (int) [10]:
dataset (artifact): @run:1758042000123-9f2ab41c.dataset
```
Enter keeps what is in brackets, so pressing it through the lot runs the
defaults. Nothing changes for a scripted run: passing any parameter, or piping
the command, skips the questions, and `--defaults` skips them explicitly.
`--no-sync` runs what is already on the engine. Worth it in a tight loop where
you know nothing changed, since syncing retires the workers and the next call
pays its imports again. A directory that declares no flows syncs nothing and
says nothing: a flow drawn on the canvas is run the same way.
`--no-cache` executes every node, including one an earlier run already
answered. See [Stage caching](../concepts/runs.md#stage-caching).
`--local` boots the engine inside this process instead of talking to a served
one, so there is no `fluksio serve` terminal to keep open. It is the same
instance either way (the same `.fluksio`, the same database, artifacts and run
history) so a run made this way and a run made through a served
engine cache against each other. It always waits, because the engine it starts
lives exactly as long as the command. Starting one costs a few seconds of
worker pool and module reconcile, against the ~15 ms of submitting to an
engine that is already up: `--local` is for "I just want to run it", not for a
loop you are iterating in.
`--local` uses the same data directory, history and artifacts as `serve`.
It starts and stops the engine for each command, so keep `serve` running
when iterating frequently.
### `fluksio status`
```sh
fluksio status [--watch]
fluksio status
fluksio status --watch
```
The home screen's top half in a terminal: whether the engine is healthy and
what is wrong if not, whether it is paired with a portal and reaching it, then
every flow with its state, its node count and whether it has unpublished
changes, and the last few runs and failures under them.
A `resources` line names each machine the engine can run a node on and how much
of it is in use, plus how many nodes are queued for one. It is absent on an
engine that accounts for nothing.
The portal reads one of three ways. `no portal` means this instance was
never enrolled. `portal hub.fluksio.com` means the link is up. `portal
unreachable` names the error, and is the one worth acting on: the dashboard is
served from the other end, so nobody can reach it while that is showing.
`--watch` keeps it on screen and refreshes every five seconds until Ctrl-C, the
cadence the dashboard polls at, since nothing here moves faster. It needs a
terminal; without one, run it without `--watch` and the output pipes cleanly.
`--local` reads the flows and history out of this directory with no engine
served, the same way `runs --local` does. It cannot be watched: an in-process
engine is the command itself, so nothing changes under it.
Shows engine health, portal connectivity, flows, recent runs and available
resources. `--watch` refreshes every five seconds. `--local` reads the local
instance without a served engine.
### `fluksio runs`
```sh
fluksio runs [--flow train] [--limit 20]
fluksio runs --flow train --limit 10
```
The runs an engine has recorded, newest first: id, status, flow, duration, how
long ago it was submitted, the commit of the repository it came from, and the
inputs it was given. Statuses are coloured when a terminal is reading the
output: `ok` green, `error` red, `cached` cyan.
Only the inputs that *differ from what the flow declares* are shown, and they
are clamped to what is left of the terminal's width. A run that took the
defaults lists none at all, and a flow taking a few kB of JSON does not push
everything else off the line. `Client.runs()` and
[`fluksio export runs`](#fluksio-export) are where the whole value is read.
`--local` reads the same history from an in-process engine, without one having
to be served.
Lists recent runs, newest first. Omit `--flow` for all flows.
`--local` reads the local instance.
### `fluksio retry`
```sh
fluksio retry <run-id> [<run-id> ...]
fluksio retry <run-id>
fluksio retry --group <sweep-id>
```
Runs the same thing again as a run of its own, carrying the flow, the inputs,
the seed and the group of the one it repeats, and recording it as its parent.
What it is for is a run an engine restart interrupted: those are marked
`abandoned` once their lease goes stale, and `--group` retries every run of a
sweep that did not end `ok`, so the missing combinations are filled in rather
than the whole grid being submitted again. The stage cache is what makes it
cheap — the nodes that finished are restored rather than recomputed.
Submits a new run with the previous inputs, seed and group, linked to its
parent. `--group` retries finished runs in that group that did not end
`ok`. Active runs must be cancelled before retrying.
A run that has not finished is refused; cancel it first.
Runs that were still *queued* when an engine stopped need none of this: the
next engine reads them back out of its own history and picks them up.
Queued runs survive engine restarts. Interrupted running runs become
`abandoned` after their lease expires; retry them to resume with any
available cached stages.
### `fluksio flavors`
@@ -469,137 +263,96 @@ next engine reads them back out of its own history and picks them up.
fluksio flavors
```
The named sizes a node can ask for, as in `@node(resources="gpu-small")`, with the
cores, memory and cards each stands for. Editing them is the Workers screen or
`POST /api/v1/flavors`; this is the read.
Lists named resource sizes, usable as `@node(resources="gpu-small")`.
Edit them in **Workers** or through the flavors API.
### `fluksio sweep`
```sh
fluksio sweep train --param lr=0.1,0.01 --param epochs=10,50 --wait
fluksio sweep train --param lr=0.01,0.1 --param epochs=10,50 --wait
```
Every combination of the parameter lists, submitted as one group: four runs
above, sharing a `group_id` and executing in parallel. Values are typed by the
flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`,
`--no-cache` and `--local` mean what they do there. `--wait` blocks until all
of them are finished and exits non-zero if any failed.
Submits every combination of the declared input values: four runs here,
sharing a `group_id`. Runs execute concurrently within the engine's limits.
To vary seeds, the flow must declare a `seed` input; use
`--param seed=1,2,3`.
`--seed`, `--sync`, `--no-sync`, `--no-cache`, `--timeout` and
`--local` work as for `run`. `--wait` exits non-zero if any run fails.
### `fluksio export`
```sh
fluksio export metrics --flow train --list
fluksio export metrics --flow train --name train.train_loss --stride 10 -o curves.csv
fluksio export runs --flow train --status ok > arms.csv
fluksio export metrics --flow train --name train.loss -o curves.csv
fluksio export runs --flow train --status ok -o results.csv
```
The two tables an analysis reads. `export metrics` is the long one, a row per
run, metric and step, which is what a plotting library takes without
reshaping; `--name` keeps the metrics it lists and `--stride` keeps every Nth
point of *each* curve. `export runs` is the wide one: a row per run with its
inputs as columns, its final numbers, its status, its duration and the commit
and digest of the code it ran.
`metrics` exports a row per run, metric and step. `--list` lists metric
names; `--name A,B` selects them; `--stride N` keeps every Nth point.
Every input the selected runs recorded becomes a column, so the schema does
not move with the selection and a filter written against one export keeps
working on the next; `--params lr,seed` narrows it to the axis a comparison is
read along. `--metrics` narrows the final numbers the same way.
`runs` exports a row per run with inputs, final numeric results, status,
duration and code identifiers. Use `--params A,B` and `--metrics A,B` to
select columns, including dotted paths such as `final_metrics.train_loss`.
An input left out of a submit is recorded at the value the flow declares for
it, so every row says what it was actually run with rather than leaving the
cell blank.
Both support `--flow`, repeated `--run ID`, `--group`, `--status`,
`--since`, `--until` and `--local`. The run ID joins results to metrics.
A node usually returns a record rather than a scalar, so both sides take
dotted paths into one: `--metrics final_metrics.train_loss,test_metrics.known.perfect`
selects three fields rather than two blobs, and `--params model.ansatz` does
the same for an input. The defaults reach the same depth: every number a
result carries becomes a column wherever it sits, and a record's inputs are
taken leaf by leaf rather than as one blob.
Output defaults to CSV on stdout. `-o FILE` writes a file; `--format jsonl`
writes JSON Lines. Parquet requires `pip install 'fluksio[parquet]'` and
`--format parquet -o FILE`.
Metric names are flow-qualified (a node of `train` writing `train_loss` records
`train.train_loss`) so `--list` prints the names the selected runs
carry when the spelling is not obvious. The engine answers it over the whole
selection, so a name only an older run ever recorded is listed too.
Both take `--flow`, `--run ID` (repeat it), `--group`, `--status`, `--since`,
`--until` and `--local`, and both put the run id on every row: it is the join
back to the run page and to what the run made.
`--format` is `csv` (the default), `jsonl` or `parquet`; output goes to stdout
unless `-o FILE` names somewhere. Parquet keeps the types, needs pyarrow
(`pip install 'fluksio[parquet]'`) and needs a file to write, since it is not a
stream. In a notebook, `Client.export_metrics()` and `Client.export_runs()`
answer the same rows as a list of dicts, which `pandas.DataFrame` takes
directly.
In Python, `Client.export_metrics()` and `Client.export_runs()` return the
same rows as lists of dictionaries.
### `fluksio artifacts`
```sh
fluksio artifacts 1758042000123-9f2ab41c
fluksio artifacts 1758042000123-9f2ab41c weights -o model.npz
fluksio artifacts <run-id>
fluksio artifacts <run-id> train.weights -o weights.json
```
The files a run produced: what `fluksio.save_artifact(...)` wrote, and any
artifact a node returned. Named alone it lists them: the message each left on,
its size, its media type and the filename the node gave it. Name one and it is
written here, under that filename unless `-o` says otherwise.
The message name is the one to pass, since it is what addresses the bytes;
`--local` reads them from this directory without an engine served.
Lists a run's returned artifacts, including their message names, sizes and
filenames. Pass a name from that list to download it. Output uses the stored
filename unless `-o` selects another path. `--local` reads the local instance.
## What lives in the data directory
```text
.fluksio/ (or ~/.fluksio, with `--global`)
├── client.json the token `serve` wrote, mode 600
├── serve.pid the engine serving this directory, while one is
├── serve.log what the engine under the dashboard printed
├── .gitignore `*`, so a database and a credential are ignored from within
├── fluksio.db SQLite: users, runs, metrics, observability, agents
├── flows/ a git repository, one directory per flow
│ ├── house/
│ │ ├── flow.json the published structure
│ │ ├── nodes/*.py the published node code
│ │ ├── flow.draft.json unpublished edits, if any
│ │ └── nodes.draft/*.py
│ ├── _lib/ shared node sources
│ ├── _dashboards/ dashboards, drafts and all
│ └── requirements.txt what the Modules screen installs
├── artifacts/ content-addressed bytes, two levels deep
├── user-venv/ the interpreter your node code runs on
├── secrets.enc encrypted credentials, deliberately outside flows/
├── alerts.json alert channels and rules
├── panels.json wall-panel pairings
├── oauth-key.pem signs agent tokens
├── cloud.json the portal enrolment, if there is one
├── secret_key signs sessions and derives the secrets key
└── env optional settings file
```
The main files and directories are:
Two things follow from this layout and are worth internalising:
| Path | Contents |
|---|---|
| `client.json` | Local API URL and token |
| `fluksio.db` | SQLite database: accounts, runs, metrics and operations history |
| `flows/` | Git repository of flows, node sources, dashboards and module requirements |
| `artifacts/` | Files stored by content digest |
| `user-venv/` | Managed node environment, when used |
| `secrets.enc` | Encrypted credentials, outside the flow repository |
| `cloud.json` | Portal enrollment, when paired |
| `secret_key` | Session signing and encryption key |
| `serve.pid`, `serve.log` | Running engine information and terminal dashboard logs |
| `env` | Optional instance settings |
**`flows/` is a real git repository.** `git log` is the history of every change
anyone made to any flow. A run records the commit it ran at, so `git show` on
that hash is literally the code that produced the number.
The flow repository records edits in Git. Runs record its commit and code
digests; code-defined nodes also depend on your project files, so keep that
project's history too.
**Backing up the data directory backs up the instance.** Everything else is
rebuildable. Copy it while the engine is stopped, or use SQLite's online backup
for the database if it is not.
Back up the data directory while the engine is stopped. An adopted venv and
your project files live outside it and need their own backup.
## Settings
Settings come from the environment, or from an `env` file in the data
directory. The ones you are most likely to touch:
Settings come from environment variables or an `env` file in the data
directory. Common choices:
| Variable | Default | What it does |
|---|---|---|
| `DATA_DIR` | `./.fluksio` via the CLI; `~/.fluksio` with `--global` | everything below it derives from this |
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links |
| `ENVIRONMENT` | `local` | `production` closes the interactive API schema |
| `MCP_ENABLED` | `false` | opens the [agent endpoint](agents.md) |
| `SECRET_KEY` | generated once | signs sessions, derives the secrets key |
| Variable | Purpose |
|---|---|
| `FLUKSIO_HOME` | Select the CLI's data directory |
| `NODE_VENV` | `auto` adopts an existing venv; `managed` requests a separate one; a path selects an interpreter |
| `REDIS_HOST` | Use Redis for flow state and queues |
| `FRONTEND_HOST` | Browser address used in mail, OAuth metadata and panel links |
| `ENVIRONMENT` | `production` disables interactive API documentation |
| `MCP_ENABLED` | Enable the [agent endpoint](agents.md) |
The full list is in [Configuration](../reference/configuration.md).
See [Configuration](../reference/configuration.md) for defaults and the full
list.
+97 -113
View File
@@ -1,146 +1,130 @@
# Writing a connector
A walkthrough from an empty directory to a node on the canvas. The normative
surface is in [the connector contract](../reference/connector-contract.md);
this is how to use it.
A connector adds a device or service integration to the node palette. It is
a Python package with a `ConnectorNode` subclass and an entry point.
This example works without hardware or another repository.
## Start from the skeleton
## Create the package
A working connector, kept as small as a correct one can be, lives at
`skeleton/` in the [connectors repository](https://git.stroblme.de/Fluksio/connectors).
Copy it:
Make this directory beside your project:
```bash
git clone https://git.stroblme.de/Fluksio/connectors.git
cp -r connectors/skeleton fluksio-connector-mydevice
cd fluksio-connector-mydevice
```text
fluksio-connector-random/
pyproject.toml
src/
fluksio_connector_random/
__init__.py
```
Rename the package (`src/fluksio_connector_skeleton/` and the `[project]` name
in `pyproject.toml`), then point the entry point at your class:
Put this in `pyproject.toml`:
```toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "fluksio-connector-random"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["fluksio>=0.1,<0.2"]
[project.entry-points."fluksio.node_types"]
my_device = "fluksio_connector_mydevice:MyDevice"
random_sensor = "fluksio_connector_random:RandomSensor"
[tool.hatch.build.targets.wheel]
packages = ["src/fluksio_connector_random"]
```
## Install it into the engine
Connectors are found through installed package metadata, so the engine has to
have yours installed. During development, install it as an editable path
dependency of the backend:
```bash
cd app/backend
uv add --editable ../../fluksio-connector-mydevice
```
Editing your connector's **code** then takes effect on the next engine restart,
with no reinstall. Changing its **metadata** (the entry point, the version,
the package name) needs `uv sync` to regenerate the installed metadata before
a restart picks it up.
Two things to know about this:
- `uv add` writes the path dependency into `backend/pyproject.toml` and
`uv.lock`. That path does not exist inside the backend's Docker image, so
**revert both files before committing**; the dependency belongs on your
machine, not in the repo. Distribution is a later milestone.
- For the same reason, develop against the **standalone backend** rather than
the Docker stack:
```bash
cd app && make dev-backend # FastAPI on :8000
make dev-frontend # Vite on :5173
```
Restart the backend and your node type appears in the add-node palette (⌘P),
labelled with the package it came from.
## Write the node
Put this in `src/fluksio_connector_random/__init__.py`:
```python
from typing import Any
import random
from pydantic import Field
from fluksio.flow.connector import CONTRACT_VERSION, ConnectorNode
class MyDevice(ConnectorNode):
class RandomSensor(ConnectorNode):
contract = CONTRACT_VERSION
title = "My device"
description = "Reads temperature and humidity over the device's HTTP API."
title = "Random sensor"
description = "Emits a random reading. No sensor required."
class Params(ConnectorNode.Params):
host: str = "192.168.1.50"
api_key: str | None = Field(
default=None, json_schema_extra={"x-secret": True}
)
ceiling: float = Field(default=1.0, gt=0)
async def poll(self) -> dict[str, Any] | None:
reading = await self._read()
return {"temperature": reading.temp, "humidity": reading.humidity}
```
Then, on the canvas: place the node, set `poll_interval` to how often the
device should be read, pick the API key from the secrets store, and add output
ports named `temperature` and `humidity`. The port names are the wiring: any
node consuming those messages is downstream of this one.
## If the device is told rather than asked
A connector that commands something implements `write()` instead of, or
alongside, `poll()`. Input ports carry the values in:
```python
def write(self, **ports: Any) -> None:
self._device.set(ports["level"])
```
Place the node with *input* ports named `level`, and whatever publishes that
message drives the device. Set `idempotent = False` on the class: a write is a
command, and a redelivery after a crash should not undo a newer one.
Keep the first version off the wire. A boolean setting the code checks before
it sends (`artnet`'s `transmit` is the example) lets a flow be built and
watched in the logs before anything physically moves.
## If the reading is bytes
A camera or a microphone publishes a reference rather than the bytes:
```python
async def poll(self):
jpeg = await asyncio.to_thread(self._grab)
return {"frame": self.save_artifact(jpeg, "f.jpg", media_type="image/jpeg")}
return {"reading": random.random() * self.config.ceiling}
```
Place the node with an `image`-typed output port and a Media widget draws each
frame as it lands. `fluksio-connector-test-media` publishes test frames and
tones this way, so the whole path can be wired up with no camera in the room.
Copy it if yours is a media device.
The entry point name, `random_sensor`, becomes the stored node type.
Choose a unique name; the engine skips names that are already registered.
## Try it without a device
## Install and try it
Give `poll` something predictable first and confirm the values reach the canvas
before pointing it at hardware:
Use the **engine's Python environment**, the one where you installed Fluksio:
```python
async def poll(self):
return {"temperature": 21.5}
```sh
pip install -e /path/to/fluksio-connector-random
```
Press Run, or wait for the poll interval, and the value appears on the edge.
Anything the connector prints shows up in the logs panel in the dock, and a
raised exception shows there with its traceback and marks the node down.
Restart the engine from its original project directory so it keeps using
the same instance data:
## Checklist before sharing it
```sh
fluksio serve
```
- `contract = CONTRACT_VERSION` declared on the class itself.
- Credentials marked `x-secret`, never read from the environment.
- `stop()` is safe to call twice, and after a failed `start()`.
- Blocking I/O wrapped in `asyncio.to_thread`.
- `poll()` returns `None` rather than raising when there is simply nothing new.
- `write()` returns quickly; the thread it runs on is the scheduler's.
- Health reported when the connection comes and goes.
- Anything that has to outlive a restart in `self.store`, keyed by device.
Restart an already running engine after installing. In the browser:
1. Add a **Random sensor** node to a live flow.
2. Set `poll_interval` to `5` seconds.
3. Add an output port named `reading`, type `float`.
4. Publish the flow and watch the value change.
The palette shows the package name and version. Code edits need an engine
restart; metadata changes also need another editable install.
For Docker, install the package in the backend image and rebuild it.
The Modules screen installs packages for **node code**, so it does not
install connectors into the engine.
## Connect a real device
Replace `poll()` with your device read. Return a dictionary keyed by declared
output port, or `None` when there is nothing new. The base class polls at
`poll_interval`, publishes changed values, and reports failures.
Keep blocking I/O off the event loop with `asyncio.to_thread`. For pushed
updates, manage the connection in `start()` and `stop()`, leave
`poll_interval=0`, and call `self.inject({...})` when data arrives.
Mark credential parameters with
`Field(default=None, json_schema_extra={"x-secret": True})`. The editor
then stores a secret reference rather than the credential in the flow.
To send commands, implement `write(self, **ports)`: it receives values keyed
by input port. Set `idempotent = False` for commands that must not be replayed
after a crash.
## Media and persistent state
Use `self.save_artifact(data, name, media_type=...)` for bytes such as camera
frames. Return the reference through an `image`, `audio`, or `video` port.
For disposable live frames, pass `volatile=True`.
Use `self.store` for short strings that must survive a restart, such as a
pairing token. Include the device address in the key: all nodes from the
same top-level package share this store.
## Before sharing
- Declare `contract = CONTRACT_VERSION` on the class itself.
- Return only declared output ports, with values matching their types.
- Mark credentials as secrets.
- Keep `stop()` safe after a failed start and when called twice.
- Report connection health if you manage your own subscription.
See [the connector contract](../reference/connector-contract.md) for lifecycle,
storage limits, and certificate pinning.
+132 -218
View File
@@ -1,323 +1,237 @@
# Writing node code
A Function node is a Python file: no base class, no decorator, no framework
import unless you want one.
A Function node runs Python in a worker process. Write its source in the
canvas, or declare functions in your project with `@node` and
[`fluksio sync`](cli.md#fluksio-sync).
```python
def process(temperature, setpoint=21.0):
"""Ask for heat when the room is below the comfort point."""
return {"heat": temperature < setpoint}
```
Declare `temperature` as an input port, `heat` as a Boolean output port, and
`setpoint` as a setting. Now you've got a tiny thermostat.
## The rules
**One function called `process`.** If the file defines exactly one public
function under another name, that one is used instead. Two, and the node
refuses to load rather than guessing.
- **Define `process`.** Without it, the worker accepts exactly one public
function defined in the source. Multiple candidates produce an error.
- **Arguments arrive by name** from input ports and settings. A setting cannot
share a name with a port. See [Where values come from](../concepts/values.md).
- **Return a dictionary keyed by output port.** Undeclared keys and values of
the wrong type fail validation. With `@node(provides=Port("heat", "bool"))`,
the SDK also accepts a bare value and wraps it for that single port.
- **Use JSON-compatible values.** For files, use artifacts. For persistent
state, use [flow state](../concepts/state.md); worker globals are not a
reliable store.
**Arguments come from ports and settings, by name.** `temperature` above is an
input port; `setpoint` is a setting typed into the node's panel. Both arrive as
keyword arguments, which is why a setting may not share a name with a port.
See [Where a node's values come from](../concepts/values.md).
**The return value is a dict keyed by output ports.** Every value is checked
against the port's declared type before it is published. A key that is not a
declared port is an error, not a silent drop. Nothing leaves a node except
through a port it declared.
**Nothing else is importable from the engine.** Node code runs in a separate
process, on a separate interpreter, with none of Fluksio's own modules on its
path. What it can import is what the [Modules](../interface/operations.md)
screen installed, which can include [your own project](#your-own-code-as-a-package),
so a node need not be a self-contained file.
**A node is a pure function of its inputs.** No context object, no global
store, no handle to reach for. A running total or a debounce timer has a
specific shape; see [Keeping state in a flow](../concepts/state.md).
Inside a node, `import fluksio` exposes reporting and artifact helpers. It
does not expose the engine's internal modules. Other imports come from the
node's Python environment; see [Packages](#packages).
## Producing values over time
A node that produces values *during* its execution is a generator. Every
`yield` is a dict keyed by output port, published the instant it happens:
Yield dictionaries to publish outputs before the node finishes:
```python
import math
def process(lr, steps):
loss = 1.0
for _ in range(steps):
loss = train_one_step(lr)
yield {"loss": loss} # published now
for step in range(steps):
loss = math.exp(-lr * (step + 1))
yield {"loss": loss}
return {"final_loss": loss}
```
Whatever the generator `return`s at the end is the node's result, which is what
downstream nodes read. If you never `return`, the last thing you yield is the
result instead.
Mark the port so the flow says what it does:
Declare `loss` as a streaming float port:
```json
{"name": "loss", "dtype": "float", "stream": true}
```
Two consequences. In a [run](../concepts/runs.md), the whole series is kept as
that run's metrics, which is why there is no `log_metric()` anywhere in the
API. And **the node's timeout starts measuring silence rather than duration**:
each emission resets the deadline, so a node yielding every few seconds can run
for hours under a timeout of 300.
Each yield is type-checked and published immediately. The final return is
the node's result; without a return, the last yield is the result. A
[run](../concepts/runs.md) keeps numeric emissions as metric series and waits
for the result before running downstream nodes. A live flow can also trigger
downstream nodes from emissions.
### `fluksio.emit`
Where a `yield` cannot reach, because the value comes from inside somebody
else's callback and they call you rather than the other way round:
Use `emit` when a callback produces the value:
```python
import fluksio
def on_epoch_end(epoch, logs):
fluksio.emit(loss=logs["loss"])
```
Pass this callback to your training framework. It publishes through the same
declared ports as `yield`. Outside a node, `emit` does nothing, so ordinary
tests can still call your code.
## Bytes: artifacts
Save files or bytes with `save_artifact`, then return the reference through an
`artifact` port:
```python
import fluksio
def process():
model.fit(callbacks=[LambdaCallback(
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
return {"weights": ...}
```
Same ports, same type checking, same publication. Prefer `yield` where you can
reach it; `emit` where you cannot.
## Bytes: artifacts
Messages are JSON, which is what lets the same value pass through Redis, the
work queue and the worker protocol unchanged. A checkpoint is not that.
```python
import fluksio
def process(dataset):
path = fluksio.load_artifact(dataset) # → a local path to read
...
return {
"weights": fluksio.save_artifact("model.pt", media_type="application/octet-stream"),
"score": 0.94,
"report": fluksio.save_artifact(
b"Experiment complete\n", "report.txt", "text/plain"
)
}
```
`save_artifact` takes bytes or a path, stores them by their SHA-256 digest, and
returns a small reference (digest, size, media type, name) which is what an
`artifact`-typed port carries.
Another node can read it with `fluksio.load_artifact(report)`, which returns
a local file path. These two helpers require a running node; for tests,
keep the underlying calculation in ordinary Python functions.
Because the address is the content's hash, a sweep whose fifty configs share
one preprocessed input stores it once, and a reference stays valid wherever the
store is reachable from, including on another machine.
`fluksio artifacts RUN NAME` is how one comes back out at a terminal.
### `fluksio.logger`
A node's `print` is kept as that node's logs, and so is anything on
`fluksio.logger` is the same logger the SDK exports at top level, so code that
runs both inside a node and outside one says it the same way:
```python
fluksio.logger.info("resuming from epoch %d", start)
```
Artifacts are stored by their SHA-256 digest, so identical bytes share storage.
Use [`fluksio artifacts`](cli.md#fluksio-artifacts) to list and download a
run's files.
### Media
Say what the bytes are and the port can be typed for them:
`image`, `audio` and `video` ports carry artifact references with a matching
media type. For example, save JPEG bytes with `media_type="image/jpeg"` and
return the reference through an `image` port. See
[Payload types](../reference/payload-types.md#image-audio-video).
```python
def process(speech): # an `audio` port
clip = fluksio.load_artifact(speech)
words = transcribe(clip)
return {"transcript": words} # a `str` port
```
```python
def process(camera_url):
for index, jpeg in enumerate(grab(camera_url)): # a generator
frame = fluksio.save_artifact(
jpeg, f"frame-{index:05d}.jpg",
media_type="image/jpeg", volatile=True,
)
frame["meta"] = {"seq": index}
yield {"frame": frame} # an `image` stream port
```
An `image`, `audio` or `video` port is an artifact reference whose media type
has to match, so a node declaring `audio` never receives a video by accident.
See [Payload types](../reference/payload-types.md#image-audio-video) for what
each carries and what rates are realistic.
`volatile=True` is for a frame rather than a result. The bytes go to a ring in
memory instead of the data volume, and the engine pushes them down the
websocket to whichever screens are drawing that message — so a camera runs at
ten frames a second without writing anything to disk. They last as long as it
takes newer frames to need the room. Leave it off for a clip somebody asked to
keep.
!!! warning "Emitted media is not kept; returned media is"
Only what a node *returns* is recorded against its run. Frames yielded
along the way are replaced in state by the next one, and the artifact sweep
removes bytes nothing refers to any more, which is what stops a camera
filling the disk. If a particular frame matters, return it: a volatile one
is copied out of the ring when the run records it, so returning it is also
what makes it outlive the next few seconds.
For transient frames, `save_artifact(..., volatile=True)` uses an in-memory
ring and streams bytes to connected screens. New frames eventually replace
old ones. A volatile artifact returned as a run result is copied to persistent
storage; emitted frames alone are not recorded as run artifacts.
## Printing
`print` works and is captured. The first 16 KB per call is kept and shown in
the flow editor's log panel and on the run's per-node record; the rest is
dropped, so a node printing in a loop cannot fill anything up.
Use it to debug, not to record results. A number worth keeping is an output
port, not a line of text.
`print` and `fluksio.logger.info(...)` are captured in the node logs. The
worker retains the first 16,384 characters per call. Use logs to debug and
output ports for numbers you want to plot or compare.
## Errors
An exception fails that node's execution, not the flow. The message you see is
one line from the frame in *your* code, not a stack through the engine.
The node keeps its last error visible after it recovers, so a failure that
fired an alert at 03:00 still says what it was at 09:00. It can also be
acknowledged from the canvas.
Exceptions appear on the node, with a short message and a traceback. In a live
flow, one failed execution does not stop the whole flow. In a batch run, node
failure makes the run fail. The canvas keeps the last error visible after
recovery until you acknowledge it.
## Timeouts
`timeout` on a node is how many seconds its code may be *silent* before it is
stopped. A yield or an `emit` resets the clock, and the first call's imports are
not charged to it: a node importing torch is not being slow, it is loading.
`timeout` is the allowed number of seconds without an emission. A `yield` or
`emit` resets the deadline; initial compilation and imports are excluded.
There is no timeout by default. Training runs for hours and a node that reports
nothing is usually working, so the engine waits: what fails a call is the worker
dying, which arrives at once rather than after a deadline. Set a timeout on the
nodes where silence means stuck (an HTTP call that should answer in seconds, a
loop that can spin) either per node or, for all of them,
with `FLOW_NODE_TIMEOUT`. `timeout = 0` says explicitly that this node has none.
There is no timeout by default. Set one on a node where silence means stuck,
or use `FLOW_NODE_TIMEOUT` for the instance default. `timeout=0` explicitly
disables it for a node. A worker disappearing fails its active call without
waiting for that deadline.
## Running a node somewhere else
A node declares the label of the machine it needs:
Set a device label to select an attached worker:
```json
{"id": "train", "device": "gpu", "device_policy": "require", "timeout": 7200}
```
`require` (the default) waits for a worker carrying that label; `prefer` runs
locally when none is attached. A node bound to a device is compiled *on that
machine*: a node importing `torch` is correct on the GPU box and a missing
module on the engine, so checking it here would fail something that is fine.
See [Remote workers](workers.md).
`require` is the default and waits for a worker with that label. `prefer` uses
the local engine when none is attached. Code is compiled on the selected
machine, which needs the packages your node imports. See
[Remote workers](workers.md) for setup and resource requests.
## Sharing code between flows
A node's source can be promoted to the shared library from its panel, and other
flows can then use it by reference. There is one copy and one place to edit, and
every flow using it runs the edit.
Promote a node's source to the shared library from its panel. Other flows can
reference that source, and edits apply to all of them. Shared sources live in
`_lib/` in the instance's flow repository.
Shared sources live in `_lib/` in the flow repository, so they are versioned
with everything else.
For code-defined flows, reuse the same decorated function. `use(fn, ...)`
changes its wiring or settings for one flow:
```python
use(fit, wire={"dataset": "augmented"}, epochs=3)
```
## Packages
Node code runs in a virtual environment of its own, on the instance's data
volume, separate from the one Fluksio itself runs on.
The startup banner shows the interpreter used for local nodes:
Declare what you import in [Modules](../interface/operations.md), or over the
API:
| Environment | How to add packages |
|---|---|
| Fluksio installed in an existing venv | Use pip or uv in that venv, then `fluksio sync` to refresh workers. Modules is read-only. |
| Managed node venv | Edit the manifest in **Modules** and apply it. |
| Remote worker | Install packages in the interpreter selected by `--python`. |
With `NODE_VENV=auto`, Fluksio adopts its existing venv unless this instance
already has a managed one. Without an existing venv, it creates a managed
environment. `NODE_VENV=managed` always requests one; the Docker image uses
this setting.
In managed mode, the Modules manifest is installed with `uv pip sync` and
versioned with the flows. Applying it refreshes local workers. The API accepts
the same manifest:
```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
curl -X POST "$FLUKSIO/modules/apply" -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"requirements": "numpy>=2\npandas\n"}'
```
It is a pip manifest installed with `uv pip sync`, versioned alongside your
flows. An install takes effect immediately; nothing restarts.
Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN` is an
access token; see [The HTTP API](api.md).
### Your own code as a package
!!! tip "If Fluksio is installed in the venv you work in, skip this"
Node code then runs on that environment, so your project and everything it
imports are already importable; see
[Getting started: data science](../getting-started/data-science.md). What
follows is for a Fluksio with a venv of its own, which is what a container
always has.
A manifest line can name a directory, so the project you already have is
installable like any other dependency:
`fluksio sync` generates an import shim pointing at your local project. A
managed environment can also install your project from a Modules manifest:
```text
-e /home/you/my-research
numpy>=2
```
A node body then imports it, and the logic stays where it already lives: in
your repository, under your own version control, importing its own siblings:
Then a canvas node can import ordinary helpers from it:
```python
from myresearch.train import fit
from myresearch.training import train_model
def process(lr, epochs):
return fit(lr, epochs)
return {"score": train_model(lr, epochs)}
```
That is the whole of it. The node is three lines, `myresearch` can be as many
modules as it likes, and nothing was copied.
Editable installs expose changed files, but warm local workers still hold
imports in memory. Apply the Modules manifest or run `fluksio sync` after
edits. Attached workers start a process per call and read the files afresh.
!!! tip "You can have those three lines written for you"
Decorate `fit` with `@node(...)` where it is defined, say which nodes make
a flow with `Flow(...)`, and `fluksio sync` generates the body above,
along with the flow document, so there is nothing to PUT by hand. The
declaration lives beside the function it describes and is checked against
its signature. See
[Getting started: data science](../getting-started/data-science.md).
!!! warning "Editable, but not live"
`-e` means edits reach the venv without reinstalling, but a node's process
already holds the imported module in memory. The engine's workers are
long-lived, so a change to your code is picked up when they are retired,
which is what **Apply** does. Pressing it after an edit is the loop.
A [worker](workers.md) you attach yourself is the exception: it starts a
process per call, so it reads your code fresh every run. If you are
iterating on the code many times an hour, point one at your own
interpreter (`fluksio-worker --python "$(which python)"`) and mark the
node with its label.
!!! note "The path is a deployment detail"
It is resolved on whichever machine runs the node, and the manifest is
committed to the flow repository, so an absolute path from your laptop
means nothing inside a container or on a GPU box. Those need their own
install of the same project; a VCS requirement
(`myresearch @ git+ssh://…@a1b2c3d`) travels where a path does not.
Paths are resolved on the machine running the node. A remote worker or
container needs its own copy or installation of your project; syncing a shim
does not ship the package.
## A worked example
The repository ships a small supervised fit as a seedable demo: three nodes,
a batch flow, streaming metrics, artifacts between stages, and a GPU-labelled
node that falls back to the engine when no worker is attached. It is the
shortest complete thing to read:
[Getting started: data science](../getting-started/data-science.md) contains a
complete runnable flow. The public repository also includes
`examples/myresearch/`, with separate preparation, training and evaluation
modules. From the repository root:
```text
prepare ──dataset(artifact)──▶ train ──weights(artifact)──▶ evaluate
│
└── loss (streaming float) ──▶ chart
```sh
fluksio sync examples/myresearch
fluksio run train --lr 0.05 --wait
```
The example is built by a seed script the maintainers run against a live
stack; the flow above is what it produces.
Start an instance first, as shown in the getting started guide.
## See also
- [Where a node's values come from](../concepts/values.md)
- [Keeping state in a flow](../concepts/state.md)
- [Runs: pipelines that finish](../concepts/runs.md)
- [Node types](../reference/node-types.md) — the ones you do not have to write
- [Node types](../reference/node-types.md)
+120 -207
View File
@@ -1,119 +1,77 @@
# Remote workers
The engine runs where the automations are. The GPU is somewhere else, the
Raspberry Pi with the relays is in a shed, and neither of them is on the same
network as the other.
A worker runs node code on another machine: a GPU host, a Raspberry Pi or a
batch allocation. It connects outward to the engine over an authenticated
WebSocket. Keep one engine; add workers where the hardware lives.
A **worker** is a process that runs the code of nodes marked for it. It dials
*out* to the engine over one authenticated websocket, so nothing on that
machine has to be reachable, and nothing has to expose the engine's state
backend across hosts, which it never should.
## Create a worker token
## Install and attach
On the engine, use a superuser access token. In these examples:
- `API_URL` is the engine's API base, including `/api/v1`, such as
`https://api.example.com/api/v1` or `http://localhost:8080/api/v1` for a LAN stack.
- `TOKEN` is your superuser bearer token. The login endpoint
`POST /api/v1/login/access-token` accepts form fields `username` and `password`.
Mint a token with the labels this machine should carry:
```sh
pip install fluksio-worker
fluksio-worker \
--url wss://api.${DOMAIN}/api/v1/workers/attach \
--labels gpu,cuda12 \
--python /opt/torch-venv/bin/python
```
`fluksio-worker` is its own distribution: the agent, the node runner, and
`websockets`. Nothing of the engine, so a GPU box does not install a database
driver in order to run a training step. An engine host already has it, and
`fluksio worker …` is the same program.
| Option | Default | What it does |
|---|---|---|
| `--url` | **required** | `wss://…/api/v1/workers/attach` |
| `--token` | `$FLUKSIO_WORKER_TOKEN` | the credential, minted on the engine; on the command line any user of the machine reads it from `ps`, so prefer the variable |
| `--name` | this host's name | sent along; the worker list shows the token's name |
| `--labels` | the token's | comma-separated; must match the labels the token was minted with |
| `--python` | this interpreter | the interpreter node code runs on |
| `--parallel` | `1` | how many node calls it will take at once |
| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |
| `--cpus` | what the job or the machine has | cores to advertise |
| `--gpus` | what the job says, else counted | GPUs to advertise |
| `--ram-mb` | what the job or the machine has | memory to advertise, in MB |
| `--max-idle` | never | stop after this many seconds with nothing running |
| `--insecure` | off | allow `ws://` and `http://` to an engine on another machine |
The worker refuses a plain `ws://` or `http://` address unless the engine is on
this machine (`localhost`, `127.0.0.0/8`, `::1`): its token and every secret a
node is handed would cross the network in clear. `--insecure` is for a network
you trust, such as an engine on the same LAN without TLS. Node code gets this
host's environment less anything named like a credential (`*TOKEN*`,
`*SECRET*`, `*PASSWORD*`), the same as on the engine; hand a node a credential
as a [secret](../interface/operations.md#secrets) instead.
`--python` is the important one. It is how this machine keeps its own wheels
(the CUDA build, the vendor SDK, the thing that will not install anywhere else)
without the engine ever installing them or knowing about them.
### What it says it has
A worker reports its inventory when it attaches (cores, GPUs and memory) and
the engine schedules against it: a node asking for two cores and a GPU goes to
a machine that has them free, not merely to one carrying the right label.
Cores and memory are read off the machine, or off the batch job that started
this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **What the job says
it was given always wins for GPUs** (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`,
or `FLUKSIO_WORKER_GPUS`): a node with eight cards may have granted this job
one, and advertising eight would be a lie the scheduler acts on. With nothing
said, NVIDIA's device nodes are counted, the same as the engine does for its
own machine, and `--gpus` overrides either. No vendor tool is asked, which is
what keeps the one dependency from becoming two. A worker that reports nothing
still attaches and is scheduled by its label alone, as every worker was before
any of them reported anything.
The engine tells each call what it may use: thread caps, and the devices it
may see. The worker starts a process per call, so it applies them at the only
moment a numerical library still reads them: before the import.
`--max-idle` is for a worker something else started for one job, a batch
scheduler say. It exits when nothing has run for that long, so the allocation
goes back rather than idling until its walltime.
## Mint the token
On the engine, as a superuser:
```sh
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
curl -X POST "$API_URL/workers/tokens" \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}'
```
Shown once, valid for a year, since a worker is a machine somebody sets up and
leaves running. The name and labels are signed into it: labels decide which
nodes, with their source and secrets, are sent to a worker, so a worker cannot
choose its own. It shows up under the token's name, and a worker whose
`--labels` differ from the token's is refused.
The response contains `token`, `jti` and the expiry. Save the token as
`FLUKSIO_WORKER_TOKEN` on the worker host. It is returned once and is valid
for 365 days. The signed name and labels control which nodes and secrets the
engine sends to that worker.
## Install and attach
Use Python 3.12 or later on the worker host:
```sh
curl -s $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" | jq
curl -X POST $FLUKSIO/workers/tokens/$JTI/revoke -H "Authorization: Bearer $TOKEN"
pip install fluksio-worker
export FLUKSIO_WORKER_TOKEN='<token from the engine>'
fluksio-worker \
--url wss://api.example.com/api/v1/workers/attach \
--labels gpu,cuda12 \
--python /opt/torch-venv/bin/python
```
The first lists what was minted, by `jti` (the tokens themselves are not
kept); the second refuses one from then on, here and at the artifact
endpoint, and drops a worker attached with it. Rotating the keypair agent
tokens are signed with still revokes every worker at once. A token minted
before tokens carried their labels is refused: mint a new one.
Use your engine's hostname. `--python` selects the environment containing the
node's dependencies, such as PyTorch or a device SDK. If you install the worker
in that environment, you can omit the option. An engine installation also
provides the same agent as `fluksio worker`.
??? note "A host where pip is not an option"
| Option | Default | Purpose |
|---|---|---|
| `--url` | required | engine's `/api/v1/workers/attach` WebSocket |
| `--token` | `$FLUKSIO_WORKER_TOKEN` | credential; prefer the environment variable to keep it out of process arguments |
| `--name` | hostname | suggested name; the engine uses the token's name |
| `--labels` | token's labels | comma-separated; when supplied, must match the token |
| `--python` | worker's interpreter | Python used for node code |
| `--parallel` | `1` | concurrent calls |
| `--artifact-url` | derived from `--url` | artifact endpoint, if hosted separately |
| `--cpus` | detected | advertised CPU count |
| `--gpus` | detected | advertised GPU count; explicit `0` means none |
| `--ram-mb` | detected | advertised memory in MB |
| `--max-idle` | `0` | exit after this many idle seconds; `0` disables |
| `--insecure` | off | allow unencrypted connections to a non-loopback host |
The two files work copied into one directory and run with `python agent.py
…`. The engine serves the runner itself at `GET /api/v1/workers/runtime`.
It is the same module its own local workers run, deliberately standard
library only.
For a trusted LAN stack, the URL is
`ws://<host-ip>:8080/api/v1/workers/attach` and requires `--insecure`.
Outside loopback, the worker otherwise requires encrypted URLs (`wss://` and
`https://`): node secrets and the worker token travel over the connection.
Node code receives a filtered host environment. Credential-like variables are
removed; pass credentials through [Secrets](../interface/operations.md#secrets).
## Send a node to it
A node declares the label of the machine it needs:
Set these fields on a node through `PUT /api/v1/flows/{name}`:
```json
{
@@ -124,94 +82,73 @@ A node declares the label of the machine it needs:
}
```
| `device_policy` | Behaviour when nothing carrying the label is attached |
This is a node fragment; keep the node's other fields and the rest of the flow
when updating it. `device` and `device_policy` are not in the node panel yet.
| `device_policy` | If no matching worker is attached |
|---|---|
| `require` (default) | the run stays `queued` and says what it is waiting for |
| `prefer` | it runs on the engine instead |
| `require` (default) | queues until a matching worker is available |
| `prefer` | falls back to the engine |
`prefer` is what makes a flow work before the GPU box exists. `require` is what
you want once it does.
Source code travels with each call and compiles on the worker, so remote-only
dependencies do not need to be installed on the engine. Calls to `emit`,
`save_artifact` and `load_artifact` work there too.
!!! note "Not in the panel yet"
Cancelling a run stops its active node processes. A disconnected worker fails
in-flight calls. While executing, the agent sends heartbeats every ten seconds;
90 seconds without contact fails the call. Heartbeats do not reset the node's
own silence timeout.
`device` and `device_policy`, which machine a node's code runs on, are set
through the API rather than the panel, with `PUT /flows/{name}`.
## Resources and artifacts
## What follows from this
Workers report CPU, GPU and memory inventory, and scheduling checks both labels
and available resources. CPU and memory detection uses Slurm allocations when
present, otherwise host figures. GPU detection uses `SLURM_GPUS_ON_NODE`,
`FLUKSIO_WORKER_GPUS` or `SLURM_JOB_GPUS` before counting NVIDIA device nodes.
Explicit flags override detection.
- **The node's source travels with every call.** Nothing has to be deployed to
the worker, and changing a node's code takes effect on the next execution.
- **A node bound to a device is compiled on that machine.** A node importing
`torch` is correct on the GPU box and a missing module on the engine, so
checking it here would fail something that is fine.
- **`import fluksio` inside a node is the worker's own reporter.** `emit`,
`save_artifact`, `load_artifact`, installed before your code runs, so an
installed `fluksio` package on that box never shadows it.
- **Cancelling a run kills what it is executing**, there or here, and leaves
other runs of the same node alone.
- **If the worker disappears mid-call**, the run fails in seconds with
`worker went away mid-call` rather than waiting out its timeout.
- **A worker sends a heartbeat every ten seconds while it executes**, so a long
node is distinguishable from a dead socket. Ninety seconds of nothing at all,
not even a heartbeat, fails the call as gone. A heartbeat says the *agent* is
alive and nothing about the node, so it never satisfies a node's own timeout:
one set to thirty seconds fires after thirty seconds of the node reporting
nothing, wherever it runs.
Artifacts are identified by content hash. Remote workers fetch and upload over
HTTP beside the WebSocket endpoint; node code uses the same artifact helpers
as on the engine. Downloads are cached by digest:
## Artifacts across machines
| Variable | Default |
|---|---|
| `FLUKSIO_ARTIFACT_CACHE` | `fluksio-artifacts` in the temporary directory |
| `FLUKSIO_ARTIFACT_CACHE_BYTES` | `1073741824` (1 GiB) |
An artifact reference names content by its hash, not a location, so it stays
valid wherever the store is reachable from. A worker that shares the engine's
filesystem writes to it directly; one that does not fetches and uploads over
HTTP, using the artifact endpoint beside the socket it already has. Either way
your node code is the same two calls.
Oldest files are removed when the cache exceeds its limit. Raise it if workers
repeatedly use large inputs.
A fetch is cached on the worker by digest, since content addressing means an
entry is never stale. Nothing expires on its own, so the cache is bounded by
size and the oldest fall out: `FLUKSIO_ARTIFACT_CACHE` says where it lives
(default a directory in the temporary directory) and
`FLUKSIO_ARTIFACT_CACHE_BYTES` how much it holds (default 1 GiB). Worth raising
where a worker reads the same large inputs repeatedly, and worth leaving alone
where it reads a media stream — those are chunks nothing asks for twice.
## Seeing what is attached
The **Workers** screen shows attached hosts, available resources and queued
nodes. The API exposes the same information:
```sh
curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq
curl "$API_URL/workers" -H "Authorization: Bearer $TOKEN"
curl "$API_URL/workers/resources" -H "Authorization: Bearer $TOKEN"
```
Name, labels, how many calls it will take at once, how many are in flight, when
it attached, when it was last seen, its Python version, a digest of its
environment, and what it says it has: cores, GPUs and memory.
## Revoke or upgrade
## Upgrading
List tokens and revoke one by its `jti`:
The engine and the worker speak a version-matched protocol, and a worker
announcing anything else is refused rather than half-understood. Protocol 2,
the one that carries inventory, is `fluksio-worker` 0.2.0. An older agent is
told so on the socket and stops, rather than retrying against an engine that
will never accept it; `pip install -U fluksio-worker` on that host is the whole
upgrade. Nothing changed in the runner served at `GET /api/v1/workers/runtime`,
so a host that copies its two files copies the same one as before.
```sh
curl "$API_URL/workers/tokens" -H "Authorization: Bearer $TOKEN"
curl -X POST "$API_URL/workers/tokens/$JTI/revoke" \
-H "Authorization: Bearer $TOKEN"
```
A token minted before tokens carried their labels and a `jti` is refused. Mint
each existing worker host a new one with `POST /workers/tokens`, naming the
labels it should carry, and swap it in for `FLUKSIO_WORKER_TOKEN`.
Revocation also disconnects attached workers and blocks artifact access.
Replacing the OAuth signing key revokes all worker tokens.
**Upgrade the engine before its workers.** A worker sends its token as an
`Authorization` header on the handshake, where it stays out of access logs, and
an engine from before that reads it only from the URL: it refuses the newer
worker at the handshake. The engine still takes the old `?token=` for one
release, so older workers keep attaching to a newer engine meanwhile.
Upgrade the engine before its workers, then run `pip install -U fluksio-worker`
on each host. Engine and worker must speak the same protocol; a mismatch is
reported and refused. Older tokens without signed labels and a `jti` need to
be replaced.
## Machines from a batch scheduler
## Slurm provisioning
A cluster is not a machine that attaches and stays; it is a queue somebody else
owns. So Fluksio does not submit *nodes* to Slurm. It submits a job whose payload
is an ordinary worker dialling back in, and from there everything works the way
it already does: the same protocol, the same artifacts, the same cancellation.
Write the clusters into `provisioners.json` beside the flows:
The engine can submit a Slurm job that starts a worker. Put this configuration
in `$DATA_DIR/provisioners.json`, replacing the example addresses and paths:
```json
[{
@@ -219,58 +156,34 @@ Write the clusters into `provisioners.json` beside the flows:
"name": "hpc",
"login": "me@login.cluster",
"ssh_key": "/secrets/hpc_ed25519",
"engine_url": "wss://api.${DOMAIN}/api/v1/workers/attach",
"engine_url": "wss://api.example.com/api/v1/workers/attach",
"max_idle_s": 300,
"provision_timeout_s": 900,
"profiles": [{
"name": "gpu-small",
"cpus": 8, "gpus": 1, "ram_mb": 65536,
"cpus": 8,
"gpus": 1,
"ram_mb": 65536,
"labels": ["gpu"],
"sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"],
"prerun": ["module load cuda/12", "source ~/venvs/flux/bin/activate"]
"prerun": ["module load cuda/12", "source ~/venvs/fluksio/bin/activate"]
}]
}]
```
A **profile** is what the scheduler is asked for, where a flavor is what a node
asks for. They are separate on purpose, and agree when you set them up to.
The engine uses system SSH to submit `sbatch`. Prepare the environment named
by `prerun` with `fluksio-worker` and your node dependencies; the generated job
does not install packages. Compute nodes must be able to connect to the
engine. Each job receives its own worker token, valid for 24 hours.
When a node needs a machine nothing attached can give, and a profile fits, the
engine `sbatch`es one over the system `ssh`, and the run waits meanwhile, saying
so. `prerun` owns the environment: a `module load`, or a venv with
`fluksio-worker` already in it. The generated script runs no `pip install`.
It hands the worker a token of its own, good for 24 hours, as
`FLUKSIO_WORKER_TOKEN` rather than `--token`, so other users of a shared
compute node cannot read it from `ps`.
One outstanding request per profile, however often it is asked for. A job that
never attaches within `provision_timeout_s` is `scancel`led, as is anything
outstanding when the engine stops. `--max-idle` is what ends the job at the
other end, so an allocation goes back rather than idling to its walltime.
Both take `0` for "no limit": `provision_timeout_s: 0` waits for as long as the
queue does, which is what a cluster that queues overnight needs, and
`max_idle_s: 0` keeps the machine for the job's whole walltime.
`GET /workers/resources` reports what is outstanding and what last went wrong.
!!! note "It needs a route out"
A compute node must be able to open a connection to the engine. That is
true of most clusters and false of air-gapped ones; there is no staging
path today.
## What a worker is not
It is not a second engine. Subscriptions, schedules, webhooks, the dashboards
and the run queue all stay in one process, which keeps a message having one
definition and a cron tick happening once. A worker executes node bodies.
Running two engines against one data directory is not supported. Distribute
work with workers.
Only one request per profile is outstanding at a time. Jobs that fail to attach
within `provision_timeout_s` are cancelled, as are pending requests when the
engine shuts down. `max_idle_s` releases idle workers. Either setting accepts
`0` to disable its limit. Provisioning status appears under
`GET /api/v1/workers/resources`.
## See also
- [Runs: pipelines that finish](../concepts/runs.md#running-a-node-somewhere-else)
- [Writing node code](nodes.md)
- [Getting started: data science](../getting-started/data-science.md)
- [Runs](../concepts/runs.md#running-a-node-somewhere-else): remote execution.
- [Writing node code](nodes.md): node functions and artifact helpers.
- [Getting started: data science](../getting-started/data-science.md): resource declarations.
+104 -157
View File
@@ -1,204 +1,151 @@
# Flows, nodes and messages
Three ideas hold the system up. Almost everything else follows from them.
A **flow** is a group of **nodes** that exchange **messages**. Nodes declare
what they consume (`requires`) and produce (`provides`); matching message names
connect them. No wires to draw.
## A flow is a graph you did not draw
## How nodes connect
A **flow** is a set of **nodes**. Each node declares the messages it needs
(`requires`) and the messages it produces (`provides`). The graph is whatever
those declarations imply:
For a Function node, declare input and output ports in its panel, then write
`process(...)`. Arguments use input port names; the returned dictionary uses
output port names:
```python
# node "read"
# Node "read": provides temperature
def process():
return {"temperature": read_sensor()}
return {"temperature": 18.5}
# node "decide"
# Node "decide": consumes temperature, provides heat
def process(temperature, setpoint=21.0):
return {"heat": temperature < setpoint}
```
`decide` is downstream of `read` because it needs `temperature` and `read`
produces it. Nobody drew a wire.
Bind both temperature ports to the same message and `decide` follows `read`.
Here `setpoint` uses a Python default; you can also add it as a node
[setting](values.md#settings-constants-of-this-node).
The consequences:
- **Fan-in is free.** Two nodes providing `temperature` are two producers of
one message. The consumer does not change.
- **A node runs when something it reads was published.** Not merely when
something upstream of it ran: a node that produced nothing this time (held
back by a rate limit, say) leaves what reads it on the value it already
has, and so does everything behind that.
- **A wire cannot be wrong.** There is no wire. There is a name that either
matches or does not, and the canvas tells you at edit time which it is.
- **Layout is not a document.** The canvas computes the arrangement, so a flow
has no stored positions to maintain, merge or fight over.
- **Flows stay small.** A graph nobody can hand-arrange is one worth keeping
small, which is the intent. Several atomic flows that name each other beat
one flow with sixty nodes in it.
- Several nodes can produce the same message. Consumers use the latest value.
- Publishing an input wakes its consumers once their required values are
available. An upstream node that publishes nothing leaves downstream nodes
waiting for a new publication.
- The editor arranges the graph automatically. Keep flows focused and share
messages between them when needed.
### Message names are namespaced
Inside flow `house`, a message named `temperature` is really
`house.temperature`. A bare name is qualified with its own flow; a dotted name
is used as written. That is how two flows share a value:
In flow `house`, `temperature` becomes `house.temperature`. A dotted name is
used as written, so another flow can consume it directly. Bind an input port
named `temperature` to `house.temperature`:
```python
# in flow `dashboard`, reading a message that flow `house` produces
def process(house_temperature): # port bound to "house.temperature"
...
def process(temperature):
return {"shown": temperature}
```
The canvas draws messages arriving from another flow as labelled endpoints, so
you can see where they come from without opening the other flow.
The canvas shows cross-flow messages as labelled endpoints.
## A node is a function with declared ports
## Node types and payloads
Most nodes are **Function** nodes: a Python file defining `process(...)`. Its
arguments are its input ports by name; its return value is a dict keyed by
output ports.
```python
def process(reading, unit="C"):
return {"shown": reading if unit == "C" else reading * 1.8 + 32}
```
`reading` is a port. `unit` is a **setting**: a constant of this node's code,
typed into its panel and stored with the flow. Both arrive as arguments, which
is why a setting may not share a name with a port. See
[Where a node's values come from](values.md).
The rest of the node types are the ones that would be tedious or unsafe to
write yourself: MQTT, HTTP, InfluxDB, schedules, switches, notifications. Each
one is configured by filling in a form the editor generates from its parameter
schema, so they all behave the same way. The full list is in
Function nodes run your Python code. Built-in nodes handle MQTT, HTTP,
InfluxDB, schedules, filtering and more through forms in their panels. See
[Node types](../reference/node-types.md).
### Ports are typed
Every port declares a payload type (`dtype`). Values are checked against it,
and dashboard widgets offer compatible messages. Scalars, structured JSON and
media references are covered in [Payload types](../reference/payload-types.md).
A port declares a `dtype`: `float`, `int`, `str`, `bool`, `json`, `record`,
`list`, `series`, `artifact`, or one of the media types `image`, `audio` and
`video`. Every value that passes through is checked against it.
Types are not decoration. They are what lets the dashboard editor offer you
only the messages a gauge can actually draw, and what lets the canvas refuse a
binding before anything runs. See [Payload types](../reference/payload-types.md).
Everything on the wire is JSON. Bytes (a checkpoint, an image, a model)
travel as an `artifact`: the bytes go to a content-addressed store and the
message carries a small reference to them. The media types are that same
reference, saying what kind of bytes are behind it.
Messages carry JSON. For binary data, store the bytes as an **artifact** and
publish its reference. `image`, `audio` and `video` are artifact references
restricted to their media family.
### Streaming ports
A port marked `stream` produces repeatedly *during* one execution rather than
once at the end: a training loss, a progress fraction, a frame from a camera, a
second of speech. A node publishes on one by being a generator and yielding, or
by calling `fluksio.emit`.
Mark an output **streaming** when it publishes during an execution: progress,
training loss or media chunks. A Function node can yield values or call
`fluksio.emit`.
Each value is delivered to the nodes reading it, in the order it was produced,
so a recogniser slower than the microphone in front of it still sees every
chunk rather than only the newest. What is in state remains the latest value,
which is what everything else reads, and what a run keeps is the whole series.
Downstream nodes receive each publication in order; message state holds the
latest value. A batch run records the full series. An output `interval` limits
live publication without thinning the run's recorded series.
An `interval` on a streaming port thins what reaches the canvas without
thinning the run's record of it.
### Keeping node code predictable
### Nodes are pure
Write Function nodes as functions of their declared inputs and settings.
Fluksio provides no hidden `context` or global flow store. To remember a value
between executions, use a message the node reads and writes; see
[Keeping state in a flow](state.md).
A node is called with the values of the messages it declares and returns the
values of the messages it provides. There is no context object, no global
store, no handle to reach for.
## Live and batch flows
That is deliberate: a node with hidden state cannot run twice in parallel,
cannot be replayed, and cannot be moved to another machine. Plenty of real
automations do need to remember something, and there is a specific way to say
so; see [Keeping state in a flow](state.md).
## Two shapes of flow
Set `mode` on the flow:
Choose **Mode** in the flow settings:
| | `live` (default) | `batch` |
|---|---|---|
| Runs | continuously | once per run, on request |
| Started by | subscriptions, schedules, webhooks | `POST /runs/flows/{name}` |
| Ends | never | when the graph drains |
| Keeps | the last value of each message | a run record: params, result, metrics, artifacts |
| Is | a thermostat, an ETL job on a cron | an experiment, a CI-style job |
| Runs | reacts continuously | once per requested run |
| Started by | subscriptions, schedules, webhooks or manual execution | Run in the editor, SDK or API |
| Finishes | stays active until stopped | when its work finishes |
| Keeps | latest message values | parameters, results, metrics and artifacts |
| Example | thermostat | experiment or build job |
A batch flow is built and validated like any other, appears on the same canvas
and is type-checked the same way. It is simply never *activated*: no
subscriptions, no schedules, no webhooks. See
[Runs: pipelines that finish](runs.md).
Batch flows use the same editor and validation, but do not activate background
subscriptions, schedules or webhooks. See [Runs](runs.md).
## Editing is separate from running
## Flow inputs
Every flow has a published version and, while you are working, a draft.
- **Saving** writes the draft. The engine keeps running the published version.
- **Publishing** promotes the draft. The engine reloads and picks it up.
- **Discarding** throws the draft away.
The store is a git repository (`flow.json` for the structure, `nodes/*.py` for
the code) and each save is a commit. So a flow's history is readable with
ordinary git tooling, and copying a flow between instances is copying a
directory.
Saving carries the version you last saw. If someone else saved in between, you
get a 409 instead of quietly overwriting their work.
## What can be wrong, and when you find out
The canvas validates continuously and names problems on the nodes they belong
to:
| Issue | What it means |
|---|---|
| `unconnected_input` | a port needs a message nothing in reach provides |
| `missing_initial_value` | the message exists but has never held a value, and nothing will give it one. Not reported on a batch flow: its inputs arrive with the run |
| `cycle` | A waits for B and B waits for A — nothing could ever start |
| `self_loop_needs_initial` | a node reads a message it also writes, with no starting value |
| `node_error` | the node's code did not load: a syntax error, a missing import |
| `unauthenticated_hook` | advisory — a webhook with no shared secret is open to anyone |
| `node_unhealthy` | the node loaded but is not working: a connector that cannot reach its device, or whose last publication failed |
A flow with any of these except the advisory one and `node_unhealthy` does not
run. A node reporting itself down is a live condition, not a build error, so
the rest of the flow keeps going and the issue clears by itself once the node
reports well again. The health summary on Home counts them, so "why is nothing
happening?" has an answer that does not involve reading logs.
## What happens at runtime
- **A flow can be started and stopped.** Stopped means its subscriptions and
schedules are torn down.
- **A flow can be paused and stepped.** Paused holds messages instead of
running them; step releases exactly one. This is how you test something
before it moves a relay.
- **A failing node does not take the flow down.** It reports an error, keeps
its last error visible after it recovers, and can fire an alert.
- **A flow whose background tasks keep crashing is quarantined.** The engine
stops restarting them and says so, rather than spinning. Publishing a change
gives it another chance.
## Values that arrive from outside
Some messages are not computed by any node: a dashboard control writes them,
the API publishes them, a batch run passes them in. Declare those as the flow's
**inputs**, with the value they start from:
Declare messages supplied by dashboard controls, the API or run parameters as
the flow's **Inputs**. Give live inputs a starting value:
```json
{"inputs": [{"spec": {"name": "setpoint", "dtype": "float"}, "initial": 21.0}]}
```
Without that, the node reading `setpoint` waits for something nothing provides,
and the canvas says so. With it, the flow starts at 21.0 and whatever writes
the message afterwards takes over.
This seeds `house.setpoint` in flow `house` if it has no value yet. Later
publications replace it. Batch inputs are run parameters, and batch outputs
select the run's result. See [Where a node's values come from](values.md).
## Drafts and publishing
- **Saving** stores a draft; the engine keeps running the published version.
- **Publishing** promotes the draft and reloads the flow.
- **Discarding** removes the draft.
Shared node code is an exception: saving it updates every flow using it
immediately. See [The flow editor](../interface/flow-editor.md#code-and-reuse).
Flows are stored in a git repository: `flow.json` describes the graph and
`nodes/*.py` holds code. Saves are versioned, and stale versions return a
conflict instead of overwriting another editor's changes.
Python installations need Git on the host to record history. Without it,
flows still save as files; Docker includes Git.
## When a flow needs attention
The canvas reports issues on the affected nodes:
| Issue | Meaning |
|---|---|
| `unconnected_input` | no node or declared flow input supplies the message |
| `missing_initial_value` | a live input has no starting value |
| `cycle` | nodes depend on each other in a loop |
| `self_loop_needs_initial` | a node reads and writes a message with no starting value |
| `missing_source` | a Function node has no saved code |
| `node_error` | code or configuration could not load |
| `unauthenticated_hook` | a webhook has no shared secret; advisory |
| `node_unhealthy` | a loaded node reports a runtime problem, such as a lost connection |
Build issues prevent the affected flow from running. The last two are runtime
or advisory issues: other work continues, and node health clears when the
connection recovers.
You can **stop** a live flow to remove its subscriptions and schedules, or
**pause** it to hold pending work while values still arrive. **Step** releases
one held item. Repeated background crashes can quarantine a flow; publishing
a change gives it another chance.
## Where to next
- [Where a node's values come from](values.md) — ports, settings, flow inputs
- [Keeping state in a flow](state.md) — the sanctioned way to remember something
- [Runs: pipelines that finish](runs.md) — the batch half of the engine
- [Writing node code](../code/nodes.md) — the practical side of `process()`
- [Where a node's values come from](values.md) — ports, settings and flow inputs
- [Keeping state in a flow](state.md) — remembering a value
- [Runs](runs.md) — batch flows
- [Writing node code](../code/nodes.md) — practical Python examples
+178 -389
View File
@@ -1,476 +1,265 @@
# Runs: pipelines that finish
A flow that automates a house never ends: a value arrives, nodes fire, and it
waits for the next one. A research pipeline is the other shape: parameters go
in, stages execute in order, and at some point it is *done* and has produced
something worth keeping. Fluksio calls the second one a **run**, and it is the
same engine either way.
A **live flow** reacts to events. A **batch flow** starts with parameters,
runs its graph once and returns a result. Each execution is a **run**, with
its own state, parameters, node records, metrics and artifacts.
This is what makes Fluksio usable where Kedro, MLflow or ClearML would be: a
run has parameters that identify it, a result, per-step metrics, artifacts and
a place in a queryable history, with no second server and no
project bootstrap on every execution.
Use runs for experiments, reports and other jobs with a finish line. For a
complete first example, start with the
[data science guide](../getting-started/data-science.md).
## A batch flow
Set `mode: "batch"` on the flow and name the messages its result should hold:
In Python, `Flow(...)` defaults to `mode="batch"`. Declare `inputs` for
run parameters and `outputs` for the final result. On the canvas, choose batch
mode and define the same inputs and outputs.
```json
{
"name": "train_polymer_gnn",
"mode": "batch",
"outputs": ["final_loss", "report"],
"inputs": [
{"spec": {"name": "lr", "dtype": "float"}, "initial": 0.1},
{"spec": {"name": "steps", "dtype": "int"}, "initial": 3000}
],
"nodes": [{"id": "train", "timeout": 7200, "device": "gpu", "...": "..."}]
}
```
A batch flow does not need activation and does not open subscriptions,
schedules or webhooks. A run executes each node once after its dependencies
finish. Missing parameters use their declared initial values.
A batch flow is built and validated like any other (it appears on the canvas,
its ports are type-checked) but it is never *activated*: no subscriptions, no
schedules, no webhooks. It runs when a run asks it to, and not otherwise.
Its `inputs` are its parameters. A run supplies values for them; anything it
does not supply keeps the declared initial value.
One thing a batch flow may not do is rate-limit a port (`interval`). A rate
limit holds a value back for a timer to release, and a run has no timer, so the
value would be dropped rather than delayed, so submitting is refused instead.
Non-streaming ports cannot use `interval` in a batch flow: there is no timer
to deliver a delayed value. Streaming ports can use it to reduce display
updates while keeping every metric point.
## Submitting
**Runs → New run** in the app is the same call with a form in front of it: pick
a flow, fill in its declared inputs, press Run. A comma-separated list in a
number field runs every combination of them as a sweep. **Use draft** runs the
working copy rather than what is published, which is how a change is tried
before it is deployed, and each run keeps its own state, so nothing it computes
reaches the live flow.
In **Runs → New run**, choose a flow, enter its inputs and press **Run**.
**Use draft** runs the working copy without publishing it.
```bash
curl -X POST $FLUKSIO/runs/flows/train_polymer_gnn \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"params": {"lr": 0.3, "steps": 4000}, "seed": 7}'
From a terminal:
```sh
fluksio run train --lr 0.1 --epochs 20 --wait
```
The answer is immediate and the run is `queued`; a training run is measured in
hours, so nothing waits for it. Poll `GET /api/v1/runs/{id}` for its status,
result, per-node record and artifacts.
Or submit through the API:
Wrong parameters are refused before anything executes: an undeclared name, or
a value of the wrong type, comes back as a 422 naming the problem.
```sh
curl -X POST "$FLUKSIO/runs/flows/train" \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"params": {"lr": 0.1, "epochs": 20}, "seed": 7}'
```
`"draft": true` on the body runs the unpublished working copy — the flow
document and the node sources the editor is showing. The copy is read again
when the run starts, so a draft published while it waited runs as the
published one.
Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN`
is an access token. See [The HTTP API](../code/api.md) for authentication.
Submission returns immediately with status `queued`. Read
`GET /api/v1/runs/{id}` for status and results. Unknown parameters or wrong
types return 422 before execution.
The API body also accepts `"draft": true`, `"no_cache": true` and an
`idempotency_key`. Reusing an idempotency key returns the original submission
instead of creating another run. Drafts are read when execution starts, so
edits made while a run is queued can affect it.
A seed is recorded on the run and fills a declared input named `seed`.
Your node code must use that input to seed its random number generator.
### Sweeps
An ensemble is the same parameters at different seeds; a grid search is the
parameters spread out. Both are one call, and the caller builds the list:
A sweep submits every combination of input values as one group:
```bash
curl -X POST $FLUKSIO/runs/flows/train_polymer_gnn/sweep \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{
"runs": [{"params": {"lr": 0.1}, "seed": 1}, {"params": {"lr": 0.3}, "seed": 1}]
}'
```sh
fluksio sweep train --param lr=0.01,0.1 --param epochs=10,50 --wait
```
From a terminal that is `fluksio sweep train_polymer_gnn --param lr=0.1,0.3`,
which builds the product of the lists you give it and posts the same call.
These four runs share a `group_id` and execute concurrently within the
engine's limits. Each has its own state, so one run cannot overwrite another's
messages. Vary seeds with `--param seed=1,2,3` if the flow declares that input.
They share a `group_id`, so `GET /api/v1/runs?group=…` is the sweep, and they
execute in parallel. That is safe because **each run has a state backend of its
own**: message names are global keys, so two runs of one flow would otherwise
overwrite each other's values. They do not.
The API equivalent is `POST /api/v1/runs/flows/{name}/sweep` with a
`runs` list of parameter dictionaries and optional seeds. Filter history
with `GET /api/v1/runs?group=<group-id>`.
## Producing values before you are finished
A training loop has numbers worth keeping long before it has a result. Those
numbers are **outputs**, not logs: a node declares a port for them and produces
them over time, which in Python is a generator.
Training curves and progress values leave through output ports. A Python
generator publishes them with `yield`; callbacks can use `fluksio.emit`.
Declare the port with `stream: true`. See
[Writing node code](../code/nodes.md#producing-values-over-time) for examples.
```python
def process(lr, steps):
loss = 1.0
for _ in range(steps):
loss = train_one_step(lr)
yield {"loss": loss} # published now, on the `loss` port
return {
"weights": fluksio.save_artifact(dump(model), "weights.npz"),
"final_loss": loss,
}
Every numeric emission is stored under its flow-qualified message name,
such as `train.loss`, with a step counter and timestamp. Retrieve it with:
```text
GET /api/v1/runs/{id}/metrics?name=train.loss
GET /api/v1/runs/series/compare?ids=<a>,<b>&metric=train.loss
```
Mark the port it streams on, so the flow says what it does:
A streaming port's `interval` reduces updates sent to the canvas, while the
run history keeps every point. In a run, intermediate emissions do not start
downstream nodes; those wait for the node's final result.
```json
{"name": "loss", "dtype": "float", "stream": true}
```
Every yield is published the instant it happens: same port, same type check,
same place on the canvas as any other value. Whatever the generator `return`s
is the node's result, and is what downstream nodes read. If you never
`return`, the last thing you yield is the result instead.
There is no `log_metric()`. A metric leaves through a port, which makes it a
message: a chart binds to it directly, a downstream node can consume it, and the
run keeps its series.
**Where a yield cannot reach** — the value comes from inside somebody else's
callback, and they call you rather than the other way round. `fluksio.emit`
writes the same ports the same way:
```python
import fluksio
def process():
model.fit(callbacks=[LambdaCallback(
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
return {"weights": ...}
```
### What a run does with them
Every number a node emits is kept as the run's series, stepped by the count of
emissions on that message. Read one back with
`GET /api/v1/runs/{id}/metrics?name=<flow>.loss`, or leave `name` off for
every series the run kept, or compare runs:
```
GET /api/v1/runs/series/compare?ids=<a>,<b>,<c>&metric=<flow>.loss
```
That answers in the `series` shape a chart widget already draws, so three
training curves side by side is a widget binding. During a run the values also
arrive live on the flow socket, so a chart bound to the port fills in as the
training goes.
A streaming port may set `interval` to thin out what reaches the canvas. The
run's history still keeps every value, because the interval is asking for the
display not to be flooded, not for the curve to have holes in it.
Emitting has a second effect: **a node's timeout measures silence, not
duration**. A node that yields every few seconds can run for hours under a
`timeout` of 300; one that says nothing for longer than its timeout is killed.
There is no timeout unless one is set, since a training node that reports
nothing is usually working. Set `timeout` where going quiet means stuck, at how long
the node may plausibly be.
In a *live* flow, an emission also wakes whatever is downstream of it, exactly
as a subscriber publishing does. In a run it does not: a run's graph is
scheduled once, and three thousand mid-node cascades would leave "the run has
finished" with nothing to mean.
A node's `timeout` measures silence. Each yield or emit resets it, so a node
can run for hours while reporting progress. No timeout is imposed by default.
## Artifacts
Bytes never travel as a message. `save_artifact` writes them to a
content-addressed store and returns a small reference (digest, size, media
type, name) which is what an `artifact`-typed port carries:
Use `fluksio.save_artifact` for checkpoints, datasets, plots and other bytes.
Return its reference through an `artifact` port. Another node calls
`fluksio.load_artifact` to obtain a local path.
```python
def process(weights): # requires: weights, dtype "artifact"
path = fluksio.load_artifact(weights)
...
Identical bytes share one stored copy. Returned artifacts are listed on the
run and can be downloaded with:
```sh
fluksio artifacts <run-id>
fluksio artifacts <run-id> train.weights -o weights.json
```
Because the address is the content's hash, a sweep whose fifty configs share
one preprocessed input stores it once, and a reference stays valid wherever
the store is reachable from. Artifacts a run produced are listed on it and
downloadable at `GET /api/v1/artifacts/{digest}`.
Use the exact message name shown by the list command. Emitted media frames
are transient; return a frame you need to keep as a run artifact.
## Stage caching
A run mostly does not redo what an earlier one already did. Before a node
executes it is fingerprinted with a sha256 over its source, its settings, the
ports it declares and the values it is about to read. If some earlier run of that
same fingerprint finished, what that one returned is restored into this run's
state and the node is skipped. It is recorded with the status `cached` and a
duration of zero, and its artifacts are listed on the new run as well, so they
stay downloadable from either.
Before executing a Python node, the engine checks for a successful previous
execution with matching source, settings, port declarations and input values.
On a hit, it restores the outputs and artifacts and records the node as
`cached`.
The settings go into the key raw, so a secret contributes its `{"$secret":
name}` reference and never its value. An artifact input counts as its content
digest: the same bytes under a different filename are the same input. Each node
on a run carries the `cache_key` it was looked up by.
For code-defined flows, sync also records digests of the project modules
reachable from each node. Editing those modules invalidates the corresponding
stage. Static import tracking does not follow computed import names or track
installed dependency versions; use `--no-cache` after changes it cannot see.
For a [code-defined flow](../getting-started/data-science.md), "its source" is
the generated shim, which imports the real function and does not change when
that function does. So the key carries one thing more: a digest of the project
modules that node's function reaches, worked out by `fluksio sync`, which is
the only side that imports your code and can see what it imports, and read
again from those files when the run starts.
Artifact inputs are compared by content digest. Secret settings contribute
their references rather than secret values. External state is not part of a
cache key, so disable caching on a node that reads changing data or performs
side effects:
Editing a helper three calls down from the node invalidates it, which is the
point: the alternative is a re-run answering with the previous code's numbers.
Editing something the node does not reach leaves the hit standing, which is
the other half: a notebook two directories away is not a reason to retrain.
The walk follows imports statically and stops at the standard library, at
anything installed, and at Fluksio itself; a module imported under a name the
code computes is not followed. An engine that cannot see the files keeps what
sync recorded instead of nothing, so a worker on another machine no longer
keys every run the same.
```python
@node(provides=Port("reading", "float"), cache=False)
def read_sensor():
return get_current_reading()
```
The run history *is* the cache; there is no second store. A node's returned
outputs are kept on its run record as canonical JSON, up to 256000 characters.
A node returning more than that is not cacheable that run. An entry
whose artifact bytes have since left the store is a miss, not an error.
Use `fluksio run ... --no-cache`, `fluksio sweep ... --no-cache`, or
`"no_cache": true` in an API submission to execute every node.
Only `python` nodes are cached, and by default all of them are. A built-in node
type or a connector node has side effects and no source to fingerprint, so
neither is ever a candidate. Turn it off for one node with
`@node(..., cache=False)` (the flow document carries it as `cache`, so the
canvas and the API can change it too) or for one run with
`fluksio run --no-cache`, `fluksio sweep --no-cache`, or `"no_cache": true` in
the submission body.
Only Python nodes are cached. Built-in and connector nodes are not. The cache
uses run history; outputs too large to store there are not cacheable, and
missing artifact bytes make an entry a miss.
A cached node replays no emissions, since those values were the story of an
execution that is not happening this time, so its series is not rewritten
either. The run it was restored from is recorded instead, and that is where the
curve is read back from: asking the reusing run for its metrics answers with
the same points, under its own flow's names. The one way to be left with a
result and no curve is for that earlier run to have been deleted. Deleting a
flow does not do that — its runs stay — so this only happens when the run
itself was deleted.
A cache hit does not replay emissions. Its metrics are read from the original
run, so comparisons still show the recorded curve. Deleting that original run
leaves cached results available but removes their curve.
## Objects that cannot be serialized
A live model, a `DataLoader`, a JAX-compiled function: none of these cross a
node boundary, and no framework flag will make them. There are exactly two
patterns, and they are both deliberate:
* **Keep them in one node.** Stages that must share live memory are one node.
Building the model and training it is one stage; the fact that Kedro would
make them two nodes is Kedro's problem, not a structure worth reproducing.
* **Cross at a checkpoint.** Save what matters as an artifact and rebuild from
it on the other side. That is the boundary that also survives the next node
running on a different machine.
Live models, data loaders and compiled functions cannot cross a JSON message
boundary. Keep stages that share live objects in one node, or save a checkpoint
artifact and rebuild the object in the next node.
## Running a node somewhere else
A node that needs a GPU declares the label of the machine that has one:
Attach a [remote worker](../code/workers.md) to the engine and declare the
worker's label as the node's `device`. `device_policy="require"` waits for
that label; `"prefer"` uses the local engine when none is attached.
```json
{"id": "train", "device": "gpu", "device_policy": "require", "timeout": 7200}
```
A worker on that machine dials **out** to the engine, because the engine
generally cannot reach it (different network, no inbound route) and because
nothing should expose Redis across hosts. Install it on the box, mint it a
token, and start it:
```bash
curl -X POST $FLUKSIO/workers/tokens -d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}' # once, as an admin
pip install fluksio-worker
fluksio-worker \
--url wss://api.${DOMAIN}/api/v1/workers/attach \
--labels gpu,cuda12 \
--python /opt/torch-venv/bin/python
```
`fluksio-worker` is its own distribution: the agent, the node runner, and
`websockets`. Nothing of the engine, so a GPU box does not install a database
driver to run a training step. Where pip is not an option, the two files still
work copied into one directory and run with `python agent.py …`; the engine
serves the runner at `GET /api/v1/workers/runtime`.
`--python` is the interpreter node code runs on, which is how the GPU box keeps
its CUDA wheels without the engine ever installing them. The node's source
travels with every call, so nothing has to be deployed there.
A few consequences worth knowing:
* `import fluksio` inside a node is the worker's own reporter — `emit`,
`save_artifact`, `load_artifact`, installed before the node's code runs, so
the installed `fluksio` package (if the box has one) never shadows it.
* A node bound to a device is **compiled on that machine**. A node importing
`torch` is correct on the GPU box and a missing module on the engine, so
checking it on the engine would fail a node that is fine.
* If nothing carrying the label is attached, the run stays `queued` and says
what it is waiting for. Submit first, switch the GPU box on later.
* Cancelling a run kills what it is executing, there or here, and leaves other
runs of the same node alone.
* If the worker disappears mid-call, the run fails in seconds with
`worker went away mid-call` rather than waiting out its timeout.
* `device_policy: "prefer"` runs locally when no such worker is attached;
`"require"` (the default) waits for one.
The selected machine needs your imported packages. Code-defined flows also
need your project installed or available there: generated import shims travel
with the call, but your package does not. Workers connect outward to the
engine; see the worker guide for tokens and resource requests.
## Durability
Submitting journals the run to a Redis stream of its own, separate from the
one the automations use: a burst of five hundred sweep runs must not stand
between a house and its heating. An engine that is down when a run is
submitted picks it up when it starts.
Queued runs are stored in the database and picked up when the engine restarts.
Running jobs refresh a lease. If the engine dies and the lease expires, those
runs become `abandoned` rather than being executed again automatically.
The row is what makes that true rather than the queue. A run is written before
the work item is added, so an engine reads its own history at startup and
wakes itself for anything still `queued` — which is what a run submitted
seconds before a restart is, and what an in-memory queue would otherwise have
lost with the process.
Retry a finished run or the unfinished results of a sweep:
From the moment a run is claimed, its database row is the record and the queue
is finished with it. Redelivering two hours of training because an
acknowledgement was late is not recovery; instead a running run refreshes a
lease, and one whose lease goes stale is marked `abandoned`, which is what a
run whose engine was killed mid-training becomes. Starting it over is a
decision rather than something that happens: [`fluksio
retry`](../code/cli.md#fluksio-retry) submits it again, and `--group` does
that for the runs of a sweep that did not finish.
```sh
fluksio retry <run-id>
fluksio retry --group <group-id>
```
A retry creates a new run with the previous inputs, seed and group, linked
to its parent. It uses the flow's current definition, and can reuse completed
cached stages. Cancel active runs before retrying them.
## Looking at what ran
**New run** is where one is started, and the **Runs** screen is the experiment
log: every run newest-first, filtered by flow, by status, or down to one
sweep. A sweep is worth filtering to, since the table then draws a column per
parameter that actually varied, which is what makes fifty runs of one flow
readable.
The **Runs** screen filters by flow, status or sweep. Each run shows its
parameters, per-node timings and logs, traceback, metrics, artifacts and
result.
Tick two or more and their curves go side by side. Shift-click to take a
range, or the header box to take everything on screen. That comparison is the
address, so a link to it is a link someone else can open.
Select two or more runs to compare curves and the parameters that differ.
Plot against step, elapsed seconds or another metric from the same runs.
Comparisons have shareable URLs.
The curves are drawn against the step by default. They can also be drawn
against elapsed seconds, which answers "which one got there sooner" and is
measured from each run's own first reading so that runs started hours apart
still lie on top of each other; or against another metric of the same runs (an
epoch, or samples seen) joined on the step the two share.
One run in full is params, the per-node record with its logs and traceback,
the artifacts it made, its metrics and its result. **Retry** submits it again
as a run of its own, keeping the flow, the inputs, the seed and the sweep it
belonged to.
A run outlives the flow it ran. Deleting a flow leaves its history standing —
being the record of what ran is the point of keeping one — so the Runs screen
still lists it, and opening it says **flow deleted** with Retry disabled,
because there is nothing left to run it against. Clearing that history is a
decision of its own: a run at a time on the screen, or
`DELETE /api/v1/runs?flow=<name>` for a whole flow's worth, which is what a
reseed wants.
Deleting a flow keeps its run history. You can still inspect those runs, but
cannot retry them until the flow exists again. Delete history separately,
after active runs finish or are cancelled.
### Taking it into a dataframe
An analysis wants a table rather than a screen, and there are two it usually
wants. `fluksio export metrics` is the long one, a row per run, metric and
step; `fluksio export runs` is the wide one, a row per run with the
inputs that varied as columns beside its final numbers:
Export metrics as a long table and results as a row per run:
```sh
fluksio export metrics --flow train -o curves.csv
fluksio export runs --flow train --status ok -o results.csv
```
Both include run IDs. Result rows also include input values, final numeric
outputs, status, duration and code identifiers. Nested values use dotted paths,
such as `metric.final_metrics.train_loss`.
Python returns the same rows directly:
```python
import pandas as pd
from fluksio.sdk.client import Client
client = Client()
curves = pd.DataFrame(client.export_metrics(flow="train"))
arms = pd.DataFrame(client.export_runs(flow="train", status="ok"))
curves = client.export_metrics(flow="train")
results = client.export_runs(flow="train", status="ok")
```
The run id is on every row of both, so a curve joins to the arm it came from
and to the run page it was recorded on, and the wide table carries the commit
and the code digest, so an exported file says what produced its numbers.
[The CLI](../code/cli.md#fluksio-export) writes the same rows as csv, jsonl or
parquet, which is where an export belongs: in the script beside the analysis.
A node's numbers usually arrive inside a record, and the wide table follows
them in: every number a result carries is a column of its own, named by its
path (`metric.final_metrics.train_loss`) and `metrics="final_metrics.train_loss"`
selects one. Inputs work the same way, and are compared leaf by leaf, so two
configurations differing in one field give that field as the axis.
Pass either list to `pandas.DataFrame`. The
[CLI reference](../code/cli.md#fluksio-export) covers selection filters,
JSON Lines and Parquet.
### Bringing in runs made elsewhere
Runs are made by running a flow, so a history recorded before the move (an
MLflow experiment, say) has to be imported instead. Each entry names the flow,
the id the run had where it ran, and what it ended with:
Import historical runs without executing them:
```python
Client().import_runs([
{
"flow": "train",
"external_id": "mlflow:study-1/3f2a…",
"params": {"lr": 0.01},
"seed": 1001,
"status": "ok",
"created_at": "2026-03-01T10:00:00Z",
"started_at": "2026-03-01T10:00:01Z",
"finished_at": "2026-03-01T10:40:12Z",
"result": {"final_metrics": {"train_loss": 0.08}},
"metrics": [{"name": "train.train_loss", "step": 0, "ts": 1772359201.0, "value": 2.3}],
"artifacts": [{"name": "weights", "filename": "model.txt",
"media_type": "text/plain", "content_base64": "…"}],
"labels": ["imported"],
"group_id": "mlflow:study-1",
},
])
from fluksio.sdk.client import Client
Client().import_runs([{
"flow": "train",
"external_id": "previous-study:run-42",
"params": {"lr": 0.05, "epochs": 10},
"status": "ok",
"created_at": "2026-03-01T10:00:00Z",
"result": {"score": 0.94},
}])
```
The parameters are completed with the flow's declared values and checked the
way a submit checks them. The run is stored with the cause `import` and no
commit or code digest, since this engine never saw the code it ran. Importing
the same `external_id` again adds nothing, and `fluksio export runs` carries
it as a column, so an exported row joins back to where it came from.
The flow must exist, and parameters are checked against its inputs. Repeated
imports of the same `external_id` reuse the existing record. Imported runs
have no code commit or digest because the engine did not execute them.
Entries may also carry metric points and base64-encoded artifacts; see the
API schema for their fields.
### A dashboard, read against runs
A run records values under the same names a dashboard binds to (a run of
`study` writes `study.loss`) so a dashboard is already a way of looking at
one. **Open in dashboard** from a run or a comparison opens any dashboard with
`?runs=a,b,c` on it, and the widgets resolve from those runs instead of from
the live engine: each chart draws a line per run, the tiles that show one
number show the first, and the controls go quiet because there is nothing left
to publish to.
Choose **Open in dashboard** from a run or comparison to view those results
with a dashboard's widgets. Charts draw one line per run, scalar widgets use
the first run, and controls are disabled.
Nothing about the dashboard is specific to runs. The page built to watch a
training run happen is the page that shows the finished ones.
If the flow has no dashboard, the same menu can create one from its streaming
ports and outputs. The API equivalent is
`POST /api/v1/dashboards/from-flow/{name}`. The generated dashboard is
published and can be edited like any other dashboard.
If a flow has no dashboard yet, the same menu offers to build one from what
the flow declares: a chart per streaming port, a tile per output:
```
POST /api/v1/dashboards/from-flow/study
```
It publishes `study_results` and is an ordinary dashboard afterwards; editing
it is how it stops being generic.
### A tile that always shows the last few
The other direction is a chart pinned to runs rather than a page opened
against them, which is what a panel over a bench wants. Set a chart's source
to **Runs**, name the metric, and pick either the latest N of a flow, one
sweep, or specific runs. It re-reads on its own and whenever a run finishes.
### When a run draws nothing
A node restored from the [stage cache](#stage-caching) has its curve read back
from the run that recorded it. Delete that run and the reusing run is left with
a result and an empty curve, and the chart says so rather than looking broken.
## What this costs, compared
The repository ships a benchmark that measures submitting a run against a
Kedro project doing the same nothing:
```
fluksio — submit accepted median 15.3 ms
submit -> result median 60.8 ms
kedro — kedro run median 1109.5 ms
```
The difference is not the orchestration; it is that Fluksio does not boot a
project per run. The engine is already up, and the workers already have the
node's code compiled. On a 510-run sweep, that gap is about nine minutes of
pure startup that never happens.
To keep a chart showing recent experiments, set its source to **Runs** and
select the latest runs of a flow, a sweep or specific run IDs.
## See also
- [Getting started: data science](../getting-started/data-science.md) — the same
material as a first setup
- [Writing node code](../code/nodes.md) — generators, `fluksio.emit`, artifacts
- [Remote workers](../code/workers.md) — attaching the machine with the GPU
- [The HTTP API](../code/api.md) — every endpoint used above
- [Getting started: data science](../getting-started/data-science.md)
- [Writing node code](../code/nodes.md)
- [Remote workers](../code/workers.md)
- [The HTTP API](../code/api.md)
+32 -51
View File
@@ -1,75 +1,56 @@
# Keeping state in a flow
Logic nodes are pure functions of their inputs. There is no `context`, no
`global`, no handle to a store. A node is called with the values of the
messages it declares and returns the values of the messages it provides.
Need a running total or a last-seen value? Make it a message. Function nodes
have no hidden `context` or global flow store.
That is deliberate: a node with hidden state cannot be run twice in parallel,
cannot be replayed, and cannot be moved to another worker. But plenty of real
automations need to remember something. This is how.
## Read and write the same message
## State is a message the node both reads and writes
A running total, a debounce timer, a last-seen reading: each is a value that
survives between runs. Give it a message name, declare it as both an input and
an output, and it is state:
Declare `reading` and `total` as inputs, and `total` as an output:
```python
def process(reading, total=0.0):
def process(reading, total):
return {"total": total + reading}
```
with `reading` and `total` as inputs and `total` as the output. Each run reads
the total the previous run wrote.
Add `total` to the flow's **Inputs** with a starting value of `0.0`. Each
execution reads the previous total and publishes the next one.
Two things make this work rather than loop forever:
- A node is never downstream of itself, so its own publication does not run
it again.
- A triggering input needs a starting value. Without one, validation reports
`self_loop_needs_initial`; a Python default alone does not supply the message.
- **A node never depends on itself.** The graph is built from message names, but
a node providing a message it also requires is not placed downstream of
itself. Publishing `total` does not re-run the node that wrote it.
- **The value must start somewhere.** The first run has nothing to read. Declare
the message as a flow input with a starting value; otherwise the flow reports
`self_loop_needs_initial` at edit time and refuses to publish.
## Feedback between two nodes
The port needs a default in the function signature (`total=0.0` above) so the
first call is well-formed even if the value is missing.
## Feeding a value back between two nodes
Sometimes the value comes from a different node, such as a controller reading back what
an estimator computed from its own last output. Written plainly that is a cycle,
and the validator rejects it, because a graph where A waits for B and B waits
for A can never start.
What is actually meant is that the fed-back value is *state*, not a trigger.
Say so, by marking the input non-triggering:
A controller may read an estimate that depends on its own previous output.
Ordinary triggering inputs would form a cycle. Mark the feedback input
**non-triggering** in the flow definition instead:
```json
{"name": "estimate", "port": "estimate", "dtype": "float", "trigger": false}
```
A non-triggering input:
It creates no dependency and does not wake the node. Whenever another input
runs the node, it reads the latest estimate. If no estimate exists yet, that
argument is omitted, so give it a Python default:
- creates no dependency, so it cannot form a cycle;
- never makes the node wait. If the message has no value yet, the port is
simply left out of the call, and the function's default applies;
- is read fresh from state whenever the node does run, for whatever reason.
```python
def process(reading, estimate=0.0):
return {"command": reading - estimate}
```
Use it for the back edge only. An input that should wake the node is an ordinary
input, and marking it non-triggering would mean the node never runs at all.
Use this on the feedback input; keep the inputs that should wake the node
triggering. Configure `trigger` through the API or flow JSON; the node panel
does not currently expose it.
## What still holds engine-side state
## Built-in state
Built-in nodes that are *about* time or change (rate limiting, filter-on-change,
delay, cron) keep what they need in the engine's own state backend, under keys
that never appear as messages. That is the engine's business, not a flow's: the
line is that node *code* you write never reaches for state, while node *types*
the engine ships may.
Storage and direct-I/O nodes are the other sanctioned exception, since talking
to a device or a database is stateful by nature.
Built-in nodes such as delay, rate limit and filter-on-change keep their own
timing or comparison state in the engine. Storage and device nodes also manage
the state needed for their connections. Your Function code still exchanges
state through declared messages.
## See also
- [Where a node's values come from](values.md) — ports, settings and flow inputs
- [Node types](../reference/node-types.md) — the built-ins that do keep engine-side state
- [Where a node's values come from](values.md)
- [Node types](../reference/node-types.md)
+26 -33
View File
@@ -1,67 +1,60 @@
# Where a node's values come from
A node function is called with one argument per thing it needs, and every one
of them is declared somewhere you can see. There are two kinds.
Function nodes receive values as named arguments. Each comes from a message
port, a node setting or a Python default.
## Ports: what the graph carries
A port binds to a message name. Whatever last published that message is the
value the node is called with, and the wiring follows from the name rather than
from a line somebody drew.
A port binds to a message name and reads its current value. Declare inputs
`reading` and `setpoint`, and output `heat`:
```python
def process(reading, setpoint):
return {"heat": reading < setpoint}
```
with `reading` and `setpoint` as inputs and `heat` as an output.
Matching message names connect nodes. A port can have a different name from
its message: bind port `reading` to `house.temperature`, for example.
## Settings: constants of this node
A setting is a value that belongs to this node's code rather than to the graph:
how many retries, which unit, how long to wait. It is typed into the node's
**Settings** section, stored with the flow, and arrives as an argument like a
port:
Use settings for values that stay fixed while the flow runs: units, retries,
thresholds. Add them in the Function node's **Settings** section; they are
stored with the flow and passed beside the ports:
```python
def process(reading, unit="C"):
return {"shown": reading if unit == "C" else reading * 1.8 + 32}
```
Because both arrive by name, a setting cannot share a name with a port. The
node reports it rather than picking one.
With a `unit` setting, its configured value is passed to the function. Without
one, the Python default applies. A setting cannot share a name with a port.
Settings are part of the flow document, so changing one is an edit that gets
published, not something that happens at runtime. **A value that should change
while the flow runs is a message, not a setting.**
There is no `global` and no `flow` context. A constant several nodes need is a
node that provides it: one place it is set, an ordinary message out, and every
consumer visibly downstream of it.
Changing a setting edits the flow draft, so publish it to apply the change.
For a value that should change at runtime, use a message. To share a constant
between nodes, publish it from one node and let the others consume it.
## Flow inputs: what arrives from outside
Some messages are not computed by any node in the flow, such as a dashboard control
writes them, the API publishes them, a batch run passes them in. Declare those
as the flow's **inputs**, with the value they start from:
Dashboard controls, the API and batch runs can supply messages without a
producer node. Declare them as **Inputs** in the flow settings:
```json
{"inputs": [{"spec": {"name": "setpoint", "dtype": "float"}, "initial": 21.0}]}
```
Without that, the node reading `setpoint` waits for something nothing provides,
and the canvas says so. With it, the flow starts from 21.0 and whatever writes
the message afterwards takes over.
The initial value seeds an unset live message. Later publications take over;
changing the starting value does not overwrite a value already in state.
Use **Publish this value now** in the input row to send it immediately.
The canvas draws each one as a labelled endpoint feeding the nodes that read
it, the same way it draws a dashboard tile or another flow, so a value never
appears from nowhere. They are edited in the flow's own panel.
The canvas draws flow inputs as labelled endpoints. Without a producer or
declared input, a consuming node reports `unconnected_input`.
A batch flow's inputs are also its run parameters, and its **outputs** name what
a run reports as its result; see [Runs](runs.md).
A batch flow's inputs are run parameters. Its **Result** section selects the
messages returned when the run finishes. See [Runs](runs.md).
## See also
- [Flows, nodes and messages](flows.md) — how the graph is built from the names
- [Keeping state in a flow](state.md) — a value a node reads *and* writes
- [Writing node code](../code/nodes.md) — the practical side
- [Flows, nodes and messages](flows.md)
- [Keeping state in a flow](state.md)
- [Writing node code](../code/nodes.md)
+115 -719
View File
@@ -1,800 +1,196 @@
# Getting started: data science
You have a training script. It works. What you do not have is any reliable
answer to "what was the learning rate on the run that got 94%?", and the
`results_final_v3_ACTUAL.csv` in your home directory is not helping.
Turn a Python experiment into a flow with recorded parameters, training curves
and files you can find again. Goodbye, `results_final_v3.csv`.
This page adds Fluksio to what you already have. It takes about five minutes,
installs one Python package, and does not ask you to restructure anything.
## Install
This example uses a toy training curve, so you can try the whole workflow
without a dataset or GPU. You need Python 3.12 or newer on Linux or macOS.
Install Git too if you want flow changes recorded in version history.
## 1. Start your instance
Create a project directory and install Fluksio in a virtual environment:
```sh
cd my-research
mkdir research-flow
cd research-flow
python3 -m venv .venv
. .venv/bin/activate
pip install fluksio
fluksio serve
```
That is the whole instance. No Docker, no database server, no ports to
open, and no login. The first run prints something like:
Fluksio creates `.fluksio/` for this project's flows, run history and artifacts.
It prints the initial admin password once and saves a token for local CLI
commands. Keep the password somewhere safe.
```text
Created the admin account admin@example.com
password: example-password
Shown once. Change it from the dashboard.
Fluksio 0.1.0 — data in /home/you/my-research/.fluksio
API http://127.0.0.1:8000/api/v1
Nodes /home/you/my-research/.venv/bin/python
your environment, adopted. Add packages with pip.
No portal. Pair this instance with:
fluksio enroll <code>
Signed in as admin@example.com
token in /home/you/my-research/.fluksio/client.json
```
Read the last two lines: you are already signed in. Signing in to your own
machine is a formality, since the password was printed by the same process that
would have checked it, so `serve` writes the token itself and every command
below just works. `fluksio login` is for an engine somewhere *else*.
**Write that password down** anyway. It is shown once, and it is what the
dashboard asks for.
### One instance per project
`.fluksio` sits beside your code, and is found the way `.git` is: from the
directory you are standing in, or any directory above it. So two repositories
on one machine are two engines: separate flows, separate run history,
separate token, and no chance of one experiment's graph turning up in the
other's.
```text
~/research/protein-fold/
.venv/ torch, fluksio
.fluksio/ its own database, flows, artifacts, token
myresearch/
~/research/climate-sim/
.venv/ jax, fluksio
.fluksio/ its own everything
climate/
```
It holds a database and a credential, so it ignores itself from within with a
`.gitignore` of `*`, the same thing `uv` writes into `.venv`. Nothing to add
to your project's own.
Give them different ports (`--port`) if you want two running at once.
!!! tip "One engine for the machine instead"
`fluksio serve --global` uses `~/.fluksio`, shared by every directory,
which is what you want for a personal server rather than a project. When
both exist, the banner says which one you are looking at and how to reach
the other. `--data-dir` still names any directory outright, which is worth
doing on a cluster where `$HOME` is a network filesystem: SQLite's
write-ahead log does not work on NFS, and `fluksio serve` warns you when
it notices.
!!! tip "Keep it running"
The engine is meant to be resident, which is what makes submitting a run
cost about 15 ms. Leave it in a `tmux` window, or write a small
`systemd --user` unit for it.
## Talking to it over HTTP
Everything the commands do is the HTTP API, and some of this page shows it
directly. For that, take the token `serve` already wrote:
At a terminal, `serve` opens a dashboard. Leave it running, then open a second
terminal in `research-flow` and activate the same environment:
```sh
export FLUKSIO=http://127.0.0.1:8000/api/v1
export TOKEN=$(jq -r .token .fluksio/client.json)
. .venv/bin/activate
```
While you are experimenting, the interactive schema at
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
For an engine on another machine, `fluksio login --url https://…` asks for a
password and stores the token the same way: in this project's `.fluksio`, or
with `--global` in `~/.fluksio`.
## Your packages are already there
If you installed Fluksio into the environment you work in (the venv that
already has torch or numpy in it) that is the environment your nodes run on.
Nothing to declare, nothing to install twice:
For the browser dashboard, sign up at [fluksio.com](https://fluksio.com), add
an instance, and enter the code it gives you:
```sh
python -m venv .venv && . .venv/bin/activate
pip install torch numpy pandas # what you were going to install anyway
pip install fluksio # and then this
fluksio serve
fluksio enroll <code>
```
`fluksio serve` says which interpreter it settled on:
Select the instance in the portal. The CLI also works without enrolling.
```text
Nodes /home/you/research/.venv/bin/python
your environment, adopted. Add packages with pip.
```
!!! tip "Use your existing research environment"
That venv is yours. Add a package the way you added the rest (`pip install
scikit-learn`) and `fluksio sync` (or a restart) retires the workers so they
pick it up. The Modules screen lists what is installed and stays read-only,
because the alternative would be Fluksio deciding what belongs in an
environment it did not make.
You can install Fluksio in the virtual environment that already holds your
research packages. Nodes use that environment by default. The startup
banner shows the interpreter; see [Packages](../code/nodes.md#packages)
for other setups.
!!! note "Your pins and ours share a site-packages"
## 2. Write a small flow
The cost of not having two environments: a package the engine depends on
is one you can now upgrade out from under it. In practice this is what
everybody does with every other tool in the venv, and the answer when it
bites is the same: pin it back, or keep Fluksio somewhere separate with
the venv of its own below.
### A venv of Fluksio's own
Sometimes you want the isolation instead: a shared instance, a container,
or an environment too precious to let a node's dependency near. Set
`NODE_VENV=managed` and Fluksio builds and owns one under the data directory:
```text
Nodes /home/you/.fluksio/user-venv/bin/python
a venv of its own; the Modules screen installs into it.
```
Then the Modules screen is how packages get in: a pip manifest, installed
with `uv pip sync` and versioned alongside your flows, so what a run imported
is recorded with what it ran:
```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"requirements\": $(jq -Rs . < requirements.txt)}"
```
Adding a package takes effect immediately; nothing restarts. The Docker image
sets `NODE_VENV=managed` for itself, because the venv in it holds the app and
nothing of yours, so a container is always this case.
!!! tip "A GPU box works the same way"
`pip install fluksio-worker` into the environment the training code runs
in, and node code runs on it: `--python` defaults to the interpreter the
agent was started with. It reads its token from `FLUKSIO_WORKER_TOKEN`.
```sh
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach --labels gpu
```
Then give the node `device="gpu"`; see [Remote workers](../code/workers.md).
## Say which functions are nodes
A **flow** is a graph of nodes. A **batch flow** is one that runs on demand
from parameters to a result, which is what an experiment is. A node is one of
your own functions. It stays in your repository, imported by its siblings as
it always was.
Say your project looks roughly like this:
Save this as `study.py` in `research-flow`:
```python
# myresearch/train.py
def fit(dataset, lr, epochs=25):
model = build_model()
for epoch in range(epochs):
loss = step(model, lr)
print(f"epoch {epoch}: {loss}") # ← this is what you are losing
torch.save(model.state_dict(), "weights.pt")
return loss
```
import json
import math
from pathlib import Path
Two changes turn it into a node, and neither of them moves it:
```python
# myresearch/train.py
import fluksio
from fluksio import Port, node
from fluksio import Flow, Port, node
@node(
requires=["dataset", Port("lr", "float")],
requires=[Port("lr", "float"), Port("epochs", "int")],
provides=[
Port("loss", "float", stream=True),
Port("weights", "artifact"),
Port("final_loss", "float"),
],
timeout=600,
)
def fit(dataset, lr, epochs=25):
"""Fit the model. A generator, so numbers escape while it is running."""
model = build_model(fluksio.load_artifact(dataset))
def fit(lr, epochs):
loss = 1.0
for epoch in range(epochs):
loss = step(model, lr)
yield {"loss": loss} # ← published now, on the loss port
torch.save(model.state_dict(), "weights.pt")
loss = math.exp(-lr * (epoch + 1))
yield {"loss": loss}
checkpoint = json.dumps({"loss": loss}).encode()
return {
"weights": fluksio.save_artifact("weights.pt"),
"weights": fluksio.save_artifact(
checkpoint, "weights.json", "application/json"
),
"final_loss": loss,
}
```
`yield` replaces `print`. Each one publishes on the node's `loss` port the
instant it happens, and the run keeps every value as a series, which is why
Fluksio has no `log_metric()` call. A metric that leaves through a port is a
message like any other: a chart can bind to it, a downstream node can consume
it, and it shows up on the canvas. A metric that escapes through a logging
function is invisible to all three.
`fluksio.save_artifact` handles the things too big to be messages: a
checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a
small reference. Nothing changes about how you write the file.
@node(requires=["weights"], provides=Port("score", "float"))
def evaluate(weights):
checkpoint = json.loads(Path(fluksio.load_artifact(weights)).read_text())
return 1.0 - checkpoint["loss"]
The decorator returns the function untouched, so `fit(dataset, 0.05)` in a
test, a notebook or a debugger is exactly what it was before. Outside a run
`fluksio.emit` does nothing rather than failing, so a progress report never
stops your own code from running.
### What is declared, and what is not
- **Ports are declared, never inferred.** `requires` names the messages the
node reads; each arrives as the parameter of the same name. A bare string is
the shorthand: `"dataset"` takes its type from whoever provides it, so a
message's type is written down exactly once.
- **Parameters with a default that are not ports become settings.** `epochs`
above is stored with the flow and tunable on the canvas without touching
this file.
- **The declaration is checked against the function.** A port with no matching
parameter, or a parameter that is neither port nor setting, is an error when
the module is imported, not when the node is first called.
- **The decorator hands the function back untouched.** `fit(dataset, lr=0.05)`
is an ordinary call, and the module is an ordinary script, so a
`if __name__ == "__main__":` check beside the nodes runs. Syncing one
defined in a script *run as* `__main__` is what cannot work, since the
generated body would have no name to import it by.
!!! note "Where a `yield` cannot reach"
If the number comes from inside somebody else's callback (Keras, Lightning,
HuggingFace `Trainer`) `fluksio.emit(loss=...)` writes the same port the
same way:
```python
model.fit(callbacks=[LambdaCallback(
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
```
### Declaring what a node needs
A training node is not like the rest of a flow. NumPy, JAX and PyTorch each
size their thread pool to every core they can see, so a few of them at once
oversubscribe the machine badly enough to starve the engine itself: the API
stops answering and every client waiting on a run dies with it. On a GPU the
same shape deadlocks: two processes each preallocating most of the card sit at
zero utilisation with nothing failing and nothing to read.
Say what one execution takes, and the engine holds it:
```python
@node(..., resources={"cpus": 2, "env": {"XLA_FLAGS": "--xla_cpu_multi_thread_eigen=false"}})
def fit(dataset, lr, epochs=25):
...
@node(..., resources={"gpus": 1, "env": {"XLA_PYTHON_CLIENT_MEM_FRACTION": "0.9"}})
def finetune(checkpoint):
...
```
Two things follow. The node waits its turn rather than starting alongside more
of itself than the machine has room for, the same waiting it already does for
a worker. And the worker it runs in is *started* with thread limits matching
what it was given, because that is the only moment a numerical library reads
them; a GPU node is told which card is its through `CUDA_VISIBLE_DEVICES`, and
nothing else is given that card while it runs.
`env` is for the tuning the engine must not invent. `XLA_FLAGS` is one composed
string whose contents depend on the version you have installed, so writing it
for you would silently replace whatever you had put there.
The engine counts NVIDIA's device nodes to find its cards, which asks no
vendor tool. A card those do not cover has to be declared (`fluksio serve
--gpus 1`, or `FLOW_GPUS`); until it is, a node asking for one is quietly
given zero and runs alongside every other, and the log says so the first time
it happens.
Workers are kept warm on purpose, so a library that takes most of the card at
import would hold it after the run finished. The workers that ran on a card
are retired when the run ends, which gives the memory back; what it costs is
the next GPU run paying for its imports again.
**The declaration is what buys that**, not touching the card. A node that
imports jax without `resources={"gpus": 1}` runs on the shared pool, is never
given `CUDA_VISIBLE_DEVICES`, and leaves a warm worker holding whatever it
preallocated. Declaring the card is what makes it a worker the engine knows
to retire, and what stops two such nodes running at once in the first place.
Host memory is held the same way, and there the declaration is not needed. A
worker that fitted a model keeps what it allocated, and on a laptop a few warm
ones are the machine. `fluksio serve --worker-max-rss 1024` retires any worker
holding more than that when its node returns, whichever pool it is in. A
node's `ram` is counted against this machine's memory, so declaring it is what
stops three large fits from starting at once.
Declaring nothing is the default and is right for most nodes: a poll, a
threshold, a message on its way somewhere. Those share the engine's worker pool
and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops
several of them at once from each sizing to the whole box.
### Ask for a size by name
Cores and gigabytes are a property of the machines you have, and those change.
A node that names a **flavor** keeps meaning something afterwards:
```python
@node(..., resources="gpu-small")
def finetune(checkpoint):
...
@node(..., resources={"flavor": "medium", "duration_s": "2h"})
def fit(dataset, lr, epochs=25):
...
```
The flavor is read every time the node is built, so editing it changes what the
next run gets. `fluksio flavors` lists them; the Workers screen edits them.
`ram` takes `"2G"` and `duration_s` takes `"30m"`, and a flavor already says how
much, so pass one or the numbers, not both. `duration_s` is a statement about the
node for whoever is planning around it, not a limit; the limit is `timeout`.
### Where it runs
This is one decision, not two. Of every machine attached (this engine and each
worker) the node goes to one that can grant what it asked for and has it free.
So a node asking for a card finds the box that has one, without naming it:
```python
@node(..., resources={"gpus": 1}) # wherever there is a card
@node(..., device="gpu", resources="gpu-small") # that box, and this much of it
```
If nothing attached can ever grant the ask, it is cut down to what is here and
runs anyway, since a flow written on a cluster still has to work on a laptop. If
something *could* but is busy, the node waits and says so.
Ask what is free, and what is waiting for it, at
`GET /api/v1/workers/resources`, on the Workers screen, or in `fluksio status`.
!!! note "Accounted, not enforced"
Nothing stops a node that ignores its declaration; the numbers are
bookkeeping plus the environment its libraries read. Real enforcement
(cgroups, rlimits) is a next step, not this one.
## Say which nodes make a flow
Membership is a list, not a directory layout: the functions can live wherever
they already do.
```python
# myresearch/pipeline.py
from fluksio import Flow, Port, use
from myresearch.data import augment, prepare
from myresearch.evaluate import evaluate
from myresearch.train import fit
train = Flow(
"train",
nodes=[prepare, fit, evaluate],
inputs=[Port("lr", "float", initial=0.01)],
outputs=["score", "final_loss"],
nodes=[fit, evaluate],
inputs=[Port("lr", "float", initial=0.05), Port("epochs", "int", initial=10)],
outputs=["score", "final_loss", "weights"],
)
```
`inputs` are the run's parameters, with the value a run gets when it names
none; `outputs` are what a run reports as its result.
`@node` declares inputs and outputs. Matching message names connect the nodes:
`fit` produces `weights`, and `evaluate` reads it. `Flow` lists the nodes,
run parameters and final results.
Nodes are connected because one **provides** a message another **requires**,
never because one imported the other. Importing `fit` into a second flow means
"the same code", not "wired to it".
Each `yield` records a loss value. The final `return` sends the checkpoint to
the next node. Artifact ports carry a small reference; the file stays in the
artifact store.
Which is how the same function serves two flows, rewired and reconfigured for
each:
```python
finetune = Flow(
"finetune",
nodes=[
prepare,
augment,
use(fit, wire={"dataset": "augmented"}, epochs=3),
evaluate,
],
inputs=[Port("lr", "float", initial=1e-4)],
outputs=["score"],
)
```
`use(fn, ...)` is one use of a node in one flow: `wire` binds a port to a
different message, keyword arguments override settings, and `id=` tells two
uses of one function apart. `train` above is unaffected by any of it.
A test can check the declarations without an engine. Importing the module
registers its flows in `fluksio.FLOWS`, and `fluksio.build(flow)` returns the
document `sync` would upload, refusing what the engine would refuse.
## Sync it
## 3. Sync and run
```sh
fluksio sync myresearch
fluksio sync study
fluksio run train --lr 0.1 --epochs 20 --follow
```
That imports the package, checks every declaration, and uploads each flow with
a generated body per node:
`sync` uploads the flow and generates node code that imports your functions.
Keep the project files available to the engine. Later `run` commands sync
edits before submitting; `--follow` waits and prints metrics as they arrive.
This run should finish with `final_loss` around `0.1353` and `score` around
`0.8647`.
```python
# generated by fluksio sync from myresearch/train.py — edit that file instead
import sys
_REPO = '/home/you/my-research'
if _REPO not in sys.path:
sys.path.insert(0, _REPO)
from myresearch.train import fit
def process(dataset, lr, **settings):
return (yield from fit(dataset=dataset, lr=lr, **settings))
```
So the flow store still holds a complete, runnable, git-versioned definition;
the body simply happens to import rather than duplicate. Your code stays in
your repository, under your version control.
The upload is stamped with that repository's commit, and every run records it
alongside the store's own. `--dry-run` prints all of this and uploads nothing;
`--no-publish` leaves it as a draft.
!!! warning "Sync after you edit"
The engine's workers are long-lived and hold your imported modules in
memory, so an edit to `myresearch` is invisible until they are retired,
which is what every `fluksio sync` does, including one that has nothing to
upload. If you are editing many times an hour, attach your own interpreter
as a worker instead: it starts a process per call and therefore reads your
code fresh every run.
```sh
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
--labels local --python "$(which python)"
```
Then give the node `device="local"`.
!!! note "The repository path is a deployment detail"
The generated body puts your repository on `sys.path` by absolute path,
which is right for the machine you synced from and meaningless in a
container. For anything else, install the package where the node runs:
`-e /home/you/my-research` in the module manifest, or
`myresearch @ git+ssh://…@a1b2c3d`, which travels where a path does not.
### Which hardware a node runs on
`device` picks the worker, the same way an executor does elsewhere:
```python
@node(requires=["dataset"], provides=[...], device="gpu", device_policy="prefer")
def fit(dataset, lr, epochs=25):
...
```
`device_policy="require"` (the default) waits for a worker carrying that label;
`"prefer"` runs it locally when none is attached, which is what you want while
the GPU box is not switched on. See [Remote workers](../code/workers.md).
### What the canvas does with a synced flow
Its node bodies are generated, so the editor shows them read-only and says
where the real code is. Everything else behaves as usual, but ports and
settings changed there are overwritten by the next sync, which is the point of
the repository being the source of truth. A node whose code you edit on the
canvas makes the next sync stop and say so rather than discarding your edit;
`--force` overrides that.
## Run it
In the portal, open **Flows → train** to see the graph, or **Runs** to inspect
the parameters, metrics and results. From the CLI:
```sh
fluksio run train --lr 0.003 --seed 7
fluksio runs --flow train
fluksio artifacts <run-id>
fluksio artifacts <run-id> train.weights -o weights.json
```
`run` syncs first, so after an edit the command is just `fluksio run`, with no
step to forget. `--no-sync` skips it for a tight loop where nothing
changed, since syncing retires the workers and the next call pays its imports
again.
Use the ID printed by `run`. Artifact names are flow-qualified; the list
command shows the exact names to download.
`--lr` is typed by the flow's own inputs, so `0.003` arrives as a float. A
parameter you did not declare, or one of the wrong type, is refused before
anything executes. It answers immediately with a queued run, since training is
measured in hours, so nothing waits for it unless you pass `--wait`, or
`--follow`, which waits and prints the numbers as the run reports them.
## 4. Try a few learning rates
No engine has to be up for any of this: `fluksio run --local` boots the
engine inside the command instead, on the same `.fluksio`: the same database,
artifacts and history a served one would use. It costs a few seconds of
startup per invocation against the ~15 ms of submitting to an engine that is
already up, so it is for the run you want now rather than the loop you are
iterating in.
```sh
fluksio sweep train --param lr=0.01,0.05,0.1 --wait
fluksio export metrics --flow train -o curves.csv
fluksio export runs --flow train --status ok -o results.csv
```
A second run of a flow you did not change mostly does not execute. Each of
your nodes is fingerprinted by its settings, the ports it declares, the values
it reads, and a digest of your repository's Python files, so editing anything
your node calls into invalidates it, not only the decorated function itself.
One an earlier run already answered is restored from that run rather than run
again, reported as `cached`, so a flow with nothing left to do finishes as
`(3/3 cached)`. Change `--lr` and only the nodes downstream of it run. The
digest is blunt on purpose: an edit anywhere in the repository re-runs
everything. `--no-cache` turns caching off for one run, `@node(...,
cache=False)` for one node; the caveat and the details are in
[Stage caching](../concepts/runs.md#stage-caching).
The sweep creates three runs in one group. Select them on **Runs** to compare
their curves. The exports give you a row per metric point and a row per run,
ready for your analysis.
From Python, the flow you declared is also the handle to its runs:
Python can read the same tables:
```python
from myresearch.pipeline import train
from fluksio.sdk.client import Client
run = train.submit(lr=0.003, seed=7).wait()
print(run.status, run.result)
print(run.metrics("train.loss")[-1]) # the whole series is kept
open("weights.pt", "wb").write(run.download("train.weights"))
client = Client()
results = client.export_runs(flow="train", status="ok")
curves = client.export_metrics(flow="train", name="train.loss")
```
Or over HTTP, which is what both of those are:
Pass either list to `pandas.DataFrame` if you use pandas.
```sh
curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"params": {"lr": 0.003}, "seed": 7}'
## Bring your own experiment
curl -s $FLUKSIO/runs/<id> -H "Authorization: Bearer $TOKEN" | jq
curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $TOKEN" | jq
```
Replace the toy calculation in `fit` with your training code. Keep ordinary
Python helpers separate from artifact storage so you can still test them
without an engine: `save_artifact` and `load_artifact` work only inside nodes.
The public repository's `examples/myresearch/` package shows this pattern
across several files.
The run carries its parameters, a digest of them, the seed, its result, how
long each node took, what it logged, every artifact it produced, and both
commits: the flow store's and your repository's. That is the answer to "what
was the learning rate on the run that got 94%?". When a run failed,
`run.failures` is the node that did it, with its traceback and its logs.
- Defaulted function arguments that are not input ports become node settings.
Use flow inputs for values you want to vary between runs.
- Use `fluksio.emit(loss=value)` when metrics arrive through a framework
callback instead of a loop you can `yield` from.
- Runs can reuse unchanged stages. Use `--no-cache` when you want every node
to execute; see [Stage caching](../concepts/runs.md#stage-caching).
- `--seed` records a seed and fills an input named `seed` when declared. Your
code still needs to pass that value to its random number generator.
Beside your commit is `code_digest`, and `fluksio runs` prints the pair as
`a1b2c3d-dirty+9f0e1a2`. The commit alone cannot identify what ran: your node
bodies are imports, so the engine executes whatever is on disk when the worker
starts, and an uncommitted tree stamps `-dirty` for every run it ever produces.
The digest is over the modules this flow's nodes actually reach, not the
whole repository, so it moves when the code behind a number moves and stays
put when a notebook beside it changes. It is read at the moment the run
starts, so in a sweep whose runs queue for hours each one records the code
that actually executed it rather than the code that was there when you
submitted. It is what makes the column worth joining an
[exported table](../concepts/runs.md#taking-it-into-a-dataframe) on.
### Declaring what a node needs
### When the engine is busy
A driver script outlives the engine being slow, because a sweep is hours long
and a blip is seconds:
- **Reads are repeated.** Every GET, and the writes that mean the same thing
twice, are tried up to three more times on a transport error or a `502`,
`503` or `504`, waiting 1 s, 2 s, then 4 s. `Client(retries=0)` turns that
off.
- **Submitting is safe to repeat.** Each `submit()` mints a key the engine
stores with the run, so an attempt that timed out on the way back is
answered with the run it already made rather than starting a second one. A
sweep keys every entry, so retrying a half-created one creates only what is
missing.
- **Waiting tolerates a gap.** `wait()` and `--follow` survive five failed
polls in a row before giving up. An answer is not a gap: a `404` for a run
that is gone stops immediately.
- **Timeouts are split** — 5 s to connect, 120 s to read, so a wrong URL fails
at once while a busy engine is given longer than the slowest thing it does
on purpose. `Client(timeout=…)` takes a number or an `httpx.Timeout`.
A command that still cannot reach the engine says
`fluksio: engine not answering (…)` and names the run it started, which is
running on the engine regardless, and `fluksio runs` finds it again.
## Sweep it
A grid search and an ensemble are the same submission, run in parallel:
```sh
fluksio sweep train --param lr=0.001,0.003,0.01 --wait
```
Every combination of the lists you give, so a second `--param` is a grid
rather than a second sweep. Where the set you want is not a product, build the
list yourself and post it:
```sh
curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"runs": [
{"params": {"lr": 0.001}, "seed": 1},
{"params": {"lr": 0.003}, "seed": 1},
{"params": {"lr": 0.010}, "seed": 1}
]}'
```
They share a `group_id`, so the sweep is `GET /runs?group=…`. Running them
concurrently is safe because each run gets a state backend of its own: two
runs of one flow cannot overwrite each other's values.
Compare the curves in one call:
```sh
curl -s "$FLUKSIO/runs/series/compare?ids=$A,$B,$C&metric=train.loss" \
-H "Authorization: Bearer $TOKEN"
```
which answers in exactly the shape a chart widget draws.
## Take it into a dataframe
The analysis itself wants a table, and `export` writes the two an analysis
asks for: the curves long, and one row per run with the parameters that
varied beside its final numbers:
```sh
fluksio export metrics --flow train --name train.loss -o curves.csv
fluksio export runs --flow train --status ok -o arms.csv
```
Both carry the run id on every row, and `arms.csv` carries the commit and the
code digest beside it, so an exported file still says what produced its
numbers. Numbers inside a record are columns of their own
(`metric.final_metrics.train_loss`) and `--metrics` and `--params` take those
dotted paths to narrow the table. Metric names are flow-qualified, so
`--list` prints every one a selection recorded. `Client.export_metrics()` and
`Client.export_runs()` answer the same rows to a notebook, ready for
`pandas.DataFrame`.
## Looking at them in the portal
None of the above needs a browser, but there is one. **Runs** is the table of
everything that has run: filter it to a flow, to a status, or to one sweep,
where it grows a column per parameter that actually varied, and tick two runs
to put their curves side by side.
A run records its metrics under the same names a dashboard binds to, so a
dashboard is another way of reading one. **Open in dashboard** carries the
runs you picked along in the URL, and every widget then answers for those runs
instead of for the live engine. If the flow has no dashboard yet, the same
menu builds one from the ports the flow declares.
See [Runs → Looking at what ran](../concepts/runs.md#looking-at-what-ran).
## What sync does, at the API level
Nothing here is privileged: `sync` is a client, and a flow is a document you
can PUT yourself. The whole of it, for the one-node case:
For heavy computation, declare resources on the node:
```python
"""Create the `train` flow by hand. What `fluksio sync` automates."""
import httpx
API = "http://127.0.0.1:8000/api/v1"
api = httpx.Client(base_url=API, timeout=60)
token = api.post(
"/login/access-token",
data={"username": "admin@example.com", "password": "example-password"},
).json()["access_token"]
api.headers["Authorization"] = f"Bearer {token}"
api.put("/flows/train", json={
"name": "train",
"title": "Model training",
# Batch: nothing is activated, nothing fires until a run asks.
"mode": "batch",
"inputs": [
{"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01},
{"spec": {"name": "epochs", "dtype": "int"}, "initial": 50},
],
"outputs": ["final_loss", "weights"],
"nodes": [{
"id": "train",
"type": "python",
"title": "Fit the model",
# An *idle* timeout once the node streams: this is how long it may go
# quiet, not how long it may run.
"timeout": 600,
"requires": [
{"name": "lr", "dtype": "float"},
{"name": "epochs", "dtype": "int"},
],
"provides": [
# `stream` says this port publishes repeatedly during one execution.
{"name": "loss", "dtype": "float", "stream": True},
{"name": "final_loss", "dtype": "float"},
{"name": "weights", "dtype": "artifact"},
],
}],
}).raise_for_status()
api.put("/flows/train/nodes/train/source",
json={"code": open("train_node.py").read()}).raise_for_status()
version = api.get("/flows/train").json()["definition"]["version"]
api.post("/flows/train/publish", json={"version": version}).raise_for_status()
@node(requires=["dataset"], provides=Port("score", "float"),
resources={"cpus": 2, "ram": "2G"})
def score_dataset(dataset):
return {"score": calculate_score(dataset)}
```
Two things worth noticing, because the decorators only move where they are
said. Ports are declared, not inferred: `process(lr, epochs)` gets its
arguments from the ports of the same name, and the types are checked on every
value. And saving writes a *draft*; `publish` is what the engine picks up.
That separation is what lets you edit a flow that is running.
The engine accounts for these resources and sets worker thread limits. GPU
nodes can request `resources={"gpus": 1}`; check that the Workers screen
reports your GPU, or set `fluksio serve --gpus 1` when automatic detection
does not cover it. Resource accounting does not impose a hard memory limit.
A flow uploaded this way carries no `origin`, which is what tells the canvas,
and the next `fluksio sync`, that it was not generated. See
[The API](../code/api.md).
To run on another machine, attach a [remote worker](../code/workers.md) and
install your project and its dependencies there.
## Small scripts you are just playing with
## Next steps
The same machinery, minus the ceremony. If what you want is "keep a record of
every version of this thing I keep tweaking", note that **your flows are
already a git repository**:
```sh
cd .fluksio/flows
git log --oneline
```
Every save is a commit: the node source, the parameters, the graph. A run
records the commit it ran at, so `git show` on that hash is literally the code
that produced the number. You get the versioning without adopting anything.
For quick iteration, keep the flow small (one node is fine), keep the engine
running, and submit from wherever you are working:
```python
from myresearch.pipeline import train
for lr in (0.001, 0.003, 0.01):
train.submit(lr=lr)
```
A submit is around 15 ms, so calling that in a loop is a reasonable thing to do.
## Get a browser onto it
The pip install gives you the engine and the API, not a web interface, since a
machine with no inbound route cannot serve one usefully. To see the
canvas, the run history and live loss curves, pair the instance with a
portal, which serves the dashboard from its side:
```sh
fluksio enroll <claim-code> --portal https://hub.fluksio.com
```
Get the claim code from the portal under **Instances → Add instance**.
Nothing needs to be exposed: your machine dials out and holds the connection
open. See [Accounts and the portal](../interface/portal.md).
If you would rather stay entirely offline, run the dashboard SPA yourself from
the app's Docker image; see the [facility path](facility-automation.md), which
is the same stack.
## Where to go next
- [Runs: pipelines that finish](../concepts/runs.md) — the full picture:
artifacts, sweeps, durability, what happens when your engine dies mid-training
- [Writing node code](../code/nodes.md) — generators, settings, what a node may
and may not do
- [The command line](../code/cli.md) — `login`, `sync`, `run`, `runs`, `sweep`
in full
- [Remote workers](../code/workers.md) — send the training node to the GPU box
and keep the rest on your laptop
- [The flow editor](../interface/flow-editor.md) — once you have a portal, this
is where the graph gets easier to change than the script did
- [Writing node code](../code/nodes.md) — ports, streaming, artifacts and packages.
- [Runs](../concepts/runs.md) — caching, comparisons and retries.
- [CLI reference](../code/cli.md) — command options and instance settings.
+114 -275
View File
@@ -1,325 +1,164 @@
# Getting started: facility automation
You have a box in a cupboard, a handful of sensors that already publish
somewhere, and an ambition to make the house do something about them. This
page brings up a Fluksio instance you can leave running for years, then wires
the first sensor through to a dashboard.
Build a small temperature flow, then connect a real sensor when you're ready.
No hardware required for the first lap.
Budget an afternoon. Most of it is the browser, which is the point: you will be
in this interface a lot, so it is worth learning it properly.
## Start the Docker stack
## What you are standing up
```text
sensors ──MQTT──▶ ┌──────────────┐ ──▶ InfluxDB (history)
│ Fluksio │
HTTP / webhooks ──▶│ flow engine │ ──▶ dashboards (what you look at)
│ │
schedules ────────▶└──────────────┘ ──▶ MQTT / HTTP (what you control)
│
└── workers on other boxes (optional)
```
One engine holds every flow. Nothing here is a plugin you install separately:
the broker client, the time-series writer, the dashboards and the alerting are
all part of the same process, editing the same graph.
## Prerequisites
- Docker and Compose v2 on the host
- The address you will type into a browser. On a home network that is the
server's own address, `192.168.1.50`, and nothing else is needed. A
hostname is optional, and covered below
- Optionally: an MQTT broker and an InfluxDB you already run. If not, the stack
can start both for you
## Set it up
Clone the app repository and try it on the machine you are sitting at:
You need Git, Docker with Compose v2, `make`, and `openssl`. Port 80 must be
free. Clone the public app repository and run its commands from the repo root:
```sh
git clone https://git.stroblme.de/Fluksio/app.git ~/fluksio
cd ~/fluksio
make init # once: .env with generated secrets, `proxy` network, prints the admin login
make dev-local # full stack at http://app.localhost (API at http://api.localhost)
make down # stop it
git clone https://git.stroblme.de/Fluksio/app.git fluksio
cd fluksio
make init
make dev-local
```
`make dev-local` needs port 80 free and only answers on `localhost`; the
sections below make it reachable from other machines. `make init` is safe to
run again: it only fills secrets still set to `changethis`, and `.env`
otherwise stays as you left it.
`make init` creates `.env`, generates secrets, creates the `proxy` network,
and prints your admin login. It is safe to run again: existing secrets stay
as they are.
For an instance you leave running, edit `.env` first (`FIRST_SUPERUSER`,
`ENVIRONMENT=production`). It is the whole configuration. The four settings that matter on day one:
Give the containers a moment to start on the first build, then open
[http://app.localhost](http://app.localhost). Sign in with the admin login:
adding Python code requires a superuser account. The API is at
[http://api.localhost](http://api.localhost), and the docs are at
[http://docs.localhost](http://docs.localhost).
| Setting | What it does |
|---|---|
| `DOMAIN` | the hostname everything is served under, if you use one. Left alone it only names the issuer that agents authenticate against |
| `FIRST_SUPERUSER` | the account you sign in with |
| `FIRST_SUPERUSER_PASSWORD` | generated by `make init`; it prints it, and it stays in `.env` |
| `ENVIRONMENT` | `production` closes the interactive API schema; `local` leaves it open |
Everything the instance owns (the database, your flows, secrets, artifacts,
the packages your node code imports) is on one Docker volume.
Backing that volume up is backing up the instance.
??? note "Even smaller: no Docker at all"
`pip install fluksio && fluksio serve` gives you the same engine with no
containers, keeping its data in `~/.fluksio`. What it does not give you is
the web interface, which the SPA container serves, so you would drive it
from the API, or [pair it with a portal](../interface/portal.md) that
serves the dashboard for you. Good for a Raspberry Pi that only runs flows;
less good as your main instance.
## Reach it from another machine
Start it one of two ways, depending on whether you have a hostname to point at
it. Neither is more supported than the other.
### On your own network, by address
The usual case: the box is in a cupboard at `192.168.1.50` and you open it from
a laptop. Layer `docker/compose.lan.yml` on, which publishes the interface on a
port and serves the API from that same origin:
For a Docker instance reached from another machine on your network, use:
```sh
APP_PORT=8080 docker compose -p fluksio-app --env-file .env \
-f docker/compose.yml -f docker/compose.lan.yml up -d --build
make dev-lan
```
Open `http://192.168.1.50:8080`. That one address is the whole application:
the interface, the API under `/api`, and the websocket that puts live values on
the canvas.
Open `http://<host-ip>:8080`. Choose another port with
`make dev-lan APP_PORT=8081`. This is a development stack; see
[running it long term](#run-it-long-term) before using it for a deployment.
Nothing in that URL is configured anywhere. The interface is built to address
the API relative to whatever origin served the page, so the same containers
answer to the machine's address, to a hostname you add later, and through an
ssh tunnel, without being rebuilt. It is also why no CORS list needs
maintaining: the browser is talking to one origin.
!!! danger "This is your network, not the internet"
A published port is plain HTTP with no certificate. Between your own
machines that is fine, and it is not something to forward from a router. To
reach the house from outside, put a reverse proxy with TLS in front of it,
or [pair it with a portal](../interface/portal.md), which needs no inbound
route at all.
### With a hostname, behind a reverse proxy
If you have DNS, or want certificates, set `DOMAIN` and run `make up`
instead. The interface is then served on `app.${DOMAIN}` and the API on
`api.${DOMAIN}`, and nothing is published on a host port.
The stack emits Traefik labels and ships a Traefik you can bring up alongside
it (`docker/compose.traefik.yml`). With Nginx Proxy Manager or Caddy instead,
attach it to the `proxy` network and forward `app.${DOMAIN}` →
`fluksio-app:80` and `api.${DOMAIN}` → `fluksio-api:8000`.
!!! tip "Developing on the same machine"
`make dev-lan` is `make dev-local` plus the published port: the app on
`http://<host-ip>:8080`, `APP_PORT` to move it. `make dev-local` recreates
the stack without that port, so re-run `make dev-lan` after one.
## Sign in
Open whichever address you chose (`http://app.localhost` for `make dev-local`)
and sign in with the login `make init` printed. You should be looking at Home:
an empty brain graph, a health summary, and a flow list with nothing in it.
When you are done, stop the stack with `make down`. Your data volumes stay
in place for the next start.
## Your first flow
Go to **Flows → New flow** and call it `house`. You land on the canvas.
Open **Flows**, create a flow named `house`, and use **Add node** to add an
**Inject** node. In its panel:
A flow is a set of nodes that talk to each other through **named messages**.
You do not draw wires: a node says which messages it needs and which it
produces, and the canvas draws the graph that follows. Renaming is safe, fan-in
is free, and two flows share a value by naming it.
- Under **Provides**, add `living_temperature` with type `float`.
- Under **Settings**, set the `living_temperature` value to `18.0` and
**Interval** to `5` seconds.
### Read a sensor
This gives you a pretend sensor. Add a **Function** node with:
Press **Add node** (or ⌘P / Ctrl-P, which opens the command palette) and pick
**MQTT**. In its panel on the right:
- **Consumes**: `living_temperature`, type `float`.
- **Provides**: `heat_wanted`, type `bool`.
- **Broker host** — your broker's hostname, `mosquitto` if you are using the
one the stack can start
- **Topic** — map each output to a topic:
`{"living_temperature": "zigbee2mqtt/living/temperature"}`
- **Provides** — add one output port named `living_temperature`, type `float`
That is a working node. Press **Publish** (⌘S) and the engine picks it up.
The canvas now draws your node with a live value on its output as soon as the
broker sends one. Click the wire to see the last payload and its history.
### Do something with it
Add a **Function** node. This is a Python node, and the code editor opens in
its panel:
In the code editor, enter:
```python
def process(living_temperature, comfortable=21.0):
"""Ask for heat when the room is below the comfort point."""
return {"heat_wanted": living_temperature < comfortable}
def process(living_temperature):
return {"heat_wanted": living_temperature < 21.0}
```
Declare `living_temperature` as an input (type `float`) and `heat_wanted` as an
output (type `bool`). `comfortable` is not a port but a **setting**,
because it is a constant of this node rather than something the graph carries.
It shows up as a field in the node's Settings section.
Message names connect the nodes automatically; there are no wires to draw.
Save the code with ⌘S / Ctrl-S, then click **Publish** in the dock. Open
**Flow settings** from the dock's pencil button and make sure **Running** is on.
The canvas now draws MQTT → your function, because the message names line up.
Nothing else was needed.
Every five seconds, `18.0` should produce `true`. Select the Function and check
the value under its `heat_wanted` output. Change the Inject value to `22.0`,
publish again, and check that the result becomes `false`. Click the edge to
inspect the temperature readings. Your first bit of automation is alive.
### Act on it
If nothing happens, open **Issues** and **Logs** in the dock. They show missing
inputs, code errors and connection problems. See [The flow editor](../interface/flow-editor.md)
for the controls and shortcuts.
Add a second **MQTT** node, this time with `heat_wanted` as an *input*, and a
topic mapping to whatever your relay listens on. A node with inputs publishes;
a node with outputs subscribes.
## Connect a real sensor
Publish the flow. You have a thermostat.
Replace the Inject node with an **MQTT** node:
!!! warning "Test before it touches a relay"
| Field | Value |
|---|---|
| **Provides** | `living_temperature`, type `float` |
| `broker_host` | your broker's hostname or IP, reachable from the engine |
| `topic` | the sensor's temperature topic |
A flow can be **paused** (it holds messages instead of running them) and
**stepped** (release exactly one). Together with the run button, which
injects a value by hand, that is how you convince yourself the logic is
right before the contactor finds out. Both live on the dock at the bottom
of the canvas.
Publish the change. The Function now responds to sensor readings instead of
the test value. If your sensor sends a JSON object such as
`{"temperature": 18.0}`, set `json_key` to `temperature`.
## Store the history
An MQTT node with input ports publishes to its configured topic; one with
output ports subscribes. Add a publishing node only after checking the payload
format your device expects. A relay may need `ON` and `OFF` rather than a
boolean.
A live value is enough to control something and useless for answering "was last
February colder?". That is what the **InfluxDB** node is for.
!!! tip "Keep the first test on screen"
Add one, and configure it to write the message you already have:
Check both warm and cold readings before connecting an actuator. **Pause**
and **Step** let you inspect queued messages one at a time. A real heating
controller also needs appropriate limits and hysteresis.
```json
{
"url": "http://influxdb:8086",
"token": {"$secret": "influx-token"},
"org": "home",
"bucket": "sensors",
"writes": {
"living_temperature": {
"measurement": "environment",
"field": "temp_c",
"tags": {"room": "living"}
}
}
}
```
Docker development stacks include a test broker at `mosquitto:1883`; it has
no published host port, so your physical sensors cannot reach it directly.
For real sensors, use your own broker. The Docker image includes the MQTT
and InfluxDB connector dependencies.
Give it `living_temperature` as an input and every value that passes gets a
point.
## Show it on a dashboard
Note the token. Credentials never sit in a flow: `{"$secret": "influx-token"}`
is a reference into an encrypted store, and the editor renders those fields as
a secret picker. Add the actual value once under **Secrets**. Flows are a git
repository you might well push somewhere, so this keeps a password out of it.
Create a dashboard and add a gauge bound to `house.living_temperature`.
Publish the dashboard to see the current reading. Dashboard controls can also
publish messages, so choose their bindings with care.
Reading back is the same node with `queries` instead of `writes`, or, for
anything a chart asks for, a pair of small Python nodes on either side that
build a Flux query and shape its rows. That indirection is deliberate: the
database node holds the connection and nothing else, so a dashboard widget
never learns which database answered it.
See [Dashboards and panels](../interface/dashboards.md) for widgets and pairing
a wall tablet.
## Put it on a screen
For longer-term history, add an **InfluxDB** node that consumes
`living_temperature`. Configure your server URL, organization, bucket, token
and `writes` mapping. Store the token under **Secrets** and select it in the
node's credential field. [Node types](../reference/node-types.md#influxdb)
has the write and query formats.
**Dashboards → New dashboard**, then drag widgets onto the grid and bind each
one to a message. A gauge on `house.living_temperature`, a switch on
`house.heat_wanted`, a chart on the history.
The Docker development stack includes an InfluxDB test service at
`http://influxdb:8086`, with organization and bucket `fluksio` and token
`fluksio-dev-token`. It is for testing; configure a persistent InfluxDB server
for data you want to keep. The production stack includes neither InfluxDB nor
an MQTT broker.
Widgets are typed the same way ports are: a switch binds to a `bool`, a gauge
to a number, an agenda to a `list`. Bind it wrong and the editor says so rather
than drawing nothing.
## Run it long term
Controls work in the other direction: a switch on a dashboard publishes the
message it is bound to, exactly as a node would. The canvas draws it as a
labelled endpoint feeding the nodes that read it, so a value never appears from
nowhere.
For a Docker deployment, run `make init`, then edit the repository's `.env`:
For a tablet on the wall, see [Dashboards and panels](../interface/dashboards.md):
a panel is a named device, it pairs with a six-character code instead of a
login, and it can only reach the dashboards you gave it.
| Setting | Example |
|---|---|
| `DOMAIN` | `example.com` |
| `ENVIRONMENT` | `production` |
| `FRONTEND_HOST` | `https://app.example.com` |
| `VITE_API_URL` | `https://api.example.com` |
| `BACKEND_CORS_ORIGINS` | `https://app.example.com` |
| `FIRST_SUPERUSER` | your administrator email |
| `TZ` | your timezone, such as `Europe/Berlin` |
## Spread it across machines
Keep the generated secrets. Stop any development stack with `make down`.
Set up DNS and a reverse proxy with HTTPS and WebSocket support before running
`make up`. That target starts the application services; it requires a proxy
on the Docker `proxy` network. Route
`app.example.com` to `fluksio-app:80`, `api.example.com` to `fluksio-api:8000`
and, optionally, `docs.example.com` to `fluksio-docs:80`.
You now have one box doing everything. Two reasons to change that: something
lives on a different network, or something needs hardware the engine's host
does not have.
`docker/compose.traefik.yml` is an optional proxy configuration. It also needs
`EMAIL`, `USERNAME` and `HASHED_PASSWORD` for certificates and dashboard
authentication; `make init` does not set these up.
The unit of distribution is the **worker**. It runs the code of nodes you mark
for it, and it dials *out* to the engine, so the Pi in the shed does not need
an inbound route, and the engine does not need to reach it.
The published port from `make dev-lan` uses HTTP. For access outside your
network, use HTTPS or [enroll with the portal](../interface/portal.md).
On the engine, mint a token:
Back up `.env` and both persistent volumes: `app-flow-data` holds the database,
flows, secrets and artifacts; `app-redis-data` holds message values, node state
and queued work. Compose prefixes volume names with the project name
`fluksio-app`. Also back up any external MQTT or InfluxDB data you need.
```sh
curl -X POST https://api.${DOMAIN}/api/v1/workers/tokens \
-H "Authorization: Bearer $TOKEN" -d '{"name": "shed-pi", "labels": ["shed", "gpio"]}'
```
## Next steps
The labels are signed into it, so the worker can repeat them with `--labels`
but not change them.
On the other machine, with that token in `FLUKSIO_WORKER_TOKEN`:
```sh
pip install fluksio-worker
fluksio-worker \
--url wss://api.${DOMAIN}/api/v1/workers/attach \
--labels shed,gpio \
--parallel 2
```
Then mark the node that talks to the shed's GPIO with `device: shed`, and it
runs there. Everything else stays where it is. A node bound to a label no
attached worker carries simply waits rather than failing, so you can write the
flow before the hardware arrives.
!!! note "What a worker is not"
It is not a second engine. Subscriptions, schedules, webhooks and the
dashboards all stay in one process, which keeps a value having one
definition. A worker executes node bodies, nothing else. Scaling *the
engine* to several processes is not supported: run one.
## Make it tell you when something breaks
Under **Alerts**, add a channel and a rule. Channels are ntfy (a push
notification on your phone), SMTP, a webhook, or a message a dashboard
notification widget reads.
What you probably want on day one is everything, to ntfy:
- add an ntfy channel with your server and topic
- add a rule with no events ticked, which means *all of them*
The engine deduplicates aggressively, so the same node failing every second is
one alert rather than thirty-six thousand, and caps the total at ten an hour
however bad it gets. See [Secrets, modules and alerts](../interface/operations.md).
## Keep it alive
- **Back up the data volume.** That is the database, the flows, the secrets and
the artifacts. Everything else is rebuildable.
- **Watch Home.** The health summary names what is wrong — a quarantined flow, a
node that will not load, a stalled queue, a flow that cannot run because its
graph does not validate.
- **Flows are git.** `git log` inside the flow store is the history of every
change anyone made, and reverting one is a revert.
## Where to go next
- [Flows, nodes and messages](../concepts/flows.md) — the model, properly
- [Node types](../reference/node-types.md) — everything you can put on a canvas
without writing Python
- [The flow editor](../interface/flow-editor.md) — the canvas, in detail
- [Keeping state in a flow](../concepts/state.md) — running totals, debounces,
and the one rule that makes them safe
- [Accounts and the portal](../interface/portal.md) — reach the instance
from outside the house without opening a port
- [Flows, nodes and messages](../concepts/flows.md): how the graph works.
- [Secrets, modules and alerts](../interface/operations.md): credentials and notifications.
- [Remote workers](../code/workers.md): run node code on another machine.
- [Configuration](../reference/configuration.md): settings and storage paths.
+17 -44
View File
@@ -1,9 +1,8 @@
# Pick your starting point
Setup differs depending on what you are here for. Pick the path that matches.
Pick the one that sounds like you. Everything past this section is the same for
both.
Choose the setup that fits your work.
Both paths lead to the same Fluksio engine, with a first flow to try along the way.
This is really just about a decision what you're planning to do with Fluksio.
<div class="fluksio-lanes" markdown>
@@ -11,13 +10,13 @@ both.
### Data science
*"I have a training script. I want to stop losing track of what I ran."*
*"I have some Python code. I want to keep track of what I ran."*
One `pip install`, one command, and you are writing Python again. No Docker, no
database, no ports to open. Flows are files, runs are rows, and the metrics are
just the numbers your loop already produces.
Install Fluksio with pip in your research environment. Turn a function into a
node, record metrics and artifacts, and compare experiments. Connect to
[fluksio.com](https://fluksio.com) for the browser dashboard.
[Set up for experiments →](data-science.md)
[Start with Python →](data-science.md)
</div>
@@ -25,46 +24,20 @@ just the numbers your loop already produces.
### Facility automation
*"I have a homelab and a pile of sensors. I want them to do something."*
*"I have a bunch of sensors. I want them to do something."*
A stack you bring up once and leave running: the engine, a broker, a
time-series database, dashboards, alerting. Most of the work happens in the
browser, and it is worth doing properly because you will live in it.
Start the Docker stack for a local engine and browser interface. Build a live
flow, connect your devices, and put their values on a dashboard.
[Set up a homelab instance →](facility-automation.md)
[Start with Docker →](facility-automation.md)
</div>
</div>
## Not sure?
## Next steps
Some rough tells:
| | Data science | Facility automation |
|---|---|---|
| **The flow** | starts, finishes, has a result | never ends |
| **You mostly** | write Python | wire nodes in the browser |
| **Time to first result** | a few minutes | an afternoon |
| **Runs on** | your laptop, or a login node | a box in a cupboard |
| **Data lives in** | SQLite beside the flows | InfluxDB, usually |
| **The thing you look at** | run history and loss curves | a dashboard, maybe on a wall |
If both describe you (a lab with instruments to drive *and* models to fit),
start with the data-science path. It is the smaller instance, and it
grows into the other one without being reinstalled: the same engine, the same
flows, just more of them running all the time.
## What is the same either way
Whichever door you came in:
- **Flows are files in a git repository.** Every save is a commit. You can read
the history with ordinary git, and you can copy a flow between instances
by copying a directory.
- **Editing is separate from running.** You edit a draft; the engine keeps
running what was published until you publish.
- **Nodes are typed.** A port declares what it carries, and a mismatch is
caught at edit time rather than at three in the morning.
- **Everything the browser does is an API call.** The dashboard is a client of
the same REST API you can script against.
- [Flows, nodes and messages](../concepts/flows.md) explains the graph.
- [The flow editor](../interface/flow-editor.md) covers editing and publishing.
- [The command line](../code/cli.md) covers local and remote use.
- [Configuration](../reference/configuration.md) lists settings and data paths.
+21 -41
View File
@@ -1,53 +1,33 @@
# Fluksio
Fluksio is a node-based automation engine. You describe what should happen as a
graph of small pieces of logic, and it keeps that graph running: reacting to
what arrives, or executing once from parameters to a result.
Fluksio is a node-based automation engine for experiments, buildings, and
homelabs. Connect small pieces of logic through typed messages, then let the
engine do the running.
It runs two kinds of graph:
- **Live flows** react to sensors, webhooks, and schedules.
- **Batch runs** take parameters, finish with a result, and keep metrics,
artifacts, and a history you can compare.
- A **live flow** produces values forever. A sensor publishes, a rule fires, a
relay closes, a dashboard on the wall shows what happened. It never ends.
- A **run** produces a value once. Parameters go in, stages execute, and it
finishes, leaving metrics, artifacts and a record to compare against.
Both use the same nodes, the same type checking, the same editor and the same
API.
## What you get
- A **flow engine** that owns your graph, checks the types on every edge, and
keeps running when a node fails.
- A **canvas** that lays flows out for you and an editor for the Python inside
each node, with the running values drawn on the wires while you work.
- **Dashboards** built next to the logic that feeds them, including ones you
can hang on a wall tablet that has no keyboard.
- **Runs**: parameters, metrics, artifacts, sweeps and a queryable history, with
no second server and no project bootstrap per execution.
- **Distributed workers**: a node marked `device: gpu` runs on the machine that
has one, which dials out to the engine rather than needing to be reachable.
- An **HTTP API** covering everything the browser does, plus an MCP endpoint for
agents.
Both use the same nodes, editor, and API. Flows are versioned in git, so your
next bright idea comes with an undo trail.
## Start here
[Getting started](getting-started/index.md) has two paths: one installs a Python
package, the other stands up a server. Everything after that is shared.
[Pick your starting point](getting-started/index.md):
To look before installing, the hosted demo at
[fluksio.com](https://fluksio.com) runs a real instance: a small-house panel, a
media screen, and a lab where a training pipeline runs and its runs are
compared, four screens on one panel.
- [Data science](getting-started/data-science.md): install with pip in your
research environment and run your first experiment.
- [Facility automation](getting-started/facility-automation.md): start the
Docker stack and build your first live flow in the browser.
## Where things are
## Explore
| If you want to… | Read |
| To… | Read |
|---|---|
| Understand what a flow, a node and a message are | [Concepts](concepts/flows.md) |
| Drive Fluksio from the browser | [The interface](interface/index.md) |
| Drive it from Python, a shell or CI | [Code and the CLI](code/cli.md) |
| Look up a node type or a payload type | [Reference](reference/node-types.md) |
| Understand flows, nodes, and messages | [Concepts](concepts/flows.md) |
| Edit flows and build dashboards | [The interface](interface/index.md) |
| Use Python, the CLI, or the API | [Code and the CLI](code/cli.md) |
| Look up nodes, payloads, and settings | [Reference](reference/node-types.md) |
Fluksio is self-hosted by default. An instance runs offline, keeps its data
on its own disk, and never contacts anything unless you
[connect it to a portal](interface/portal.md) yourself.
Your instance keeps its data on its own disk. Connecting it to a
[portal](interface/portal.md) is optional; local flows and runs work offline.
+137 -244
View File
@@ -1,324 +1,217 @@
# Dashboards and panels
A dashboard is a grid of widgets bound to message names, the same names that
wire the graph. It is its own document rather than a set of nodes, so it reads
across flows without belonging to any of them, and a flow stays the logic it
was.
There is no second tool and no separate deployment. You build the panel next to
the thing feeding it.
A **dashboard** is a grid of widgets bound to message names. It can show values
and send controls across several flows. A **panel** assigns published dashboards
to a paired screen, such as a wall tablet.
## Building one
**Dashboards → New dashboard**, then drag widgets onto the grid from the widget
picker and bind each to a message.
1. Open **Dashboards → New dashboard**.
2. Enter edit mode, choose **Add widget** and select a type.
3. Bind the widget to a message in its settings. On desktop, drag and resize
tiles on the grid.
4. **Publish** when the dashboard is ready.
Like flows, dashboards are drafts until you publish. A wall panel reads only
the published version, so a half-arranged page never reaches the wall.
Edits stay in a draft; paired screens show the published version. On phones,
tiles stack in one column and dragging is disabled, but settings remain editable.
On a phone the dashboard is read, not arranged: the grid stacks to one column
and dragging is off. Picking a widget and editing its settings still works.
Controls can publish real values while you edit, so choose their targets with
care. Your new Switch may already be flipping something!
## The widgets
### Display
| Widget | Binds to | Notes |
| Widget | Binds to | Shows |
|---|---|---|
| **Value** | anything | a formatted reading with a unit and a precision |
| **Gauge** | `float`, `int` | min, max, unit |
| **Chart** | `float`, `int` | up to five series; see *Two kinds of chart* below |
| **Bar** | `float`, `int` | up to eight readings, a row each, with a scale per row |
| **Icon** | numbers, booleans, weather strings | maps a value onto a glyph |
| **Text** | — | markdown you write; a label, a note, an instruction |
| **Agenda** | `list` | upcoming items, e.g. from a calendar connector |
| **Forecast** | `list` | a short outlook strip |
| **Notification** | `record` | title, body and severity — what an alert channel writes |
| **Media** | `image`, `audio`, `video` | a camera frame, a clip; see *Media tiles* below |
| **Player** | `record` | what a streamer is playing, with its transport; see *Player tiles* below |
| **Clock** | — | the time, in a size a wall can read |
| **Website** | — | another site's page in the tile; see *Website tiles* below |
| **Value** | anything | formatted reading, unit and precision |
| **Gauge** | `float`, `int` | reading within a min/max range |
| **Chart** | numeric messages, a query response or run metrics | up to five lines |
| **Bar** | `float`, `int` | up to eight readings with separate scales |
| **Icon** | numbers, booleans or strings | a glyph selected from the value |
| **Text** | — | Markdown |
| **Agenda** | `list` | upcoming items |
| **Forecast** | `list` | an outlook strip |
| **Notification** | `record` | title, body and severity |
| **Media** | `image`, `audio`, `video`, `artifact` | image, audio or video |
| **Player** | `record` | playback status and transport controls |
| **Clock** | — | current time |
| **Website** | — | embedded page or image |
Every widget carries a **title**, and **Show title** decides whether the panel
draws it. Turned off, the tile is just the reading, which is what a row of
gauges under one heading wants. The title is still the widget's name: what a
screen reader calls its controls, and what a published value is labelled with.
Every widget has a title. **Show title** controls its visibility; the title
still names the widget for accessibility and published values.
### Input
| Widget | Publishes | Notes |
| Widget | Publishes | Settings |
|---|---|---|
| **Button** | a fixed value | one-shot: run it, open it, reset it |
| **Button** | fixed value | one-shot action |
| **Buttons** | a value per button | grid of actions |
| **Switch** | `bool` | on/off |
| **Slider** | `float`, `int` | min, max, step; **Drawn as** makes it a vertical fader |
| **Input** | text or a number | free entry |
| **Selector** | one of a list | a mode, a scene, a preset — as a menu, or as a row of choices with the active one held |
| **Colour** | `[h, s, v]`, `[r, g, b]` or `"#rrggbb"` | a hue wheel with saturation and brightness, for an RGB fixture |
| **Slider** | `float`, `int` | min, max, step; horizontal or vertical |
| **Input** | text or number | free entry |
| **Selector** | selected value | menu or row of choices |
| **Colour** | list or string | HSV, RGB or hex |
The colour wheel sends `[h, s, v]` by default (hue 0-360 degrees, saturation
and value 0-100 percent, which is what a DMX encoder expects) and its
**Sends** setting switches that to `[r, g, b]` (0-255 each) or to a
`"#rrggbb"` string, because fixtures differ. The first two bind a `list`
message, the third a `str`.
Colour defaults to `[h, s, v]`: hue 0–360°, saturation and brightness 0–100%.
**Sends** can select RGB (`[r, g, b]`, 0–255) or hex (`"#rrggbb"`).
HSV and RGB bind to `list`; hex binds to `str`.
A control publishes the message it is bound to, exactly as a node would. On the
flow canvas it is drawn as a labelled endpoint feeding the nodes that read it,
so nobody has to guess where the value came from.
Controls publish to their bound messages. The flow canvas shows them as
endpoints feeding the nodes that consume those messages.
### Typed bindings
Bindings are checked in the editor and on the server. See
[Payload types](../reference/payload-types.md).
Widgets are type-checked against the message the same way ports are: a switch
takes a `bool`, a gauge takes a number, an agenda takes a `list`, a
notification takes a `record`. Bind one wrong and the editor says so rather
than drawing nothing. The same table is enforced on the server.
## Charts
Charts draw their series in one fixed order of colours: the primary first, then
hues picked to stay apart, colour blindness included. A chart with more than
one series still always draws a legend, so no line is told by colour alone.
Choose the chart's data source:
## Two kinds of chart
- **Live:** recent numeric message history, updated as values arrive.
- **Query:** publishes a `record` request containing `range_s` and
`interval_s`, then draws a flow's `series` response. Include those fields in
the answer so the chart can reject responses for a different window.
- **Runs:** compares a numeric metric from named runs, a sweep or the latest
runs of a flow. See [Runs](../concepts/runs.md).
**Live.** Bind up to five messages and the chart draws the engine's own ring
buffer of recent values, extended over the socket as new ones land. Nothing
else is involved. This is what you want for "the last few hours of the living
room".
Multi-line charts show a legend as well as different colours.
**Querying.** The chart publishes a *request* (the window and resolution it
wants) exactly as a slider publishes a value, and draws the `series` some flow
answers with. What serves that request is the flow's business: typically a
small Python node that builds a Flux query, an InfluxDB node that runs it, and
another Python node that shapes the rows.
!!! tip "Skip the unfinished bucket"
The widget never learns which database answered it, so swapping the store
changes one flow and nothing else. An answer states what it was computed for,
so an answer to a different question is ignored.
!!! note "Drop the bucket that is still filling"
The request carries a window and an interval, and the binning is the flow's
own work, so the newest bucket only ever holds the part of an interval
that has elapsed. Drawn, it reads as a fall that never happened. The
engine's own [Activity charts](index.md#activity) end on the last closed
bin for that reason; a flow answering a chart has to drop or hold back its
newest bucket the same way.
If your query aggregates into time buckets, leave out the bucket still
filling. Otherwise an incomplete total can look like a sudden drop.
## Media tiles
A media widget draws what its message points at: a picture, a clip with
controls, a video. Media does not travel as a message; a reference to it does.
The tile takes the bytes behind whichever reference the message holds, and
redraws when a new one arrives.
Media messages contain artifact references; the widget loads the bytes they
name. **Crop or fit** controls image sizing. **Play as it arrives** enables
autoplay, subject to the browser's rules for sound.
**Crop or fit** decides how a picture fills the tile. **Play as it arrives**
starts a clip by itself, though a browser only plays sound once somebody has
touched the page, so a screen nobody has tapped stays silent.
For local streams, `volatile=True` artifacts can be pushed over the websocket
with their message. Stored artifacts are fetched individually, as are frames
viewed through a portal. Use occasional snapshots for remote screens.
Rate is the thing to get right, and it is decided by the node publishing rather
than by the tile. A frame the engine is holding in memory is pushed down the
same websocket that carries the value, so ten a second is a real view on the
local network; a stored one is fetched, a round trip each, which suits a glance
at a door every second or two. Through the portal everything is fetched, so
make it every few seconds there. Above that, put the camera's own address in
**Live stream** and the browser plays it from source, leaving the messages to
carry the occasional still that a flow can actually react to.
Panels see media the same way, and only their own: a screen is sent, and may
fetch, the bytes its own tiles are showing and nothing else.
For continuous video, set **Live stream** to a browser-compatible camera URL.
The browser must be able to reach that URL; enrollment does not expose the
camera itself. See [Media payloads](../reference/payload-types.md#image-audio-video).
## Player tiles
A player is the one tile that both reads and publishes, so it has two bindings.
It **shows** a `record` describing what is playing and **publishes to** a `str`
carrying what to do about it:
A Player reads a status `record` and publishes commands to a separate `str`
message.
| Field of the record | Means |
| Status field | Meaning |
|---|---|
| `title`, `artist`, `album` | what is playing |
| `status` | `play`, `pause`, `stop`, `load`, or `off` for a streamer with no power |
| `position`, `duration` | seconds, both |
| `title`, `artist`, `album` | track details |
| `status` | `play`, `pause`, `stop`, `load` or `off` |
| `position`, `duration` | seconds |
The buttons and the bar publish words: `toggle`, `next`, `prev` and
`seek:<seconds>`. Those are a streamer's own vocabulary rather than this app's,
which is what lets one tile drive whatever is on the other end: the node that
receives them decides what they mean for its device.
The position counts forward in the browser between readings, so the bar moves
at one second while the device is polled at whatever rate suits it. Every
reading that arrives is taken as the truth and the count restarts from it.
Controls send `toggle`, `next`, `prev` and `seek:<seconds>`. Your receiving
node translates those into device actions. The position advances in the
browser between readings and resets to each fresh reading.
## Website tiles
A website tile frames whatever page its **Address** points at: a weather map, a
camera's own web interface, a timetable. A site that refuses to be framed shows
its own refusal in the tile.
Set **Address** to a page to embed, or choose **Image** for a snapshot or MJPEG
URL. A website may refuse embedding.
The page runs sandboxed. By default it gets an origin of its own that matches
no site, so it can run scripts but has no cookies or storage, and many
interactive pages will not load like that. **Trusted site** lets it keep its
real origin, with its cookies and storage, which is what visiting the site
would give it. Turn it on only for a site you trust. A trusted tile pointing at
this app's own address, or the hub's when the app is reached through one, shows
a notice instead: on that origin the page could reach into the app and read
your session.
**Image** draws the address as a picture scaled to fit the tile, rather than
as a page. Use it for a camera's MJPEG stream (`http://<camera>/stream`) or a
snapshot URL, which a page frame would show at the camera's native size.
Embedded pages run in a sandbox without cookies or storage by default.
**Trusted site** permits the site's own origin, cookies and storage.
Use it only for a trusted destination. The app blocks trusted embeds of its
own origin, including the portal origin, to protect your session.
## Dashboard settings
Most of what a dashboard carries is a widget: a tile bound to a message. A
handful of things are not, because they belong to the whole surface rather than
to any tile on it, and a screen bolted to a wall has nobody standing at it to
set them.
Open **Dashboard settings** from the dock. Each setting can hold a fixed value
or follow a message chosen in **Driven by**. The fixed value is the fallback
when no usable message value is available.
| Setting | Is | Driven by |
| Setting | Choices | Message type |
|---|---|---|
| **Look** | `Fluksio`, `Material` or `Glass` | a `str` message |
| **Theme** | `System`, `Light` or `Dark` | a `str` message |
| **Palette** | the dashboard's colours, in order | a `list` message |
| **Background** | the URL of an image | a `str` message |
| **Lock** | read-only on or off | a `bool` message |
| **Look** | `fluksio`, `material`, `glass` | `str` |
| **Theme** | `system`, `light`, `dark` | `str` |
| **Palette** | ordered hex colours | `list` |
| **Background** | image URL | `str` |
| **Lock** | read-only on/off | `bool` |
They all work the same way, and both halves are optional:
To change one on a schedule, have a scheduled node publish its bound message.
- **Just a value.** Set Theme to `Dark` and that dashboard is dark wherever it
is shown, whatever the device or the browser prefers. This costs no flow at
all, and it is what most wall panels want.
- **Driven by a message.** Pick one in **Driven by** and a flow takes the
setting over, exactly as it drives a tile. The value you set stays the
fallback: what the dashboard uses before the first message arrives, and
whenever the message is silent.
### Look and theme
A bound setting is type-checked like a widget binding, so a Theme pointed at a
`float` is refused by the editor and by the server. The panel's credential
is extended to it, so a paired screen may read its own theme message and
nothing further.
**Fluksio** is the default look. **Material** uses flat tonal cards; **Glass**
uses translucent tiles. Widgets work the same in all three.
There is no schedule field. **A schedule is a node publishing to the bound
message**: an `inject` with a cron expression feeding a `change` node that maps
the hour onto a palette is the whole of "warmer after sunset". The same channel
drives anything else, including locking a panel down remotely.
### Look
**Fluksio** is the app's own design (`--card` surfaces separated by a hairline
and a low shadow, pill controls, one slate-blue accent) and is what a dashboard
wears until you choose otherwise. **Material** lays flat, tonal cards on a plain
ground. **Glass** floats translucent tiles over a soft, slowly moving one.
They are three complete sets of components, not three stylesheets, but they are
the same dashboard: every widget, every control and every keystroke behaves
identically, so switching look never changes what a panel can do.
Theme follows the chosen preference unless a palette sets the background
colour.
### Palette
A palette is an ordered list of colours, and **position is the role**:
Paste a colour list or a [Coolors](https://coolors.co) palette link. Colours
have roles by position:
| # | Role |
| Position | Role |
|---|---|
| 1 | the ground the dashboard sits on |
| 2 | the surface a widget is |
| 3 | the primary — fills, active controls, the first chart line |
| 4 | the accent |
| 1 | background |
| 2 | widget surface |
| 3 | primary controls and first chart line |
| 4 | accent |
| 5 | text |
| 6+ | the next chart lines, before the app's own chart colours |
| 6+ | additional chart lines |
Paste a [coolors.co](https://coolors.co) link (or a list of hex colours) and the
whole dashboard is recoloured: every widget, the rail and the charts. Leave the
later roles off and they are worked out from the ones you gave, so **three
colours are a whole dashboard**. **Rotate** turns the list when the roles landed
in the wrong order. Text that could not be read on a surface is replaced with
black or white there, so no palette can produce a line nobody can see.
Missing roles are derived, so three colours are enough to get started.
**Rotate** shifts their order. Text contrast is corrected automatically;
failure and success indicators keep their status colours.
While a palette is set, **Theme is idle**: the first colour is the ground, so
whether the panel reads light or dark is already decided by the palette itself.
Fault and success keep their own colours in every palette, so a failure is never
paintable as a reading.
### Background and lock
### Background
An image drawn under the widgets, covering the canvas. It replaces the ground
the Glass look brings with it. Bound to a message, a flow decides the picture.
### Lock
Lock is a read-only *surface*, not a permission. The controls stay visible,
stop publishing and read as disabled, and the panel says **Read-only** in the
corner. What a paired screen is allowed to reach is still decided by its own
credential, below.
Background draws an image beneath the widgets. Lock disables publishing from
controls and shows **Read-only** when viewing the dashboard. It does not
restrict API permissions and does not disable controls in edit mode.
## Showing one
- `/view/{name}` — a browser tab pointed at one dashboard. Needs an ordinary
session.
- **Panels** — a named device that pairs instead of logging in. Below.
Open `/view/{name}` for a dashboard-only view with your normal session.
For a dedicated screen, pair a panel.
## Panels: hanging a screen on a wall
A wall tablet has no keyboard, so it pairs.
As a superuser:
1. **Dashboards → Panels**, add a panel named after where it hangs, and tick
the dashboards it shows. More than one and the screen draws a rail to switch
between them, with each dashboard's own icon on it. On a phone the rail is a
row floating along the bottom; the app shows it there too, above its own
buttons, for a dashboard that hangs on such a panel.
2. Tick **Touch friendly** if the screen is touched rather than pointed at.
Controls and the rail grow to a finger's size — the rail gets taller without
taking a wider column, so the arrangement does not move. It sits here rather
than on a dashboard because it describes the screen: the same dashboard may
also be open in a browser with a mouse. A phone gets it anyway, from its own
width.
3. Point the device's browser at the link the dialog shows. The device then
displays a six-character code.
4. Type that code into the same panel's **Pair device** field. The line under
it names what is holding the code. Check it is the screen you just hung,
because approving adopts whatever answered. The screen picks the credential
up within a few seconds and never asks again.
1. Open **Dashboards → Panels**, add a panel and select its dashboards.
2. Enable **Touch friendly** for larger controls.
3. Open the displayed pairing link on the device. It shows a six-character code.
4. Enter that code for the panel, check the device description, then press
**Pair device**.
What the screen holds is not a login. It reaches that panel's published
dashboards and the messages its own widgets read or publish, and nothing else.
A message no tile on it draws is refused in both directions.
The screen collects its credential and shows the assigned published dashboards.
Several dashboards get a navigation rail.
The *credential* cannot be made strictly read-only, and that is honest rather
than an oversight: a querying chart publishes its request, and a control on a
panel is the reason you put one there. The dashboard's own **Lock** setting
above stops its controls publishing and can be driven by a flow, which is how
you quieten a screen remotely, but that is the surface behaving, not the
credential being narrowed.
A panel credential permits its assigned dashboards, their bound messages and
the artifacts they show. It also permits publishing from its controls and query
charts. **Lock** changes the viewing surface; it does not narrow the credential.
Deleting the panel revokes the credential and the assignment together, which is
how you retire a device and what it showed. Unpairing revokes only the
credential: the panel, its dashboards and their arrangement stay exactly where
they are, and the screen falls back to asking for a new code, which is how you
swap the device out without rebuilding what hangs there.
**Unpair screen** revokes a locally paired credential and keeps the panel's
dashboard assignments. Deleting the panel removes the assignment and its
access.
!!! note "If the link is wrong"
!!! note "If the local link is wrong"
The pairing link is built from the instance's `FRONTEND_HOST`. If that
is not the address devices on your network actually reach, fix the setting
rather than the link: it is the same one password-reset mails and the OAuth
metadata are built from.
Pairing links use `FRONTEND_HOST`. Set it to the app address reachable
from the screen's network.
### A screen somewhere you cannot reach
Another building, someone else's network, no route in. An instance
[enrolled with a portal](portal.md) shows a second link,
`https://hub.${DOMAIN}/i/{instance-id}/panel`, and the same three steps
work through it: the portal serves that one page without a session, forwards
the pairing calls down the tunnel, and mints the credential when you approve
the code. The pairing line then reads *via portal*.
An [enrolled instance](portal.md) also shows a portal pairing link. Use that
link on the remote screen, then approve its code in the same dialog. The
device description says *via portal*.
The portal names the panel and nothing else. What the panel may read is decided
on the instance, on every call, by the same check a locally paired screen
passes. Two differences: it acts as the account the instance was enrolled
with rather than as whoever approved it, and deleting the panel stops it here
immediately while the portal's copy of the token expires on its own, which is
also why unpairing, which works on a credential this instance signed, does
not reach a remote screen. Revoke that one at the hub.
The instance enforces the same dashboard and message limits. Remote
credentials are issued by the portal, so **Unpair screen** does not revoke
them; revoke them at the portal. Deleting the panel removes access on the
instance.
## See also
- [Payload types](../reference/payload-types.md) — what a widget can bind to
- [Flows, nodes and messages](../concepts/flows.md) — where the names come from
- [Accounts and the portal](portal.md) — reaching all of this from outside
- [Payload types](../reference/payload-types.md)
- [Flows, nodes and messages](../concepts/flows.md)
- [Accounts and the portal](portal.md)
+99 -127
View File
@@ -1,177 +1,149 @@
# The flow editor
The canvas is where most of the work happens. It floats its chrome over a
full-bleed graph, so what you are looking at is always the flow.
Build a flow by naming its messages and configuring its nodes. Fluksio draws
the connections and arranges the canvas for you.
## The canvas
Nodes are laid out for you, left to right on a desktop and top to bottom on a
phone, and the arrangement is recomputed as the graph changes. There is
nothing to drag and nothing to tidy.
The graph runs left to right on desktop and top to bottom on phones.
The edges come from message names, so there is no layout to store or to merge
when two people edit the same flow.
- **Nodes** show their execution status.
- **Edges** pulse when values pass through. Click one to inspect its payload.
- **Endpoints** show flow inputs, dashboard widgets and nodes in other flows
that exchange messages with this flow.
What is drawn:
- **Your nodes**, with a status dot — running, succeeded, failed, or nothing at
all when idle. Status is always named in words too, never colour alone.
- **Edges**, which pulse when a value goes down them. Click one to inspect it.
- **Endpoints** — anything wired into this flow that is not a node in it: a
dashboard control writing a message, a tile reading one, a node in another
flow on the far side of a dotted name. Drawn as labelled boxes at the
boundary, so a value never appears to come from nowhere.
Values are live. The editor holds a websocket to the engine, and the connection
banner tells you when it is not.
Values update over a websocket. A connection banner tells you when the editor
cannot reach the engine.
## Adding a node
**Add node** on the dock, or ⌘P / Ctrl-P for the command palette, which also
jumps between flows and offers your shared nodes. Pick a type and it appears on
the canvas with its panel open.
Choose **Add node** in the bottom dock, or use ⌘P / Ctrl-P. The command palette
also jumps between flows and offers reusable nodes. Select a type to add it
and open its settings.
## The node panel
Floats on the right; a full-screen sheet on a phone.
**Consumes** and **Provides** declare the node's ports. Each has a port name,
payload type, bound message and delivery interval. Outputs also have a
streaming toggle.
**Consumes** and **Provides** are the ports. Each port has two lines: its name
and payload type (a `list` also says what its items are), then the message it
is bound to, how often at most it is delivered, and, on an output, a
*streaming* toggle for ports that publish repeatedly during one execution. The
message field suggests messages already in the flow, which is usually how you
wire something: type the name that already exists.
Message fields suggest existing names. Matching an output's message is how
you connect an input.
A port is named after its message until you name it yourself. Its name field
then stays empty and shows the name it follows. A name typed there stays when
the message changes, which is how a port is bound to a message called
something else, as a [`wren`](../reference/node-types.md#wren) target is;
clear it to follow the message again. The name is what the node's code sees,
so it is letters, digits and `_`, not starting with a digit, and no two inputs
or two outputs of a node share one. The field keeps the name it had otherwise.
A blank port-name field follows the message's last segment. Set a port name
to keep the argument name fixed while changing its message binding. Names use
letters, digits and `_`, cannot start with a digit, and must be unique among
the node's inputs or outputs.
Renaming an output renames the message everywhere it is read. Inputs are not
renamed, because an input is more often re-pointed at a different message than
renamed.
Renaming an output's message updates its consumers in the flow. Changing an
input binding points that input at a different message.
**Settings** is generated from the node type's parameter schema, so an MQTT
node shows broker, topic and QoS, and an ntfy node shows server, topic and
priority. A field marked as a credential renders as a picker over your stored
[secrets](operations.md) instead of a text box. A setting holding JSON, such as
a Switch node's rules, shows one line of it; click it to edit the whole value
in a larger editor, which saves only once the text parses. Escape or Cancel
leaves the setting as it was. An inject's payload and a flow input's starting
value of type `json`, `record` or `list` open the same editor, which also holds
the shape (a `record` is an object, a `list` an array); emptying it leaves no
value.
### Settings
A Function node gets a second settings section where you add your own: a
setting is a constant of this node's code and arrives as an argument beside its
Built-in nodes show forms for their configuration. Credential fields offer
stored [secrets](operations.md#secrets). JSON settings open a larger editor
and save only when the value parses.
Function nodes also let you add settings passed as arguments alongside the
ports. See [Where a node's values come from](../concepts/values.md).
**Timeout** (Function nodes) is how many seconds the code may be *silent*
before it is stopped: it measures silence, not duration, so a node that yields
every few seconds can run for hours under a timeout of 300. Left empty there is
no limit, which is the default. Set one where a node going quiet means it is
stuck rather than working. A `0` says so explicitly.
**Timeout** on a Function node limits silence, not total duration. A node
yielding regularly can run longer than its timeout. Empty or `0` means no
limit, the default.
**Code** is the editor. It saves as you type and applies on ⌘S; the node
reloads without the flow stopping. A node file defines `process(...)`; if it
defines exactly one public function under another name, that one is used
instead.
**Resources** selects a shared worker pool, a named flavor or custom CPU,
memory and GPU requirements. Placement by `device` and `device_policy` is
configured through the API or flow JSON; see [Remote workers](../code/workers.md).
Code runs as the engine, so changing it is a superuser's. Anyone else sees the
editor read-only, a Command node's settings disabled, and no Function, Command
or reusable nodes to add.
### Code and reuse
**Shared** turns a node's code into a library entry other flows can reuse.
Editing a shared node edits the copy every flow using it runs, which is the
point and also the caution.
Define `process(...)` in **Code**. If the file instead contains exactly one
public function, Fluksio uses that function.
!!! note "Not in the panel yet"
Code saves as you type. ⌘S / Ctrl-S in the code editor saves immediately;
**Publish** applies ordinary node code to the running flow. Generated code
from `fluksio sync` is read-only: edit its source repository and sync again.
`device` and `device_policy`, which machine a node's code runs on, are set
through the API rather than the panel, with `PUT /flows/{name}`. See
[Remote workers](../code/workers.md).
Only superusers can add or edit code-running nodes. Other users can read code
and configure permitted built-in nodes.
**Reuse** shares node code as a library entry. Saving shared code takes effect
immediately in every flow using it, so check its users before changing it.
## The flow panel
Click the canvas title to open the flow's own settings.
Open **Flow settings** from the pencil button in the dock.
- **Running** — the switch that starts and stops the flow. Stopped means none
of its subscriptions, schedules or webhooks exist.
- **Mode** — `live` or `batch`. Live reacts to what arrives; batch only runs
when a run asks it to. See [Runs](../concepts/runs.md).
- **Inputs** — messages the flow takes from outside, with the value each starts
from. For a batch flow these are also the run's parameters.
- **Outputs** (batch only) — which messages a run reports as its result.
- **Contents** — a reminder that this flow's name namespaces every message
inside it.
| Setting | Purpose |
|---|---|
| **Running** | starts or stops the published live flow's subscriptions, schedules and webhooks |
| **Mode** | chooses continuous `live` operation or requested `batch` runs |
| **Inputs** | declares external messages and starting values; batch inputs become run parameters |
| **Result** | selects batch output messages |
| **Contents** | shows the node count and message namespace |
For live inputs, **Publish this value now** sends the starting value immediately.
Changing a stored starting value alone does not replace an existing message.
## Inspecting an edge
Click a wire. You get the last payload that went down it, when, and a sparkline
of its recent numeric history. You can also republish that value from the node
that produced it, which re-runs everything downstream. It is the fastest way to
test a change without waiting for the real sensor.
Click an edge to see its latest payload, timestamp and recent numeric history.
Republish its value from the producer to run downstream nodes without waiting
for the next sensor reading.
## The dock
Along the bottom, floating over the canvas.
| Control | What it does |
| Control | Action |
|---|---|
| **Add node** | opens the palette |
| **Fit** | returns to the view the flow opened with |
| **Run** | injects a value by hand; on a batch flow, opens the parameters dialog |
| **Pause / Resume** | holds messages instead of running them |
| **Step** | releases exactly one held message |
| **Publish** | puts the draft live |
| **Discard** | throws the draft away |
| **Issues** | what is stopping this flow from running, per node |
| **Logs** | what the nodes printed, filterable |
| **Add node** | opens the command palette |
| **Fit to screen** | fits the graph; desktop only |
| **Run** | executes the current live flow once; for batch flows, asks for run parameters |
| **Pause / Resume** | holds or releases pending work while values still arrive |
| **Step** | releases one held item while paused |
| **Flow settings** | opens the flow panel |
| **Publish** | puts the saved draft live |
| **Discard** | removes unpublished changes |
| **Issues** | lists problems by node |
| **Logs** | shows filterable node output |
Pause and step together are how you test a flow before it moves anything
physical: pause it, inject a value, step it through, watch the values appear on
the edges, then publish.
**Run** uses the current draft when one exists without publishing it. It
executes real node code, including device and network actions. To try a single
value in a live flow, publish it from a flow input or edge inspector.
Pause and step are useful for inspecting held work. They do not make a draft
run a simulation.
## Shortcuts
| Chord | Action |
Use ⌘ on macOS and Ctrl elsewhere.
| Shortcut | Action |
|---|---|
| ⌘P / Ctrl-P | command palette |
| ⌘S / Ctrl-S | publish the flow — or, with focus in the code editor, apply the code |
| ⌘Z / ⌘⇧Z | undo / redo (the flow; the code editor has its own) |
| ⌘C / ⌘V | copy and paste nodes, including between flows |
| ⌘S / Ctrl-S | publish the flow; inside the code editor, save code |
| ⌘Z / Ctrl-Z | undo a graph edit |
| ⌘⇧Z / Ctrl-Shift-Z | redo a graph edit |
| ⌘C / Ctrl-C | copy selected nodes |
| ⌘V / Ctrl-V | paste nodes, including into another flow |
## Drafts and publishing
Text fields and the code editor keep their own undo and clipboard behavior.
Saving writes a draft; the engine keeps running what was published. Publishing
promotes it and reloads.
## Drafts, conflicts and issues
If someone else saved the same flow while you were working, publishing answers
with a conflict rather than overwriting them. The Flows list offers **Publish
all changes** when several flows are sitting on drafts.
Edits are saved as drafts. The engine keeps running the published version until
you publish; shared code follows the exception above. **Discard** returns to
the published version. Delete a never-published flow to remove it.
## Issues
If another editor saved the same flow, a stale save or publish returns a
conflict. Reload and review their changes before retrying.
The canvas validates as you edit and marks the node each issue belongs to:
- a port that needs a message nothing in reach provides
- a message that exists but has never held a value
- a dependency cycle
- a node reading a message it also writes, with nothing to start it from
- code that did not load
- a webhook with no shared secret (advisory — it does not stop the flow)
- a node that loaded but reports itself down, such as a connector that cannot
reach its device
A flow with any of these except the last two does not run, and the health
summary on Home counts it. The last one clears on its own once the node reports
itself well again.
The canvas reports unconnected inputs, missing starting values, cycles and
code-load errors. These prevent the affected flow from running. Unauthenticated
webhooks are advisory; runtime node-health problems clear when the node
recovers. See [Flow issues](../concepts/flows.md#when-a-flow-needs-attention).
## See also
- [Flows, nodes and messages](../concepts/flows.md) — the model behind the canvas
- [Node types](../reference/node-types.md) — everything the palette offers
- [Writing node code](../code/nodes.md) — what goes in the editor
- [Flows, nodes and messages](../concepts/flows.md)
- [Node types](../reference/node-types.md)
- [Writing node code](../code/nodes.md)
+56 -97
View File
@@ -1,128 +1,87 @@
# The dashboard app
The web interface is a single-page app served at `app.${DOMAIN}`, or, for an
instance reached through a portal, at `${DOMAIN}/i/{instance-id}`.
Either way it is the same application, and it is a client of the same REST API
you can script against.
Open your instance's app address, or select it at [fluksio.com](https://fluksio.com)
after enrollment. For the local Docker setup, use
[app.localhost](http://app.localhost) and the admin login printed by
`make init`. See [Getting started](../getting-started/index.md) for setup.
Sign in with the account the instance was created with. On a fresh
instance that account was printed once, on the first start.
## Finding your way around
## The shell
The sidebar opens as a sheet on phones.
A floating sidebar on the left, the selected screen filling the rest. On a
phone the sidebar collapses to a sheet.
| Entry | What lives there |
| Entry | Use it to |
|---|---|
| **Home** | the brain graph, health, and everything that recently happened |
| **Flows** | the list of flows, and the canvas for each |
| **Dashboards** | the widget canvases, and the panels that display them |
| **Runs** | starting a run with values you choose, and the log of what ran |
| **Workers** | the machines that nodes can be placed on |
| **Secrets** | credentials your nodes reference without holding |
| **Modules** | the Python packages your node code may import |
| **Alerts** | where failures get sent |
| **Admin** | users (superusers only) |
| **Search** | anything in this instance, by name |
| **Settings** | your account, appearance, remote access, and the assistant |
| **Home** | see the brain graph, health and recent activity |
| **Flows** | create flows and open their editors |
| **Dashboards** | build widget dashboards and pair wall panels |
| **Runs** | start batch runs and review their results |
| **Workers** | manage machines that execute nodes |
| **Secrets** | store credentials; superusers only |
| **Modules** | see or install packages used by node code |
| **Alerts** | choose where failures are reported |
| **Admin** | manage users and OAuth clients; superusers only |
| **Search** | find items by name |
| **Settings** | change your account, appearance and remote access |
**Assistant** appears when a provider is configured. **Documentation** opens
the docs in another tab.
### Search
**Search** at the foot of the sidebar, or ⌘K / Ctrl-K from anywhere, opens a
panel that finds things by name as you type: flows with the nodes inside them
and the inputs and outputs they declare, dashboards and the widgets on them,
panels, secrets, modules, workers and alert channels. Picking a node opens its
flow with that node in focus, an input or output opens its flow's settings,
where it is declared, and a widget opens its dashboard.
Use **Search**, or ⌘K / Ctrl-K, to find flows, nodes, flow inputs and outputs,
dashboards, widgets, panels, secrets, modules, workers and alert channels.
Selecting a result opens its editor or settings.
It searches this instance. Reached through a portal, other instances
are behind **All instances** at the top of the sidebar.
Search covers the current instance. When connected through a portal,
**All instances** returns to the instance list.
## Home
The one screen you leave open. Four things share it.
### The brain graph
Every flow drawn as a neuron, wired to the flows it exchanges messages with.
This is the brand mark made live, and it is also the fastest read on the
instance: a neuron pulses when its flow is running work, and its ring turns
terracotta when the flow cannot run as written. A neuron with a problem keeps
its label showing so you can see which one it is without hovering.
The graph combines nodes from published flows. Nodes talking to the same
broker topic, URL or other external target share a neuron; computation nodes
appear individually. Messages connect them, and activity makes them pulse.
Each flow also has a switch beside it in the list, which starts and stops it.
A [`wren` node](../reference/node-types.md#wren) anywhere adds a chat button
at the graph's top right, in the header on a phone, which opens the chat on
its [Control tab](../code/agents.md#control).
A problem keeps the affected neuron's label visible. A
[`wren` node](../reference/node-types.md#wren) also adds a chat button for its
[Control tab](../code/agents.md#control).
### Flows and dashboards
Two columns under the graph, exactly as tall as each other, most recently
worked on first. The flows column is the list with the switches; the dashboards
column is a mosaic, each tile a schematic of that dashboard's layout: blocks
where its widgets sit, shaded by what kind of widget each one is. Neither
column grows past about six rows: past that it scrolls in place rather than
pushing the health block down the page.
The tiles are a footprint, not a live view. They show you which dashboard is
which at a glance; the readings are on the dashboard itself.
The flow list includes switches to start and stop flows. Dashboard tiles show
a small sketch of each layout; open one for its live readings.
### Health
Always answers, degraded or not. The tiles cover:
- **Flows** — total, running, paused, quarantined, and how many cannot run
because their graph does not validate
- **Nodes** — how many failed to load
- **Runs running** — batch runs in flight right now, and how many are
waiting. Only on an instance that has run something
- **Queue** — depth, and how old the oldest pending item is
- **Loop lag** — whether the engine's event loop is keeping up
`status: degraded` comes with a list of named problems, in words. "3 flow(s)
cannot run: house, pv, hallway" is more useful than a red dot, so that is what
it says.
Health shows flow status, node load errors, queue depth and event-loop lag.
After you have used batch runs, it also shows running and queued runs.
A degraded instance lists the problems that need attention.
### Activity
Charts of executions and failures over the selected range (1h / 6h / 24h / 7d),
with the recent runs, recent failures, dead-lettered work and the audit trail
underneath.
Choose 1h, 6h, 24h or 7d to review executions and failures, recent runs,
dead-lettered work and the audit trail. Hover a chart to filter the lists to
that moment; click to hold the selection while you read.
The charts are scrubbable: hover a moment and the lists below filter to it,
click to hold it while you read. That turns "something went wrong around two
o'clock" into the actual rows.
## Managing flows and dashboards
## Flows
The lists show titles, unpublished changes and status. Use **New flow** or
**New dashboard** to create one. The Flows list offers **Publish all changes**
when drafts are waiting.
The list shows each flow's title, node count, whether it has unpublished
changes, and whether it is enabled, paused or quarantined. The toolbar searches,
creates, and offers **Publish all changes** when several flows have drafts.
Opening one takes you to [the flow editor](flow-editor.md).
### Deleting several at once
Press and hold a card, or ctrl-click it, to pick it, then tap the rest. While
anything is picked, **New flow** in the toolbar becomes a trash button, and it
asks once before deleting the lot. Unpicking the last one puts the list back;
so does Escape. The Dashboards screen works the same way.
## Everything else
- [The flow editor](flow-editor.md) — the canvas, the code editor, running and
testing
- [Dashboards and panels](dashboards.md) — widgets, bindings, and hanging a
screen on a wall
- [Secrets, modules and alerts](operations.md) — the three screens that keep an
instance running
- [Accounts and the portal](portal.md) — reaching an instance from outside
its network
To delete several items, hold a card or Ctrl-click it, then select the rest.
The create button becomes a trash button. Deletion asks for confirmation;
Escape clears the selection.
## Appearance
Light and dark follow your system by default; **Settings → Appearance**
overrides it. Both themes are first-class, and the wall-panel view is designed
to be legible in dark from across a room.
Light and dark follow your system by default. Change this under
**Settings → Appearance**. Dashboards also have their own theme settings.
## Next stops
- [The flow editor](flow-editor.md)
- [Dashboards and panels](dashboards.md)
- [Secrets, modules and alerts](operations.md)
- [Accounts and the portal](portal.md)
+65 -98
View File
@@ -1,139 +1,106 @@
# Secrets, modules and alerts
Three screens that have nothing to do with each other except that an
instance you actually depend on needs all of them.
Store credentials, manage node dependencies and choose where failures go.
These screens help keep an instance humming.
## Secrets
A node never holds a credential. A parameter written as
Open **Secrets** as a superuser, then add a name and value. The list shows
names only; saved values are not displayed again.
Reference a secret from a node setting:
```json
{"$secret": "influx-token"}
```
is replaced with the stored value when the node is built, and the editor
renders any field a node type marks as a credential as a picker over what you
have stored.
Fluksio resolves it when building the node. Credential fields in the editor
offer a secret picker. Missing names produce an error on the affected node.
**Secrets** is where the values live. Add a name and a value; the value is
never shown again, and the list only ever shows names.
Secrets are encrypted with a key derived from `SECRET_KEY` and stored outside
the flow repository. Using references keeps credentials out of saved flows.
They are encrypted at rest with a key derived from the instance's
`SECRET_KEY`, and kept **outside** the flow repository. That matters because
flows are a git repository you may well push somewhere: what gets committed and
shared never contains a password.
!!! warning "Keep your encryption key"
A name that no longer resolves is a clear error on the node that wanted it
("No secret named 'x'; add it under Secrets") rather than a mysterious
authentication failure.
!!! warning "Rotating `SECRET_KEY`"
The secrets store is encrypted with a key derived from it. Change
`SECRET_KEY` and the store no longer decrypts. Re-enter the secrets, or
plan the rotation properly.
Changing `SECRET_KEY` makes existing secrets unreadable. Keep the old key
for re-encryption, or re-enter the secrets. Source installations include
`backend/scripts/rotate_secret_key.py` for rotating the secrets store.
## Modules
Node code runs in a virtual environment of its own, on the instance's data
volume, separate from the packages Fluksio itself runs on. A pin of yours can
never shadow one of ours, and vice versa.
**Modules** shows the Python environment used by node code, its installed
packages and Python version. There are two setups:
**Modules** is a pip manifest and an **Apply** button, which only a superuser
sees: a package runs its own code on install. It is installed with
`uv pip sync`, so a line you delete is uninstalled rather than left behind.
The manifest is versioned alongside your flows, so what a deployment installed
is recorded with what uses it.
- **Managed environment:** Docker uses a separate node environment. A
superuser edits **Requirements** and presses **Apply**. Packages are synced
with `uv pip sync`; deleting a requirement removes it. The manifest is
versioned with the flows, and successful installs refresh the workers.
- **Adopted environment:** a fresh pip installation inside your own virtual
environment can use that environment for nodes too. Modules is read-only.
Install dependencies there with pip or uv, then sync or restart Fluksio.
The page shows what is currently installed, the interpreter's Python version,
and whether the manifest and the environment agree. A failed resolve comes back
as `uv`'s own output, in the pane, because that is the only thing anyone can
act on.
Set `NODE_VENV=managed` to request a separate environment. Installation
failures show the package manager's output on the page.
An install takes effect immediately: the worker processes pick up the new
interpreter state without the engine restarting.
!!! note "Big scientific stacks"
Installing torch into this venv works and is often the simplest thing. The
alternative, when you already have an environment you would rather not
duplicate, is to attach it as a worker: `fluksio-worker --python
/path/to/venv/bin/python`. See [Remote workers](../code/workers.md).
For an existing scientific environment on another machine, attach a worker
with `fluksio-worker --python /path/to/venv/bin/python`. See
[Remote workers](../code/workers.md).
## Alerts
Everything that goes wrong already travels the engine's event bus. Alerts is
where you say who hears about it.
Create a channel, choose events in a rule, and use **Test** to check delivery.
Enable alerts for the rules to send.
### Channels
| Kind | Settings |
| Kind | Destination |
|---|---|
| **ntfy** | server, topic, and a token for a protected topic |
| **SMTP** | an address to send to (the instance's mail settings do the rest) |
| **Webhook** | a URL to POST to |
| **Dashboard** | a message name a notification widget reads |
| **ntfy** | server and topic; optional token |
| **SMTP** | recipient address, using the instance's mail settings |
| **Webhook** | URL that receives a POST |
| **Dashboard** | declared message read by a Notification widget |
| **webpush** | browsers subscribed from the Alerts page |
Any channel setting may hold a `{"$secret": "name"}` reference instead of a
literal, so a webhook URL with a token in it does not sit in the config.
Channel settings may use `{"$secret": "name"}` references. Browser push also
needs notification permission and a secure browser context.
### Rules
A rule is a set of events and the channels they go to. **A rule with no events
ticked covers all of them**, including ones added in later versions, which is
usually what you want for the first rule you write.
A rule selects events and destination channels. Leaving all events unchecked
matches every supported alert event, including future additions.
| Event | Fires when |
| Event | Trigger |
|---|---|
| `node_error` | a node failed |
| `node_health` | a connection dropped |
| `flow_quarantined` | a flow crashed often enough that the engine stopped restarting it |
| `task_crashed` | a background task crashed |
| `engine_degraded` | the engine is struggling |
| `cascade_dropped` | work was given up on |
| `queue_unavailable` | the work queue is unreachable |
| `node_error` | a node fails |
| `node_health` | a connection goes down |
| `flow_quarantined` | repeated crashes stop a flow's background work |
| `task_crashed` | a background task crashes |
| `engine_degraded` | engine health degrades |
| `cascade_dropped` | the engine drops work |
| `queue_unavailable` | the work queue becomes unreachable |
**Test** sends one message through a channel so you find out it is misconfigured
now rather than at three in the morning.
Repeated faults are deduplicated with a 15-minute default cooldown.
Flapping connections are suppressed, and the engine caps alert dispatches at
ten per hour. A quiet channel may be cooling down; **Test** helps distinguish
that from a broken configuration.
### What it mostly does is not send
## Admin: users and clients
This is the part worth knowing before you decide it is broken:
Superusers use **Admin** to add, edit and remove accounts, and manage OAuth
clients for [agents](../code/agents.md).
- the same fault repeating is **one** alert, with a cooldown (15 minutes by
default per rule)
- a connection flapping is one story, not one alert per transition
- however bad it gets, **ten alerts an hour** is the ceiling
An engine that is genuinely on fire produces a handful of messages, not
thirty-six thousand.
## Admin: users
Superusers get an **Admin** screen: add, edit and remove accounts, and see who
is a superuser.
Accounts that arrived through a portal are badged *Portal*. They have no
password (they sign in at the portal, not here) and they are never
superusers, so they read node code but cannot change it. Deleting one cuts their access immediately and independently of the
portal, so it works even if the portal cannot be reached. See
[Accounts and the portal](portal.md).
The same screen manages OAuth clients, which is how [agents](../code/agents.md)
are registered.
Accounts admitted from a portal are marked *Portal*. They sign in through the
portal, have no local password and are not superusers. To block local access,
edit the account and clear **Is active?**. See [Accounts and the portal](portal.md).
## Settings
Your own account: name, email, password, and appearance (light, dark, or
follow the system).
Change your name, email, password and appearance under **Settings**.
**Remote access** is where an instance is paired with a portal, remote
users are admitted, and the link is cut again. That has [its own
page](portal.md).
**Remote access** manages enrollment and admitted users; see
[Accounts and the portal](portal.md).
**Assistant** turns on the chat that builds and edits flows: pick a provider,
paste a key, fetch the model list and choose one. The key is stored encrypted
on the instance and never shown again. One provider serves everyone who signs
in here, and until one is set nothing in the interface mentions the feature.
How the chat works, and what it may do, is on the [agents
page](../code/agents.md#the-assistant).
A superuser can configure **Assistant**: choose a provider, store its key,
fetch models and select one. The encrypted key is shared for assistant access
on this instance. The chat appears after configuration; see
[The assistant](../code/agents.md#the-assistant).
+59 -115
View File
@@ -1,142 +1,86 @@
# Accounts and the portal
Fluksio is self-hosted by default. An instance runs offline, keeps its data
on its own disk, and never contacts anything unless you tell it to.
The optional portal lets you open an instance from [fluksio.com](https://fluksio.com),
including a pip installation with no local dashboard. The instance connects
outward over a WebSocket, so you do not need to open an inbound port.
The **portal** is optional, and it exists to solve two specific problems:
Flows still run on your instance. The portal serves the interface and carries
API requests, responses and live updates through the connection.
1. **Your machine has no inbound route.** A homelab behind CGNAT, a cluster
node with no open ports, a laptop. Opening one is work, and often not
allowed.
2. **You want a browser on it anyway.** A `pip install fluksio` instance
has no web server for the dashboard at all.
## Connect an instance
An enrolled instance dials *out* to the portal and holds one websocket
open. The portal serves the dashboard from its own side, and only the API calls
travel down the tunnel, so the interface loads at portal speed and your
machine stays unreachable from the internet.
1. Sign up or sign in at **fluksio.com**.
2. Open **Instances → Add instance** and copy the code.
3. On the machine running Fluksio, open another terminal in the same directory
as `fluksio serve` and run:
## Enrolling
```sh
fluksio enroll <code>
```
Two halves: whoever performs the second step decides what the instance's owner
gets.
The CLI uses `https://hub.fluksio.com` by default. If you started the engine
with `--data-dir` or `--global`, use the same option when enrolling. A running
engine picks up the connection automatically; otherwise start it with
`fluksio serve`.
**On the portal** (`hub.${DOMAIN}`, or [fluksio.com](https://fluksio.com) for
the hosted one): **Instances → Add instance**, give it a name, and copy
the code.
From a local dashboard, you can instead use **Settings → Remote access**:
leave the default portal URL, paste the code, and press **Connect**.
Enrollment requires a superuser account.
**On the instance**, either from the dashboard:
> **Settings → Remote access**, enter the portal URL and the code, press
> **Connect**.
or from the command line, which is the path for an instance with no web
interface of its own:
The portal owner's sessions use the local account that enrolled the instance.
Check **Acting as** under Remote access. With several local superusers, the
CLI can choose one explicitly:
```sh
fluksio enroll ABCD-1234 --portal https://hub.fluksio.com
fluksio enroll <code> --as someone@example.com
```
The code is single-use and expires in fifteen minutes. A fresh code on an
instance that is already paired replaces the pairing, so moving an instance to
another portal is one command rather than a disconnect and a reconnect.
A new valid code replaces an existing connection. An expired or invalid code
leaves the current connection intact. Once connected, open the instance from
the portal's **Instances** list.
A portal session then arrives as *that local account*, which the settings screen
states plainly. Use `--as someone@example.com` to enrol as a specific local
account when the instance has several superusers.
## Let someone else in
Once enrolled, the instance appears under **Instances** with its
status, when it was last seen and its version. **Open** takes you to its
dashboard at `${DOMAIN}/i/{instance-id}`.
Each person gets an ordinary local account, with no superuser rights. You can
admit them in either of these ways:
**Upgrade a self-hosted portal before its instances.** An instance dials the
tunnel with its credential in an `Authorization` header, where it stays out of
access logs, and a portal from before that reads it only from the URL. The
portal still takes the old `?token=` for one release, so older instances keep
attaching meanwhile.
- Share an invitation link from the portal's **Instances** list.
- Ask them for a code from **Instances → Join an instance**. Enter it on your
local instance under **Settings → Remote access → Add remote user**.
## What the portal can and cannot do
Their account appears under **Admin → Users** with a *Portal* badge. They
sign in through the portal rather than with a local password. Invitations are
time-limited; generate a fresh one if it expires.
The portal holds one credential for your instance and proxies requests down
the tunnel. What those requests may do is decided **on the instance**, by
the same checks a local session passes.
## Manage or end remote access
The trust anchor is a signing keypair on the portal. Every instance pins
its public half at enrolment and rejects anything else, which is what stops a
hijacked DNS entry or a mis-issued certificate from impersonating the portal.
**Settings → Remote access** shows the connection status and the latest
connection error. The instance needs outbound access to the portal, and any
proxy on that route must support WebSockets.
## Letting someone else in
- To remove a person's access, remove them from the portal's sharing dialog
or edit their account under **Admin → Users** and clear **Is active?**.
- To end all portal access from the instance, press **Disconnect**. This stops
the connection and invalidates portal credentials locally, without requiring
the portal to be online.
- To replace the pairing, use **Connect with a new code**. Remote users and
panels may need to pair again.
Anyone else on the portal reaches your instance only if you let them, and they
arrive as a local user of their own rather than as you. There are two ways,
and the difference is who starts.
Local sign-in and flows continue after disconnecting. Portal requests follow
the instance's normal account permissions, and the instance verifies portal
tokens against the public signing key pinned during enrollment.
### Share a link
## Custom portals
On the portal, the share icon on your instance's row gives a QR code and a URL.
Whoever opens it, signed in to Fluksio on that device, joins. It works once and
expires in fifteen minutes; **New link** replaces it, which is also how a link
you would rather not have sent is cancelled.
The CLI accepts another compatible portal with `--portal https://<portal-host>`;
the dashboard has the same URL field. HTTPS is required except for a portal on
loopback.
Nothing has to be entered on the instance. It learns of the new member when
they first arrive and asks the portal who they are, then gives them their local
account.
### Take a code
For a machine whose owner would rather admit people at the machine itself.
1. **They**: **Instances → Join an instance**, and copy the code. It is
bound to their portal account and expires in fifteen minutes.
2. **You**, on the instance: **Settings → Remote access → Add remote
user**, and enter the code.
Either way they now see the instance under **Instances**, marked *Shared by*,
with **Open** and nothing else. Renaming, re-keying and removing stay with you.
The instance redeems the code against the portal using its own credential.
A portal session cannot do this, which is what stops somebody you let in from
letting others in.
On the instance they appear under **Admin → Users**, badged *Portal*, never
a superuser and with no password.
## Cutting it off
| From | Action | Effect |
|---|---|---|
| The portal | share dialog → **Remove** | that member stops reaching it from the portal; their user on the machine stays until it is deleted there |
| The portal | **Leave**, on a row shared with you | gives up your own access to somebody else's instance |
| The portal | **New code** | rotates the credential and drops the current link |
| The portal | **Remove** | deletes the registration and cuts the connection |
| The instance | **Disconnect** | unilateral and immediate — the portal's tokens stop verifying here whatever the portal still has on file |
| The instance | delete a user under **Admin → Users** | that one person, immediately, independent of the portal |
**New code** also cuts every credential the portal minted for this instance,
including [wall panels paired through it](dashboards.md#a-screen-somewhere-you-cannot-reach).
The instance's own **Disconnect** is the one to reach for if you are ever
unsure: it does not need the portal's cooperation.
## Running your own portal
The portal is the `index` stack's `hub` service: accounts, the registry of
connected instances, and the websocket each one dials in on. Two things
about it are load-bearing:
- **It runs a single process.** It keeps its attached instances in the
memory of the process holding their sockets, so a second worker would answer
for links it does not hold. Scaling out needs a routing layer first.
- **Back up the signing keypair with the database.** Replacing it forces every
instance to be enrolled again.
Websocket upgrades must be enabled on the `hub` hostname in whatever proxy
fronts it. Without them every instance sits in a reconnect loop and the
portal shows them all offline.
This public repository contains the instance and its connector. The portal
service is not included, so the Docker stack does not start one.
## See also
- [The `fluksio` command](../code/cli.md) — `fluksio enroll`, and what it writes
- [Dashboards and panels](dashboards.md) — pairing a wall screen through the portal
- [Getting started: data science](../getting-started/data-science.md) — the
shortest path from a pip install to a browser
- [The fluksio command](../code/cli.md): data directories and enrollment options.
- [Dashboards and panels](dashboards.md): pair a wall screen through the portal.
- [Getting started](../getting-started/index.md): install and open your first instance.
+12 -9
View File
@@ -1,24 +1,25 @@
# Fluksio
> Fluksio is a node-based automation engine. A flow is a graph of small
> functions wired by message name, kept running by one resident engine. The
> same engine serves two shapes of work: live flows that never end (buildings,
> labs, homelabs) and batch runs that finish and leave a record (experiments,
> ML pipelines, CI-style jobs). Self-hosted by default; an optional portal
> exists only to reach an instance that has no inbound route.
> Fluksio is a node-based automation engine for live flows and batch runs.
> Nodes connect through typed, named messages. Start with pip for data science
> or Docker for facility automation; connect to a portal for optional remote
> access. Data stays on the instance.
## Getting started
- [Pick your starting point](/getting-started/): the two setup paths and how to choose
- [Data science](/getting-started/data-science/): pip install, wrap a training script, runs and sweeps
- [Facility automation](/getting-started/facility-automation/): Docker stack, MQTT to dashboard, InfluxDB, workers
- [Pick your starting point](/getting-started/): choose data science or facility automation
- [Data science](/getting-started/data-science/): pip setup, a Python function, and a first run
- [Facility automation](/getting-started/facility-automation/): Docker setup, a live flow, dashboards, and devices
## Concepts
- [Flows, nodes and messages](/concepts/flows/): the graph comes from message names, not drawn wires
- [Where a node's values come from](/concepts/values/): ports, settings, flow inputs
- [Keeping state in a flow](/concepts/state/): a message a node both reads and writes
- [Runs: pipelines that finish](/concepts/runs/): batch mode, metrics, artifacts, sweeps, durability
## The interface
- [The dashboard app](/interface/): the shell, Home, health and activity
- [The flow editor](/interface/flow-editor/): canvas, node panel, dock, shortcuts, drafts
- [Dashboards and panels](/interface/dashboards/): widgets, typed bindings, wall screens
@@ -26,6 +27,7 @@
- [Accounts and the portal](/interface/portal/): enrolment, remote users, revoking
## Code and the CLI
- [The fluksio command](/code/cli/): serve, enroll, worker, and the data directory
- [Writing node code](/code/nodes/): process(), generators, fluksio.emit, artifacts
- [The HTTP API](/code/api/): auth, flows, messages, runs, observability, websocket
@@ -34,6 +36,7 @@
- [Writing a connector](/code/connectors/): packaging a device integration as a node type
## Reference
- [Node types](/reference/node-types/): every built-in type and its settings
- [Payload types](/reference/payload-types/): float, int, str, bool, json, record, list, series, artifact
- [Configuration](/reference/configuration/): every environment variable
+128 -179
View File
@@ -1,238 +1,187 @@
# Configuration
Every setting comes from the environment, or from an env file. Which file
depends on how the instance was started:
Most installations can keep the defaults. Settings come from environment
variables and an env file:
| Started with | Reads |
| How you start Fluksio | Env file |
|---|---|
| `fluksio serve` | `env` inside the data directory (`$FLUKSIO_ENV_FILE`) |
| the Docker stack | `.env` beside `docker/` |
| `fluksio serve` | `<data-dir>/env`, unless `FLUKSIO_ENV_FILE` names another file |
| Docker with the Makefile | `.env` in this repository's root |
Anything already exported wins over the file.
Exported variables override the file. `fluksio serve` flags override both.
For CLI installations, select storage with `--data-dir`, `FLUKSIO_HOME` or
`--global`; see [The fluksio command](../code/cli.md).
## Storage
| Variable | Default | Notes |
| Variable | Default | Purpose |
|---|---|---|
| `DATA_DIR` | `flow-data` (`./.fluksio` via the CLI; `~/.fluksio` with `--global`) | everything below derives from this |
| `DATABASE_URL` | SQLite in `DATA_DIR` | any SQLAlchemy URL |
| `FLOWS_DIR` | `$DATA_DIR/flows` | the git repository holding flows |
| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials, kept outside the repo |
| `ALERTS_FILE` | `$DATA_DIR/alerts.json` | channels and rules |
| `PROVISIONERS_FILE` | `$DATA_DIR/provisioners.json` | clusters a machine can be started from; absent means none |
| `DATA_DIR` | `flow-data`; CLI uses `./.fluksio` or `~/.fluksio` with `--global` | base directory for instance data |
| `DATABASE_URL` | SQLite at `$DATA_DIR/fluksio.db` | SQLite URL; other database engines are unsupported |
| `FLOWS_DIR` | `$DATA_DIR/flows` | flow git repository |
| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials |
| `ALERTS_FILE` | `$DATA_DIR/alerts.json` | alert channels and rules |
| `WEBPUSH_FILE` | `$DATA_DIR/webpush.json` | push keys and browser subscriptions |
| `PROVISIONERS_FILE` | `$DATA_DIR/provisioners.json` | optional worker provisioning configuration |
| `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings |
| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens |
| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any |
| `NODE_VENV` | `auto` | which interpreter node code runs on — see below |
| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | agent and worker signing key |
| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | optional portal enrollment |
| `AI_CONFIG_FILE` | `$DATA_DIR/ai.json` | optional assistant provider configuration |
| `CHATS_DIR` | `$DATA_DIR/chats` | assistant transcripts |
| `NODE_VENV` | `auto` | interpreter for node code |
Set `DATA_DIR` and the rest follow. Set one explicitly and it wins, which is
what the container images do to pin everything onto `/data`.
Changing `DATA_DIR` moves the derived paths unless you set them individually.
Docker stores instance data under `/data` on the `app-flow-data` volume.
`NODE_VENV` is the exception, being about an environment rather than a path:
| Value | What node code runs on |
| `NODE_VENV` value | Behavior |
|---|---|
| `auto` (default) | the venv Fluksio was installed into, when it was installed into one and there is no venv of its own already built. `pip install fluksio` beside your own packages is this case, and the packages are then already there — the Modules screen turns read-only, because that environment is not Fluksio's to install into. |
| `managed` | a venv the engine builds under `DATA_DIR` and owns, which the Modules screen installs into with `uv pip sync`. The container images set this: the venv in them holds the app and nothing of anybody else's. |
| a path | that interpreter, or that venv, whatever it is. |
| `auto` | keeps an existing managed venv; otherwise uses the installation's venv when available, or creates a managed one |
| `managed` | creates a venv under `DATA_DIR`; the Modules screen manages its packages |
| interpreter or venv path | uses the specified Python environment |
An instance that already has a managed venv keeps it on upgrade under
`auto`, because it may hold packages somebody installed on purpose.
Docker uses `managed`. When Fluksio adopts your own venv, the Modules screen
is read-only; install packages into that environment yourself.
!!! warning "The four files that must be on persistent storage"
Back up the whole data directory and the persistent `SECRET_KEY`. In Docker,
keep `.env` and `app-flow-data`; back up `app-redis-data` too when preserving
live state. Compose prefixes volume names with the project name `fluksio-app`.
Keep runtime files on persistent storage so rebuilds preserve credentials,
pairings and subscriptions.
`secrets.enc`, `alerts.json`, `panels.json` and `oauth-key.pem` are written
at runtime. In a container, anything not on a volume lands in the writable
layer and is lost on the next rebuild, un-pairing every screen and
revoking every agent.
## Live state
## State
| Variable | Default | Notes |
| Variable | Default | Purpose |
|---|---|---|
| `REDIS_HOST` | unset | without it, flow state lives in memory and does not survive a restart |
| `REDIS_PORT` | `6379` | |
| `REDIS_HOST` | unset | unset uses memory; a Redis host enables persistent flow state |
| `REDIS_PORT` | `6379` | Redis port |
Flow state is the last value of every message, node memory, and the run queue.
Redis here is persistence, not a cache. Run it with append-only persistence on
(`--appendonly yes`, which the compose stack sets).
Live state includes message values, node memory and queued work. In-memory
state is lost on restart. The Docker stack uses Redis with append-only
persistence on its own `app-redis-data` volume. Its default sync policy may
lose about a second of recent writes after an unclean shutdown.
That writes with Redis's default `appendfsync everysec`, so an unclean stop of
Redis or of its host loses up to a second of queued work. `appendfsync always`
closes the window and cuts engine throughput to about a third, so the default
stands.
## Accounts and access
## Identity and access
| Variable | Default | Notes |
| Variable | Default | Purpose |
|---|---|---|
| `SECRET_KEY` | generated | signs sessions and derives the secrets-store key |
| `ACCESS_TOKEN_EXPIRE_MINUTES` | `11520` (8 days) | |
| `FIRST_SUPERUSER` | — | absent means the CLI creates one on first run |
| `FIRST_SUPERUSER_PASSWORD` | — | absent means one is generated and printed once |
| `DOMAIN` | `localhost` | what the API and OAuth issuer are built from |
| `FRONTEND_HOST` | `http://localhost:5173` | used in mails, OAuth metadata and panel pairing links |
| `BACKEND_CORS_ORIGINS` | `[]` | comma-separated; `FRONTEND_HOST` is always allowed |
| `FORWARDED_ALLOW_IPS` | `127.0.0.1` (`127.0.0.1,172.16.0.0/12` in the compose stack) | proxies whose `X-Forwarded-For` is believed, as uvicorn reads it; the sign-in, sign-up and OAuth limits count the rightmost hop none of them added |
| `SECRET_KEY` | generated | signs sessions and encrypts the secrets store; the CLI persists it and `make init` writes it to `.env` |
| `ACCESS_TOKEN_EXPIRE_MINUTES` | `11520` (8 days) | session-token lifetime |
| `FIRST_SUPERUSER` | unset | initial administrator email; the CLI creates an account if needed |
| `FIRST_SUPERUSER_PASSWORD` | unset | initial password; the CLI generates and prints one if needed |
| `DOMAIN` | `localhost` | API hostname and OAuth issuer; Docker also uses it for routing |
| `FRONTEND_HOST` | `http://localhost:5173` | interface URL for links and metadata |
| `BACKEND_CORS_ORIGINS` | `[]` | allowed origins, as comma-separated URLs or a JSON list; `FRONTEND_HOST` is also allowed |
| `FORWARDED_ALLOW_IPS` | `127.0.0.1` | trusted proxies; Docker adds `172.16.0.0/12` |
| `SIGNUP_ENABLED` | `false` | allows account creation from the login page; otherwise use **Admin → Users** |
## Where the interface looks for the API
!!! warning "Keep your secret key"
The dashboard is a static bundle, so this one is a **build** argument of the
`frontend` image rather than a setting the running stack reads.
Changing `SECRET_KEY` signs everyone out and makes existing encrypted
secrets unreadable. Back it up with your data.
| Argument | Used by | Effect |
|---|---|---|
| `VITE_API_URL` | `frontend` at build time | the address the interface calls |
## Interface API address
The compose stack takes it from `.env`, falling back to `https://api.${DOMAIN}`.
`scripts/setup.sh` writes it there with the scheme `ENVIRONMENT` implies, so a
local build calls `http://` and does not fail a certificate check nothing is
there to satisfy.
`VITE_API_URL` is a frontend **build argument**, so changing it requires
rebuilding the frontend image.
Empty is the useful value: the interface then addresses the API relative to
whichever origin served the page, so one image answers on a hostname, on a
`http://<host-ip>:<port>`, and through an ssh tunnel alike, and no origin has
to be added to `BACKEND_CORS_ORIGINS`, because there is only one.
`docker/compose.lan.yml` builds it that way and puts an `/api` proxy in front
of the backend to complete it; see
[getting started](../getting-started/facility-automation.md#on-your-own-network-by-address).
| Setup | Value |
|---|---|
| Production Docker | set `.env` to `https://api.<your-domain>` |
| `make dev-local` | overlay sets `http://api.localhost` by default |
| `make dev-lan` | overlay sets an empty value, using `/api` on the page's own origin |
Set it to an absolute URL only when the API genuinely lives somewhere else, and
remember it is fixed at build time: changing it means rebuilding that image.
Without an explicit value, the production Compose file falls back to
`https://api.${DOMAIN}`. Set `FRONTEND_HOST` and `BACKEND_CORS_ORIGINS` to
match your deployment as well; `make init` generates secrets, not deployment
URLs.
!!! danger "Rotating `SECRET_KEY`"
The secrets store is encrypted with a key derived from it. Change it and
the store stops decrypting, and every session is signed out. Re-enter your
secrets, or plan the rotation properly.
The LAN overlay also configures nginx to proxy `/api` and WebSocket requests
to the backend. An empty value needs that proxy when the frontend and backend
are served separately. See [facility automation](../getting-started/facility-automation.md#run-it-long-term)
for deployment settings.
## Environment
| Variable | Default | Notes |
| Variable | Default | Purpose |
|---|---|---|
| `ENVIRONMENT` | `local` | `local`, `staging` or `production` |
| `PRIVATE_API_ENABLED` | `false` | unauthenticated test-only endpoints; needs `ENVIRONMENT=local` too |
| `API_DOCS_ENABLED` | `false` | serves `/docs`, `/redoc` and the OpenAPI document to any caller; otherwise only to one on this machine. Never in `production` |
| `SIGNUP_ENABLED` | `false` | lets anyone who reaches the login page create an account; otherwise a superuser adds them under **Admin** |
| `TZ` | `UTC` | the timezone every schedule is written in |
| `PRIVATE_API_ENABLED` | `false` | test endpoints; also requires `ENVIRONMENT=local` |
| `API_DOCS_ENABLED` | `false` | opens `/docs`, `/redoc` and `/api/v1/openapi.json` to network callers outside production |
| `TZ` | host timezone; Docker defaults to `UTC` | timezone used by cron schedules |
`TZ` is the container's own, not a setting the code reads: an `inject` or a
`delay` with a cron expression fires on local time. Left at `UTC`, "off at
02:00" means two in the morning UTC, which in most of the world is neither two
o'clock nor the same hour in summer as in winter. Set it to where the
instance is.
API documentation is available to loopback callers outside production.
`API_DOCS_ENABLED=true` opens it to other callers, as the Docker development
stack does. Production disables it entirely. Staging and production also
refuse `changethis` values for the session key or administrator password.
`production` closes `/docs`, `/redoc` and the OpenAPI document, because the
schema enumerates every endpoint the instance serves, including the paths
webhook nodes mounted at runtime. Elsewhere they answer a caller on the
engine's own machine, so `fluksio serve --host 0.0.0.0` does not hand them to
the network; `API_DOCS_ENABLED` opens them to everyone, as the dev stack does. It also turns a `changethis` secret from a
warning into a refusal to start.
Set `TZ` in `.env` for schedules that should follow local time, such as
`Europe/Berlin`.
## The engine
## Engine and resources
| Variable | Default | Notes |
| Variable | Default | Purpose |
|---|---|---|
| `FLOW_MAX_WORKERS` | `4` | node-code subprocesses run in parallel |
| `FLOW_MAX_CASCADES` | `4` | cascades in flight at once; throughput is this over the mean cascade time, so raise it where nodes wait on a network rather than a CPU |
| `FLOW_MAX_RUNS` | `4` | batch runs driven at once. A different limit from the one above: a run drives a whole graph, and its nodes are bounded by `FLOW_MAX_WORKERS`. This is what a sweep queues behind |
| `FLOW_NODE_TIMEOUT` | `0` | seconds a node may be silent, unless it sets its own; 0 is no limit |
| `FLOW_CPUS` | `0` | cores nodes that declare `resources` may be given; 0 works it out as every core but two, which are what keeps the engine answering while the machine is busy |
| `FLOW_GPUS` | `0` | GPUs on this machine, each held by one node at a time. 0 counts NVIDIA's device nodes; say a number for anything they miss |
| `FLOW_RAM` | `0` | megabytes nodes that declare `ram` may be given; 0 works it out as the machine's memory, or the container's limit where that is lower, less a gigabyte for the engine |
| `FLOW_WORKER_MAX_RSS_MB` | `0` | a warm worker holding more resident memory than this when its node returns is retired rather than kept; 0 keeps every worker warm |
| `OBS_RETENTION_DAYS` | `30` | how long metrics, events and run records are kept |
| `ARTIFACT_GC_INTERVAL_S` | `3600` | how often artifact bytes nothing refers to are swept away; 0 never sweeps |
| `ARTIFACT_GC_GRACE_S` | `3600` | how long a freshly written artifact is spared, whatever refers to it |
| `MAX_ARTIFACT_BYTES` | `2147483648` | the largest body `PUT /artifacts` will take; 0 removes the limit |
| `ARTIFACT_VOLATILE_DIR` | worked out | where frames a flow only shows live are held; empty picks a directory under `/dev/shm` named for the data directory, and falls back to the temporary directory |
| `ARTIFACT_VOLATILE_BYTES` | `50331648` | how much that ring holds before the oldest frames fall out; 0 turns it off and volatile saves land in the store |
| `FLOW_MAX_WORKERS` | `4` | local node-code subprocesses |
| `FLOW_MAX_CASCADES` | `4` | concurrent live message cascades |
| `FLOW_MAX_RUNS` | `4` | concurrent batch runs |
| `FLOW_NODE_TIMEOUT` | `0` | maximum node silence in seconds, unless the node overrides it; `0` disables |
| `FLOW_CPUS` | `0` | CPU budget; `0` detects cores and reserves two for the engine |
| `FLOW_GPUS` | `0` | local GPU count; `0` detects NVIDIA device nodes |
| `FLOW_RAM` | `0` | memory budget in MB; `0` detects host/container memory and reserves 1 GiB |
| `FLOW_WORKER_MAX_RSS_MB` | `0` | retire a local worker above this memory use after a call; `0` disables |
| `OBS_RETENTION_DAYS` | `30` | retention for metrics, events and run records |
The three concurrency limits are also flags on `fluksio serve`
(`--max-workers`, `--max-cascades`, `--max-runs`), as are the card count and
memory, `--gpus` and `--ram`, and the worker ceiling, `--worker-max-rss`. The flags
outrank the file, and the engine says which numbers it started with in its
first lines. Each pool size must be at least 1 and the others at least 0:
a number below that is refused as a flag error naming it, rather than read as
the default. Leave one empty (or unset) to get the default.
The three concurrency limits must be at least `1`. They also have CLI flags:
`--max-workers`, `--max-cascades` and `--max-runs`. Resource flags are `--gpus`,
`--ram` and `--worker-max-rss`.
An artifact is referred to by a run that recorded it or by a message currently
holding it; anything else is what a camera published four hours ago, and the
sweep is what keeps a flow streaming media from filling the disk. It stands
aside entirely while a run is in flight, since a node may store a checkpoint
long before it returns the reference to it.
Nodes with declared resources are scheduled against the available budget.
Undeclared nodes use the shared pool and receive thread caps. Attached workers
report their own inventory; the engine's resource settings describe only its
own machine. See [Remote workers](../code/workers.md) and the **Workers** screen.
Frames saved with `volatile=True` skip all of that. They go to a ring in memory
instead of the volume, the oldest falling out once the newest need the room,
and the engine pushes them down the websocket to whichever screens are drawing
them — which is what a camera at ten frames a second needs and the store cannot
give it. A frame a run *records* is copied into the store on the way, so
returned media is kept and emitted media is not. Under Docker the ring lives in
the container's `/dev/shm`, whose default is 64 MB: raise `shm_size` alongside
`ARTIFACT_VOLATILE_BYTES`.
## Artifacts
A node that declares nothing is not accounted against `FLOW_CPUS`; it runs on
the shared pool and is given `FLOW_CPUS / FLOW_MAX_WORKERS` as a thread cap, so
several at once cannot each size themselves to the whole machine. Setting
`OMP_NUM_THREADS` (or any of its siblings) on the engine yourself overrides
that default.
| Variable | Default | Purpose |
|---|---|---|
| `MAX_ARTIFACT_BYTES` | `2147483648` (2 GiB) | upload limit; `0` removes it |
| `ARTIFACT_GC_INTERVAL_S` | `3600` | cleanup interval; `0` disables |
| `ARTIFACT_GC_GRACE_S` | `3600` | minimum age before cleanup |
| `ARTIFACT_VOLATILE_DIR` | auto-detected | live-media directory under `/dev/shm`, or the temporary directory |
| `ARTIFACT_VOLATILE_BYTES` | `50331648` (48 MiB) | live-media ring size; `0` stores volatile saves durably |
Workers are kept warm between calls, and a library like JAX keeps what it
allocated for as long as its process lives — so a worker that fitted a model
can sit on gigabytes after its run is over, on the shared pool as much as on a
declared one. `FLOW_WORKER_MAX_RSS_MB` retires such a worker when its node
returns, and the next call pays its imports again. That bounds idle memory
only; what a node holds while it runs is accounted, and only if it declared
`ram`.
`FLOW_CPUS`, `FLOW_GPUS` and `FLOW_RAM` are this machine's figures. An attached worker reports its own when
it dials in, and a node goes to whichever machine can grant what it asked for,
so a GPU on a worker needs no `FLOW_GPUS` here. What every machine has free, and
which nodes are queued, is `GET /api/v1/workers/resources` and the Workers
screen. Named sizes live in the database and are `GET /api/v1/flavors`;
`PROVISIONERS_FILE` is where machines can be started from. See
[declaring resources](../getting-started/data-science.md#declaring-what-a-node-needs).
Cleanup removes durable artifacts no longer referenced by recorded runs or
current messages, and pauses while a run is active. Volatile artifacts use a
bounded ring; the oldest disappear as it fills. Recorded run outputs are
copied into durable storage. If you increase the ring size in Docker, increase
`shm_size` too.
## Agents
| Variable | Default | Notes |
| Variable | Default | Purpose |
|---|---|---|
| `MCP_ENABLED` | `false` | opens the `/mcp` endpoint **and** OAuth client registration |
| `MCP_TOKEN_EXPIRE_MINUTES` | `60` | an agent's token is a bearer secret held by a program |
| `MCP_REFRESH_EXPIRE_DAYS` | `30` | |
| `OAUTH_CODE_EXPIRE_SECONDS` | `60` | |
| `MCP_ENABLED` | `false` | enables `/mcp` and OAuth client registration; `.env.example` sets it to `true` |
| `MCP_TOKEN_EXPIRE_MINUTES` | `60` | access-token lifetime |
| `MCP_REFRESH_EXPIRE_DAYS` | `30` | refresh-token lifetime |
| `OAUTH_CODE_EXPIRE_SECONDS` | `60` | authorization-code lifetime |
See [Agents over MCP](../code/agents.md).
## Mail
## Mail and monitoring
Needed for password-reset mails. Without `SMTP_HOST` and `EMAILS_FROM_EMAIL`,
mail is simply off.
Password-reset mail requires `SMTP_HOST` and `EMAILS_FROM_EMAIL`.
| Variable | Default |
|---|---|
| `SMTP_HOST` | — |
| `SMTP_HOST` | unset |
| `SMTP_PORT` | `587` |
| `SMTP_USER` / `SMTP_PASSWORD` | — |
| `SMTP_USER` / `SMTP_PASSWORD` | unset |
| `SMTP_TLS` / `SMTP_SSL` | `true` / `false` |
| `EMAILS_FROM_EMAIL` | — |
| `EMAILS_FROM_NAME` | `Fluksio` |
| `EMAILS_FROM_EMAIL` | unset |
| `EMAILS_FROM_NAME` | `PROJECT_NAME`, normally `Fluksio` |
| `EMAIL_RESET_TOKEN_EXPIRE_HOURS` | `48` |
| `SENTRY_DSN` | unset; enables error reporting when configured |
## Monitoring
| Variable | Default | Notes |
|---|---|---|
| `SENTRY_DSN` | — | error reporting, if you want it |
## Health check
`GET /api/v1/utils/health/` is a *deep* check: it fails when the event loop is
wedged or the state backend is gone, not just when the process is up. That is
what the container healthcheck probes, and what an autoheal sidecar restarts
on.
## See also
- [The `fluksio` command](../code/cli.md) — what the data directory holds
- [Getting started: facility automation](../getting-started/facility-automation.md)
`GET /api/v1/utils/health/` checks the event loop and state backend. Docker
uses it as a health check; `make up` also starts the autoheal service to restart
an unhealthy backend.
+133 -208
View File
@@ -1,180 +1,130 @@
# The connector contract
A connector is the device-facing node class. It talks to something outside the
engine (a device, a service, a protocol) and publishes what it finds as
ordinary typed messages, so the rest of a flow cannot tell the difference
between a reading from a heat pump and one from a `print` statement.
This document is normative: everything here is what the engine relies on and
what a connector may rely on in return. The authoring walkthrough is in
[Writing a connector](../code/connectors.md).
Connectors integrate devices and services through typed messages.
This reference covers the contract the engine supports; start with
[Writing a connector](../code/connectors.md) for a working package.
## Contract version
```python
from fluksio.flow.connector import CONTRACT_VERSION, ConnectorNode
class MyConnector(ConnectorNode):
contract = CONTRACT_VERSION
title = "My connector"
description = "Reads a device."
```
`contract` must be declared **on the connector class itself**. The loader reads
it from the class `__dict__` rather than through inheritance, so a connector
written against an older base cannot pass itself off as current. A mismatch is
logged and the connector is skipped; nothing else in the engine changes.
Declare `contract` on the class itself: inheritance does not count.
The current contract version is `1`. A mismatch is logged and the connector
is skipped. Breaking changes bump the version; compatible additions do not.
The version is bumped only when a change would break connectors written against
the old surface. Additions that leave existing connectors working do not bump
it.
`title` defaults to the entry point name; `description` may be empty.
## Packaging and discovery
A connector is an ordinary Python package. It advertises its node class through
an entry point in the `fluksio.node_types` group:
Register the class in your package's `pyproject.toml`:
```toml
[project]
name = "fluksio-connector-skeleton"
version = "0.1.0"
[project.entry-points."fluksio.node_types"]
random_sensor = "fluksio_connector_skeleton:RandomSensor"
my_device = "fluksio_connector_mydevice:MyConnector"
```
- The **entry point name** is the node type as it appears on the canvas and in
a stored flow. It must not collide with a built-in type (`python`, `mqtt`,
`http`, `influxdb`, `delay`, `mlp`); a collision is logged and skipped.
- The **package name and version** are the manifest. There is no second
metadata file to keep in step, and the editor shows the provenance
(`fluksio-connector-skeleton 0.1.0`) on the node type.
The entry point name is the stored node type. Names already registered,
including built-in types, are skipped. The editor shows the package name
and version from its installed metadata.
Discovery happens once, when the engine starts. A connector's code is imported,
and Python does not re-import a changed module, so installing or upgrading one
means restarting the engine.
## Declaring what a connector is
```python
class RandomSensor(ConnectorNode):
contract = CONTRACT_VERSION
title = "Random sensor" # shown in the add-node palette
description = "Emits a random reading."
```
`title` falls back to the entry point name; `description` may be empty.
Discovery runs at engine startup. Install connectors in the engine's
environment and restart after installation or code changes.
## Parameters
Settings are a pydantic model, and the editor builds a form from its JSON
schema. Subclass the base so `poll_interval` stays available:
Subclass `ConnectorNode.Params` to keep `poll_interval` and add your settings:
```python
from pydantic import Field
class Params(ConnectorNode.Params):
host: str = "localhost"
api_key: str | None = Field(default=None, json_schema_extra={"x-secret": True})
api_key: str | None = Field(
default=None, json_schema_extra={"x-secret": True}
)
```
- Booleans render as switches, numbers and strings as inputs. Objects and
arrays are not rendered, so keep settings flat.
- A field carrying **`x-secret`** renders as a picker over the stored secrets
and writes a reference, `{"$secret": "name"}`, rather than the value. The
engine resolves it when the node is built, so a credential never lands in
`flow.json`. Always mark credentials this way.
- The parsed model is available as `self.config`.
The node panel builds a form from this model's JSON schema: switches for
booleans, inputs for numbers and strings, and JSON fields for objects and
arrays. Parsed settings are available as `self.config`.
Mark credentials with `x-secret`. The editor stores a reference such as
`{"$secret": "device-key"}`, and the engine resolves it while building the node.
## Ports and messages
Ports are declared by whoever places the node, not by the connector, and they
are `MessageSpec`s like any other node's: a message name, a `DType`, and an
optional `interval`.
The flow author declares ports as `MessageSpec` values. Each has a message
name, a `DType`, and optional delivery settings.
- `poll()` returns a dict **keyed by output port**, and the base class maps it
onto message names; `write()` receives a dict **keyed by input port** the
same way.
- Every value is checked against the port's declared `DType` on the way out. A
wrong type is an error, not a hint.
- `interval` on a port is enforced by the engine, not by the connector: an
output port publishes at most every *n* seconds, an input port wakes its node
at most that often. A connector should poll at the rate the device is
comfortable with and leave delivery rates to whoever wires it up.
- A key no port declares is an error, and it fails the whole reading rather
than the one value. A mistyped metric name is how a training curve goes
missing.
- `poll()` returns values keyed by **output port**.
- `write()` receives values keyed by **input port**.
- The base class maps local port names to message names and validates types.
- Unknown output keys fail the reading. Return only declared ports.
- Port `interval` limits are enforced by the engine; the connector controls
how often it reads the device.
### When the device decides what the ports are
A connector for a device whose readings vary by model (which components a
relay has, which entities were flashed onto a board) cannot know its ports in
advance, and returning everything the device reports would fail on the first
value nobody bound. Narrow the reading to the ports that were declared:
For devices with variable readings, filter to the configured outputs:
```python
declared = {spec.port for spec in self.output_ports if spec.name}
return {port: value for port, value in reading.items() if port in declared}
```
`spec.port` is the local, unqualified name and stays that way for the node's
whole life, so the set can be taken fresh each time. Say something in the log
when a declared port is not one the device has, once rather than once per poll: from
the canvas a renamed entity and a typo look the same, and both leave a port
silent forever.
Log missing device fields once so a typo does not leave a port silently empty.
## Polling
Set `poll_interval` and implement `poll()`; the base class runs the loop:
```python
async def poll(self) -> dict[str, Any] | None:
async def poll(self):
return {"reading": await self._read_device()}
```
- Return `None` when there is nothing new.
- **Only changed values are published.** A device polled every few seconds
usually says the same thing, and every publication wakes everything
downstream, so the loop compares against what it last published, meaning what it
actually published, so a publication that failed is retried next tick rather
than counting as said.
- Raising is not fatal: it is reported as a health problem and retried on the
next tick.
- The loop calls `inject`, which runs the graph, on a worker thread. `poll()`
itself runs on the event loop and must not block it.
Set `poll_interval` in seconds; `0` disables polling. Return `None` when
there is nothing new. The base loop publishes only changed values and retries
a failed reading or publication on the next tick. It reports connection health
around each poll.
A connector that is pushed to rather than polled leaves `poll_interval` at 0,
overrides `start`/`stop` to open and close its own subscription, and calls
`self.inject({...})` when something arrives.
`poll()` runs on the event loop. Use `asyncio.to_thread` for blocking reads.
The base loop already calls `inject` on a worker thread.
For pushed readings, leave `poll_interval=0`, manage the subscription in
`start()` and `stop()`, and call `self.inject({...})` when values arrive.
Offload `inject` too when calling it from the event loop.
## Writing
A connector that commands something rather than only reading it implements
`write()`. It is called when the node's inputs are satisfied, with the values
keyed by input port:
```python
def write(self, **ports: Any) -> dict[str, Any] | None:
def write(self, **ports):
self._device.set(ports["level"])
return None
```
- Settings are already parsed on `self.config`; `write()` is handed ports only.
- Return `None` unless the device answers something worth publishing, in which
case return it keyed by output port, exactly as `poll()` does.
- It runs on a worker thread rather than the event loop, but the thread is the
scheduler's, so a long block holds up the cascade. Queue the work if the
device is slow.
- A connector that only reads leaves it alone.
- `poll()` and `write()` are independent: a connector may have both, and a
node with inputs and no `poll_interval` never polls.
Settings stay on `self.config`; arguments contain input ports only.
Return `None`, or a dictionary keyed by output port if the device responds
with useful values.
A write is a command, not a value: set `idempotent = False` on the class so a
redelivery after a crash does not undo a newer command that already landed.
Writes run on a scheduler thread. Return promptly or queue slow work.
Polling and writing are independent; a connector may support both.
Set `idempotent = False` for commands that must not be redelivered after a
crash. Leave `write()` unchanged on a read-only connector.
## Devices whose readings are bytes
A camera frame or a recorded clip is far too big to be a message, so a
connector publishes a reference to it instead:
```python
async def poll(self) -> dict[str, Any] | None:
import asyncio
async def poll(self):
jpeg = await asyncio.to_thread(self._grab)
return {
"frame": self.save_artifact(
@@ -183,135 +133,110 @@ async def poll(self) -> dict[str, Any] | None:
}
```
`save_artifact` stores the bytes and returns what an `image`, `audio` or
`video` port carries: the media type has to match the port's type. It only
works once the node has started, since the store is the engine's and is handed
over then.
`save_artifact(data, name="", media_type="application/octet-stream",
volatile=False)` returns a reference for an artifact or media port.
Match the media type to an `image`, `audio`, or `video` port.
`volatile=True` is what a camera publishes with. The frame goes to a ring in
memory rather than the data volume and is pushed down the websocket to
whichever screens are drawing it, so a wall panel sees ten frames a second and
the SD card under it is never written to. Frames last until newer ones need the
room; leave the flag off for a reading somebody asked to keep.
The artifact store is available after `start()`. Use `volatile=True`
for disposable live frames: they go to a bounded memory ring and expire as
new frames arrive. Use durable storage for recordings.
Each reading is a new artifact, which the poll loop publishes because its
digest differs from the last. Set `poll_interval` to what somebody actually
wants to look at. Through a portal the bytes are fetched rather than pushed, so
a remote panel wants a frame every second or two; higher rates than that are
for the local network, and full-rate video still belongs on the camera's own
stream.
Local screens receive pushed live frames; portal screens fetch the bytes.
Choose a modest poll rate for remote viewing.
## Keeping state across restarts
A connection, a session or a device handle lives in the connector's own
attributes and goes with the process. The few things that must not, such as
the certificate a device presented on first contact or a token it paired with,
go in `self.store`:
Keep connections and device handles in node attributes. Store short strings
such as pairing tokens or pinned certificates in `self.store`:
```python
pem = await asyncio.to_thread(self.store.get, f"cert:{self.config.host}")
await asyncio.to_thread(self.store.set, f"cert:{self.config.host}", pem)
await asyncio.to_thread(self.store.delete, f"cert:{self.config.host}")
value = await asyncio.to_thread(self.store.get, "token:device")
await asyncio.to_thread(self.store.set, "token:device", "value")
await asyncio.to_thread(self.store.delete, "token:device")
```
- Keys and values are strings; `get` returns `None` for a key that is not
there, and `delete` returns whether there was one.
- The store belongs to the connector's **top-level package**, not to a node:
every node of the package sees the same keys, so two nodes on one bridge
share what was stored for it. Put the device in the key.
- It lives in the engine's database, so it survives a restart and needs
nothing `fluksio serve` does not have. It is available before `start`.
- Limits: a key is 1 to 200 characters without `/` or whitespace, a value at
most 64 kB, and a package at most 256 keys. Past any of them, `set` raises
`ValueError`. Bytes worth keeping belong in `save_artifact`.
- Every call is a database round trip, so on the event loop it goes through
`asyncio.to_thread`.
- Keys and values are strings. `get` returns `None` for a missing key;
`delete` returns whether it existed.
- All nodes from the connector's top-level package share the namespace.
Include a device identifier in each key.
- Storage is in the engine's database and is available before `start()`.
- Keys allow 1–200 characters, without `/` or whitespace. Values allow
64 KiB and each package allows 256 keys; exceeding a limit raises `ValueError`.
- Calls block on the database: use `asyncio.to_thread` on the event loop.
A superuser can see the keys (never the values) and clear them, all of them or
one at a time, under *Stored state* in a connector node's settings, or through
the API, which names the connector by any node type it installs:
Superusers can list keys and clear them under **Stored state** in the node
panel. Values are never returned. The equivalent API paths are:
```sh
curl -s $FLUKSIO/connectors/hue/state -H "Authorization: Bearer $TOKEN"
curl -X DELETE $FLUKSIO/connectors/hue/state/cert:10.0.0.2 -H "Authorization: Bearer $TOKEN"
curl -X DELETE $FLUKSIO/connectors/hue/state -H "Authorization: Bearer $TOKEN"
```
| Method | Path |
|---|---|
| `GET` | `/api/v1/connectors/{node_type}/state` |
| `DELETE` | `/api/v1/connectors/{node_type}/state/{key}` |
| `DELETE` | `/api/v1/connectors/{node_type}/state` |
So a connector has to cope with a key disappearing: clearing is how somebody
tells it to start over.
Handle missing keys: clearing one asks the connector to pair or learn again.
### Pinning a device's certificate
A device on the local network presents a certificate no system trusts and that
names a serial or a MAC rather than its address, so `verify=True` fails and
`verify=False` trusts anyone. `fluksio.flow.connector_tls` trusts the
certificate the device presented on first contact instead, kept in `self.store`
under `cert:<address>`:
`fluksio.flow.connector_tls` supports trusting a local device's certificate
on first contact:
```python
from fluksio.flow.connector_tls import context_for, pinned_certificate, raise_if_changed
import httpx
from fluksio.flow.connector_tls import (
context_for,
pinned_certificate,
raise_if_changed,
)
address = f"{self.config.host}:{self.config.port}"
pem = pinned_certificate(self.store, address, self.config.timeout) # blocking
pem = pinned_certificate(self.store, address, self.config.timeout)
try:
with httpx.Client(verify=context_for(pem)) as client:
...
response = client.get(f"https://{address}/status")
response.raise_for_status()
except httpx.ConnectError as exc:
raise_if_changed(exc, pem, address, self.config.timeout, "my_node_type")
raise_if_changed(exc, pem, address, self.config.timeout, "my_device")
raise
```
- First contact sends a TLS hello and nothing else; a device that does not
complete one raises `OSError` and nothing is pinned.
- Any other certificate is refused in the handshake, before a key or a command
is sent. `raise_if_changed` turns that refusal into `CertificateChanged`,
which names the new SHA-256 and the key to clear, under the node type given.
- The pinned certificate's dates are not checked: a pin trusts that exact
certificate, so one that expired, or an engine clock that is off, does not
lock the device out.
- Clearing the key accepts whatever the device presents next.
Run this blocking work off the event loop. First contact pins the certificate
under `cert:<address>`; a changed certificate is refused before sending a
credential or command. `raise_if_changed` reports `CertificateChanged`
with the fingerprint and key to clear.
## Lifecycle
Pins trust the exact certificate, including after its expiry. Clearing the
key accepts the certificate presented on the next connection.
## Lifecycle and health
```python
async def start(self, app=None) -> None: ...
async def stop(self, app=None) -> None: ...
async def start(self, app=None):
await super().start(app)
async def stop(self, app=None):
await super().stop(app)
```
- `start` is called when the connector's flow starts, and after every rebuild.
- `stop` is called before a rebuild and when the flow is stopped. **It must be
idempotent**: it is called whether or not `start` succeeded.
- Overriding either means calling `super()` if the polling loop is also wanted.
- The `app` argument is the FastAPI application, for the rare connector that
needs to mount a route. Most ignore it.
`start` runs when a flow starts or rebuilds. `stop` runs before rebuilding
or stopping, even after a failed start; make it idempotent. Call `super()`
when overriding these hooks if you want the base polling loop.
A stopped flow gets no `start` at all: that is what stopping it means.
The optional `app` is the FastAPI application, for connectors that mount
routes. Most connectors can ignore it.
## Health
Report connection changes if you manage your own subscription:
```python
self.report_health("ok")
self.report_health("degraded", "3 of 5 registers timed out")
self.report_health("down", str(exc))
self.report_health("degraded", "Some readings timed out")
self.report_health("down", "Connection lost")
```
Three values, `ok`, `degraded` and `down`, plus an optional detail string.
Reporting the same status twice is free, since only changes are published. A node
reporting `down` is named among its flow's issues and counted on the health
summary on Home; `degraded` means still working, and is not. The polling loop
already reports around `poll()`; a connector managing its own connection should
report when it connects and when it loses the connection.
Repeated statuses do not emit another event. `down` appears among flow issues
and on Home; `degraded` means the connection still works. Loading failures
are reported separately.
Health is about the connection, not about loading: a connector that fails to
load is reported separately and never runs.
## What a connector must not do
- **Hold state that belongs in the flow.** Logic nodes are stateless and run in
parallel; a connector may hold a connection, a session, or a device handle,
and keep what has to outlive a restart in `self.store`, but a value another
node needs belongs in a message.
- **Block the event loop.** `poll`, `start` and `stop` are async and run on the
loop. Use `asyncio.to_thread` for blocking I/O.
- **Read credentials from the environment.** They come through parameters, so
the person deploying a flow can change them without touching the host.
Keep shared flow values in messages, credentials in secret parameters, and
blocking I/O off the event loop.
+49 -75
View File
@@ -1,17 +1,18 @@
# Node types
Every type the canvas can place. Each is configured by filling in a form the
editor generates from its parameter schema, so they all behave the same way.
Anything here can also be written as a **Function** node. These types cover the
shapes that come up most often, as a form to fill in rather than repeated code.
Built-in nodes cover common integrations, timing, and logic. Configure their
settings in the node panel; use a **Function** node for your own Python.
`GET /flows/node-types` returns this list with each type's full parameter
schema, including any [connectors](#connectors) installed on your instance.
MQTT and InfluxDB need `pip install 'fluksio[server]'` on a Python installation.
The Docker image includes these dependencies. Run the broker and database
separately, or use the development stack's test services.
## Function
**`python`** — your own code, run on every incoming message.
**`python`** — run your own code when its triggering inputs arrive.
The one type with a source file. Its arguments are its input ports and its own
settings; its return value is a dict keyed by output ports. See
@@ -46,9 +47,11 @@ A topic may be a filter: `+` matches one level, `#` the rest. Everything a
filter matches lands on the same port, so use one port per topic where the
difference matters.
`json_key` is for a device that wraps its reading: Victron publishes
`{"value": 47}` on every path. Without it, a payload object is unwrapped only
when it happens to carry the port's own name as a key.
Set an explicit topic: the default `*` is a literal topic, not MQTT's
match-all filter.
For `{"value": 47}`, set `json_key` to `value`. Without it, an object is
unwrapped only if it contains the port's name as a key.
### HTTP
@@ -81,6 +84,7 @@ Outputs only makes it a **webhook**: the engine mounts a route at
| `url`, `token`, `org`, `bucket` | connection; `token` takes a secret reference |
| `write_precision` | `ns`, `us`, `ms` (default) or `s` |
| `query_range` | default window for queries, e.g. `-1h` |
| `timeout` | operation timeout in seconds; default `10.0` |
| `writes` | per-input: `measurement`, `field`, `tags` |
| `queries` | per-output: `measurement`, `field`, `tags`, `range`, `aggregation` |
@@ -96,12 +100,10 @@ Outputs only makes it a **webhook**: the engine mounts a route at
}
```
**Query passthrough.** An incoming message holding a `flux` key is run as
written, and the rows come back on the first output port as
`{"rows": [{ts, value, field, measurement, tags}], ...}`.
A database node holds the credentials and the connection, nothing else.
Building a query and shaping its rows are Function nodes on either side.
**Query passthrough.** An incoming object with a `flux` key supplies the query.
The first output receives an object with `rows`, each containing `ts`, `value`,
`field`, `measurement`, and `tags`. Use Function nodes to build queries and
shape results for charts.
### Notification
@@ -115,10 +117,10 @@ Building a query and shaping its rows are Function nodes on either side.
| `priority` | `default` | `min`, `low`, `default`, `high`, `urgent` |
| `tags` | — | comma-separated ntfy tags |
| `token` | — | for a protected topic; takes a secret reference |
| `timeout` | `10.0` | request timeout in seconds |
This is a *flow* deciding something is worth saying. The engine reporting that
it broke is [Alerts](../interface/operations.md#alerts), which is a different
thing configured elsewhere.
For engine failures rather than flow messages, configure
[Alerts](../interface/operations.md#alerts).
## Timing
@@ -135,9 +137,7 @@ thing configured elsewhere.
| `at_start` | `false` | emit once when the flow starts |
| `start_delay` | `1.0` | how long to wait before that first emission |
The scheduler: a `cron` expression here is what makes a flow run by the clock.
It is also the most-placed node in a real instance, mostly as a button someone
presses.
Use `cron` for scheduled emissions, or trigger the node by hand for testing.
### Delay & schedule
@@ -154,17 +154,12 @@ Order of operations: rate check → delay → forward. With a `cron` and no inpu
it emits the current time on each tick; with inputs it emits the last value it
received.
The engine sleeps until a delay is due rather than polling for it, so a `delay`
fires within a few milliseconds of its deadline on an idle engine, and a due
timer is taken off the queue before work that is merely waiting. It is not a
real-time guarantee: the wait ends on a normal thread, and a node that then
needs a busy cascade slot waits for one.
Timers wait until their deadline; busy workers can delay execution.
!!! note "Not in a batch flow"
A rate limit holds a value back for a timer to release, and a run has no
timer, since the value would be dropped rather than delayed. Submitting a batch
flow with a rate-limited port is refused instead.
Batch flows reject rate-limited non-streaming ports. Runs have no live
timer to release a held value.
### Trigger
@@ -181,14 +176,11 @@ starts from a value arriving. For a cron tick, see [`inject`](#inject).
| `passthrough` | `false` | send the incoming value instead of `first` |
| `wait_port` | — | an input carrying the wait, when it differs per message |
The shape this exists for: *the door opened, so turn the light on, and off
again in two minutes unless it opens again.*
For example: turn a light on when the door opens, then off after two quiet
minutes.
`wait_port` covers the case where how long to wait is itself a value: a
rollershutter takes 26 seconds up and 28 down, so the node that decides the
direction says how long to run for as well. A wait of zero or less sends
nothing afterwards, and still cancels whatever the last message scheduled,
which is how a *stop* is commanded exactly once.
Use `wait_port` when the delay comes from an input. A wait of zero or less
cancels the pending second value and schedules nothing new.
## Logic
@@ -243,8 +235,8 @@ The cheapest fix for a sensor that reports every second and changes every hour.
### Command
**`exec`** — run a command in the engine's container and read its output.
Adding one or changing its settings is a superuser's, since the command is code.
**`exec`** — run a command where the engine runs and read its output.
Adding one or changing its settings requires a superuser.
| Setting | Default | Notes |
|---|---|---|
@@ -255,11 +247,8 @@ Adding one or changing its settings is a superuser's, since the command is code.
Outputs the command's stdout, stderr and exit code.
!!! warning "Inside the container, not on the host"
A flow ported from something that read the host's journal, or poked a host
script, needs either a mount or a small listener on the host side. This
node cannot see the host.
In Docker, the command runs inside the backend container. Mount host files
explicitly if the command needs them.
### File
@@ -272,8 +261,7 @@ Outputs the command's stdout, stderr and exit code.
| `format` | `text` | `text` or `json` |
| `newline` | `true` | end each written record with a newline |
Confined to a directory the engine owns. A flow that could name any path would
be a way to read the secrets store or overwrite a node's source.
Paths stay inside the engine's files directory.
## Numeric
@@ -297,17 +285,15 @@ use a [batch flow](../concepts/runs.md) and a Function node.
**`wren`** — turn a typed command into values for the devices wired to it.
[Wren](https://github.com/stroblme/wren) is a small language model that runs
on the engine's CPU: about 40 MB, a fifth of a second per command. It reads the
house as text, the areas, the devices in each and their states, and answers a
command with calls on those devices. It is a separate package, installed into
the engine with `pip install wren`; without it the node says so on the canvas.
[Wren](https://github.com/stroblme/wren) is an optional local language model
for device commands. Install it in the engine's environment with
`pip install wren`; the node reports a missing dependency if it is absent.
| Input | Type | Notes |
|---|---|---|
| `prompt` | `str` | the command, and the only input that runs the node |
| `presence` | `str` | the area the user is in; anything else reads as away |
| `outdoor` | `float` | °C; 20 without it |
| `outdoor` | `float` or `int` | °C; 20 without it, clamped to −5…30 |
| `dark` | `bool` | without it, dark before 07:00 and from 18:00 |
The outputs are `actions`, a list of records, and any number of **targets**. A
@@ -346,43 +332,31 @@ reads.
| `apply` | `false` | write to the targets; off, the node only reports |
| `min_confidence` | `0.93` | below this nothing is written |
Each record in `actions` is `{message, value, applied, delay_s, call,
confidence}`, one per call and target. A call nothing is wired for, such as
`sensors.temperature` or a light in a room without one, gets a record with an
empty `message`. The confidence is the lowest probability the model gave any
part of its answer: at the default threshold, nine in ten right answers pass,
and 38 of the 40 wrong ones in the model's own evaluation do not.
Each record in `actions` contains `message`, `value`, `applied`, `delay_s`,
`call`, and `confidence`. Unwired calls have an empty `message` and are not
applied. Keep `apply=false` while checking how the model interprets commands.
A call with a delay ("in 10m") is held by the engine's work queue, like the
Trigger node's second value: it survives a restart where the queue is in
Redis, and not otherwise.
Every house the model was trained on had a living room with a light, media
and heating, and a light in every other area. The node fills in what is not
wired, so the model reads a house it knows; a call on one of those is
reported, never applied.
The node adds default devices to the model's view for missing expected devices.
They are reported but cannot receive writes unless wired.
!!! note "What the current model cannot do"
No levels: a light is on or off, never dimmed, coloured or half open.
Delays are whole minutes. Only one device per kind and area. Only the ten
areas above: one it was not trained on is answered confidently and wrongly,
which is why such a port is refused rather than tried. Commands are English,
phrased roughly like "turn on the lights in the kitchen".
Commands are English, such as "turn on the lights in the kitchen".
The model uses the areas above, one device per kind and area, on/off
actions, and whole-minute delays. It does not dim lights or set colors.
The chat's [Control tab](../code/agents.md#control) sends commands by hand.
## Connectors
Anything else in the palette came from an installed **connector** package: a
node type written against a published contract and discovered through the
`fluksio.node_types` entry point group. The editor shows which package supplied
it.
A connector declares its contract version, and one written for a version this
engine does not speak is ignored rather than half-loaded. Installing or
upgrading one takes effect on the next engine restart, because Python does not
re-import a changed module and a rescan would promise more than it delivers.
Installed connector packages add types through the `fluksio.node_types`
entry point group. The editor shows the source package. Incompatible contract
versions are skipped; installing or upgrading a connector requires an engine
restart.
The contract itself is in [The connector contract](connector-contract.md), and
[Writing a connector](../code/connectors.md) walks through building one.
+82 -119
View File
@@ -1,207 +1,170 @@
# Payload types
Every port declares a `dtype`, and every value that passes through it is
checked against that declaration.
Every port declares a `dtype`. Fluksio checks values against it and offers
compatible messages when you bind dashboard widgets.
This is not decoration. It is what lets the dashboard editor offer you only the
messages a gauge can actually draw, what lets the canvas refuse a binding before
anything runs, and what lets a downstream node know the shape of what it is
getting before the flow starts.
Messages carry JSON. Binary data lives in the artifact store; messages carry
references to it.
Everything on the wire is JSON. That is what lets the same value pass through
the state backend, the work queue and the worker protocol unchanged.
## The scalars
## Scalars
| `dtype` | Accepts |
|---|---|
| `float` | any number — `int` or `float`, but not `bool` |
| `int` | a whole number, not `bool` |
| `str` | a string |
| `bool` | exactly `true` or `false` |
| `float` | integer or floating-point number, excluding booleans |
| `int` | integer, excluding booleans |
| `str` | string |
| `bool` | `true` or `false` |
`bool` is an `int` subclass in Python but is not a number here, so a switch
cannot bind to a temperature.
`NaN` and infinity are rejected, including in structured payloads. For an
unavailable numeric result, omit that output or return `None` to publish
nothing. A numeric port does not accept `None` as its value.
`NaN` and infinity are refused, wherever they sit, including inside a `json`,
`record`, `series` or `list`. JSON cannot spell either, so one that travelled
would come back as a response nobody can parse and a row the database rejects,
a long way from the node that made it. An empty subset or a division with no
denominator is what usually produces one; publish `None` instead.
## The structured ones
These are *declared shapes* rather than "some JSON", which is what makes a
widget binding checkable.
## Structured payloads
### `record`
Flat named scalars.
An object with string keys and scalar or null values. Nested objects and
lists are not allowed.
```json
{"title": "Boiler", "body": "Pressure low", "severity": "warning"}
```
A record cannot contain a record, so its shape is readable from the declaration
alone.
Read by the **Notification** widget. It is also what an alert channel of kind
*dashboard* writes.
Notification and Player widgets read records. Dashboard alert channels also
publish them.
### `list`
Ordered items of one declared shape. The port also declares `item`:
An array with a declared item type:
| `item` | Meaning |
| `item` | Accepts |
|---|---|
| unset | `record` — what the agenda and forecast widgets read |
| `float`, `int`, `str`, `bool` | a list of scalars |
| `json` | anything |
| unset or `record` | flat records; used by Agenda and Forecast |
| `float`, `int`, `str`, `bool` | scalars of that type |
| `json` | arbitrary JSON values |
A list of lists, or a list of series, is refused. One declared level is the
point.
`list` and `series` cannot be declared as item types. Use `item: "json"` if
items need arbitrary nested structures.
### `series`
Labelled lines of `(timestamp, value)` pairs, which is what a chart draws.
Labelled lines of numeric `[timestamp, value]` pairs:
```json
{
"lines": [
{"label": "living", "points": [[1717000000, 21.4], [1717000060, 21.5]]}
],
"range": "-24h"
"range_s": 86400,
"interval_s": 60
}
```
Keys beside `lines` are carried through untouched, which is how a querying
chart puts the window and resolution it asked for on the request and reads them
back off the answer. That is what stops an answer to a *different* question
from overwriting the picture.
Additional top-level keys are preserved. A Query chart expects the
`range_s` and `interval_s` from its request in the response; this prevents
an answer for another window from replacing its chart.
`GET /runs/series/compare` answers in this shape, which is why comparing three
training curves is a widget binding rather than a screen of its own.
The run-series comparison endpoint also returns this shape.
### `artifact`
A reference to stored bytes.
A reference to bytes in the content-addressed artifact store:
```json
{"digest": "sha256:…", "size": 4194304, "media_type": "application/octet-stream", "name": "weights.pt"}
```
Binary payloads (tensors, checkpoints, images) never travel as a message. The
bytes go to a content-addressed store and the message carries this. A
thirty-megabyte checkpoint never sits in the state backend, and the reference
stays valid wherever the store is reachable from, including on another machine.
`digest` and `size` identify the content. `media_type` and `name` describe it.
The reference works wherever the store is reachable.
Node code produces one with `fluksio.save_artifact` and opens one with
`fluksio.load_artifact`. See [Writing node code](../code/nodes.md#bytes-artifacts).
Create one with `fluksio.save_artifact` and read its bytes with
`fluksio.load_artifact`. See [Artifacts](../code/nodes.md#bytes-artifacts).
### `image`, `audio`, `video`
The same reference, narrowed to a kind of media by its `media_type`.
Artifact references restricted to the matching `media_type` family:
```json
{"digest": "sha256:…", "size": 61344, "media_type": "image/jpeg", "name": "frame.jpg",
"meta": {"width": 1280, "height": 720, "seq": 41}}
{
"digest": "sha256:…",
"size": 61344,
"media_type": "image/jpeg",
"name": "frame.jpg",
"meta": {"width": 1280, "height": 720, "seq": 41}
}
```
An `audio` port takes `audio/*` and refuses anything else, so a speech
recogniser declares what it eats rather than taking any bytes at all and
finding out. An `artifact` port still accepts all three: media narrows
artifact, not the other way round.
An `audio` port accepts `audio/*`; `image` and `video` work the same way.
An `artifact` port accepts all three. Optional `meta` fields are for the
consumer; Fluksio does not interpret them.
`meta` is optional and nothing here reads it: sample rates, dimensions and
sequence numbers are for whoever consumes the media.
A media stream is a [streaming port](../concepts/flows.md#streaming-ports)
publishing one reference per frame or chunk. Saving with
[`volatile=True`](../code/nodes.md#media) keeps bytes in memory and lets local
websocket clients receive them with the value. Stored media and portal views
fetch the bytes separately.
Bytes still never travel as a message. A camera publishes one reference per
frame and a microphone one per chunk, which makes a media stream an ordinary
[streaming port](../concepts/flows.md#streaming-ports). What that costs
depends on how the bytes reach the screen:
| Rate | Where it works |
|---|---|
| A clip a second (speech) | anywhere, including through the portal |
| Ten frames a second (a camera worth watching) | on the local network, with `volatile=True` |
| A frame every second or two (a glance at a door) | anywhere, including through the portal |
| Full-rate video, 30 fps and up | not here — see below |
The difference is one flag. A frame saved with
[`volatile=True`](../code/nodes.md#media) is held in memory and pushed down the
websocket in front of the value naming it, so a screen draws it without asking
for anything. A stored one is fetched instead: a round trip per frame, which is
a glance rather than a view. Through a portal every frame is fetched, so a
remote panel is in the second row whatever the flag says.
Above that, video is not a message-plane problem — every frame would still be
an event. Point a media widget's **stream URL** at whatever the camera already
serves and the browser plays it from source; the messages then carry the
occasional still, and the flow reacts to those.
Use occasional snapshots through a portal. For continuous video, point a
Media widget's **Live stream** at a browser-compatible source URL reachable
by the screen.
### `json`
Anything JSON-serializable. The escape hatch, and the right answer when a
payload genuinely has no fixed shape.
Any JSON-serializable value. Use it for payloads without a fixed shape.
Prefer a specific type when one fits; it gives widgets and other authors a
clearer contract.
Reach for it last. A `json` port tells the canvas, the widget picker and the
next author nothing.
## Referencing a run's output
## Naming a run's output
A run parameter can name an existing value instead of repeating it:
Any *run parameter* is also accepted as text, since nobody wants to paste an
object into a shell. `@run:<id>.<output>` names what an earlier run produced,
whatever its type, an `artifact` reference or a `json` config alike. A bare
`sha256:…` digest names content in the artifact store. Both resolve before the
run starts, so the CLI, the run dialog and a python caller all mean the same
thing by the same string.
- `@run:<id>.<output>` resolves an earlier run's output.
- `sha256:…` resolves an artifact digest.
Both spellings are reserved on every input, `str` included: an input that has to
carry one of them literally is asking for a value this engine reads as a name.
These strings resolve before the run starts, regardless of the input's type.
Both prefixes are reserved even for `str` inputs; they cannot be passed as
literal text in run parameters.
## What a widget will bind to
## Widget bindings
| Widget | Accepts |
|---|---|
| Gauge, Chart, Slider, Bar | `float`, `int` |
| Gauge, live Chart, Slider, Bar | `float`, `int` |
| Query Chart | `series` response, `record` request |
| Switch | `bool` |
| Agenda, Forecast | `list` |
| Notification | `record` |
| Value | anything |
| Icon | weather strings, booleans and numbers alike |
| Notification, Player | `record` |
| Colour | `list` for HSV/RGB, `str` for hex |
| Media | `image`, `audio`, `video`, `artifact` |
| Clock, Text | nothing — they bind to no message |
| Value, Icon, Button, Buttons, Input, Selector | no fixed type restriction |
| Clock, Text, Website | no message binding |
Enforced on the server as well as in the editor.
Player transport commands publish to a separate `str` message.
Binding checks apply in the editor and on the server.
## Icon and colour names
A forecast is a `list` of records, one per column, each shaped
`{label, icon, value}` with an optional `color`. The flow decides what a
column is called and which glyph it gets; the widget only draws it.
Forecast reads a list of records with `label`, `icon` and `value`, plus
optional `color`:
```json
[{"label": "Today", "icon": "cloud-rain", "color": "primary", "value": "14°"}]
```
`icon` and `color` are names from a fixed vocabulary, the same one the icon
widget's picker offers:
The Icon widget uses the same vocabulary:
| | Names |
|---|---|
| Icons | `sun`, `moon`, `cloudy`, `cloud-rain`, `cloud-drizzle`, `cloud-snow`, `cloud-lightning`, `cloud-fog`, `wind`, `umbrella`, `snowflake`, `droplets`, `thermometer`, `thermometer-sun`, `flame`, `fan`, `sun-snow`, `door-open`, `house`, `bed`, `lightbulb`, `plug`, `music`, `volume-2`, `volume-x`, `zap`, `battery-charging`, `arrow-up`, `arrow-down`, `triangle-alert`, `check`, `circle-check` |
| Icons | `sun`, `moon`, `cloudy`, `cloud-rain`, `cloud-drizzle`, `cloud-snow`, `cloud-lightning`, `cloud-fog`, `wind`, `umbrella`, `snowflake`, `droplets`, `thermometer`, `thermometer-sun`, `flame`, `fan`, `sun-snow`, `door-open`, `house`, `bed`, `lightbulb`, `plug`, `cctv`, `music`, `volume-2`, `volume-x`, `zap`, `battery-charging`, `arrow-up`, `arrow-down`, `triangle-alert`, `check`, `circle-check` |
| Colours | `default`, `muted`, `primary`, `success`, `danger` |
An empty icon draws nothing. An unknown one draws a placeholder glyph and an
unknown colour draws as `default`; either warns once in the browser console,
listing the names it expected.
An empty icon draws nothing. Unknown icons use a placeholder; unknown colours
use `default`. Each warns once in the browser console.
## Type failures
A value that does not match its port's declaration raises on the node that
published it, naming the port and what arrived. It does not get published, and
it does not reach anything downstream. A wrong value stopping at its source is
easier to diagnose than one propagating.
A mismatched value raises an error on the node that produced it, naming the
port and received value. It is not published downstream.
## See also