diff --git a/docs/code/agents.md b/docs/code/agents.md index 8286b6f..4ce4ad4 100644 --- a/docs/code/agents.md +++ b/docs/code/agents.md @@ -1,145 +1,67 @@ # Agents over MCP -The engine exposes the flow API to agents over the -[Model Context Protocol](https://modelcontextprotocol.io). Point Claude Code, -Claude Desktop, Codex or anything else that speaks MCP at it, and it can read, -build, publish and run flows. +Fluksio exposes flow tools over MCP. An agent can inspect, edit, publish, +and run flows with the permissions of the user who authorizes it. -## Switching it on +## Connect an agent -```sh -MCP_ENABLED=true -``` +Set `MCP_ENABLED=true` in the engine's environment and restart it. The Python +package defaults to off; the Docker template enables it. -It is off by default: switching it on opens client registration to whoever can -reach the host. +Configure your MCP client with an HTTP endpoint: -The endpoint is `https://api.${DOMAIN}/mcp`, or `http://api.localhost/mcp` on a -local stack. +- Local Docker stack: `http://api.localhost/mcp` +- Deployment: `https://api./mcp` -## Connecting a client +The client registers, opens browser authorization, and receives an OAuth +token after you approve access. The issuer and browser redirects use +`DOMAIN`, `ENVIRONMENT`, and `FRONTEND_HOST`, so configure those for the +addresses your client and browser can reach. -```sh -claude mcp add --transport http fluksio https://api.${DOMAIN}/mcp -``` +Manage registered clients under **Admin → Agents**. Revoking a client +invalidates its access. MCP uses a separate signing key at +`OAUTH_PRIVATE_KEY_FILE`; rotating that key invalidates all MCP tokens. -The OAuth flow does the rest: the client registers itself, you approve it in -the browser, and it exchanges the grant for a token. Nothing to paste. +MCP is not forwarded by the [portal tunnel](../interface/portal.md). +Connect directly to the instance's API host. -## What an agent may do +## Available tools -An agent acts as **the person who approved it**, and only through the same REST -API the dashboard uses. It cannot do anything you could not do in the browser, -and it cannot do anything *you* cannot do. An agent approved by a non-superuser -is not one either. - -Its tokens are signed with a keypair of their own, kept beside the flow store. -Deleting that file revokes every agent's access without logging anyone out of -the dashboard. - -Registered clients are managed under **Admin**. - -## The tools - -Read-only: - -| Tool | Returns | +| Task | Tools | |---|---| -| `list_flows` | every flow with its node count and state | -| `get_flow` | one flow's definition, node status and issues | -| `get_node_source` | a node's Python | -| `list_node_types` | every type and its parameter schema | -| `get_flow_state` | the current value of each of a flow's messages | -| `get_message_history` | one message's recent numeric values | -| `get_graph` | every flow as one graph | -| `list_shared_nodes` | the shared library and its usages | -| `list_secrets` | secret **names**, never values | -| `get_modules` | the packages node code can import | -| `get_health` | flows, nodes, queue and loop lag | -| `get_metrics` | executions, errors and timings per minute | -| `list_failures` | what went wrong recently | -| `list_runs` | recent cascades | +| Inspect flows | `list_flows`, `get_flow`, `get_node_source`, `list_node_types`, `get_graph`, `list_shared_nodes` | +| Inspect values | `get_flow_state`, `get_message_history` | +| Inspect the instance | `list_secrets` (names only), `get_modules`, `get_health`, `get_metrics`, `list_failures`, `list_runs` (live cascades) | +| Edit flows | `save_flow`, `save_node_source`, `validate_flow`, `publish_flow`, `discard_draft`, `delete_flow` | +| Operate flows | `run_flow`, `trigger_node`, `start_flow`, `stop_flow`, `pause_flow`, `resume_flow`, `cancel_node` | +| Install node packages | `apply_modules` | -Editing: +Tools call the same API as the browser. Drafts, validation, version checks, +and superuser restrictions apply. Secrets cannot be read back. -| Tool | Does | -|---|---| -| `save_flow` | write a draft | -| `save_node_source` | write a node's code into the draft | -| `validate_flow` | the issues, without saving | -| `publish_flow` | put the draft live (takes the version it saw) | -| `discard_draft` | throw the draft away | -| `delete_flow` | delete it | -| `apply_modules` | replace the package manifest | - -Operating: - -| Tool | Does | -|---|---| -| `run_flow` | run every node once from the values given | -| `trigger_node` | feed one node and run what is downstream | -| `start_flow` / `stop_flow` | activate or tear down | -| `pause_flow` / `resume_flow` | hold messages, or let them through | -| `cancel_node` | kill a node's code while it runs | - -## Why it is shaped this way - -The tools do not reach into the engine. Each one calls the same REST endpoint -the dashboard calls, over an in-process transport. - -That keeps exactly one description of what a flow is and how it may be changed: -the validation, the draft/publish split, and the version check that stops two -clients overwriting each other. An agent saving a flow through a private back -door could write one the canvas cannot open. - -The caller's token rides along on every hop, so the API sees the agent's own -identity rather than a service account. The audit trail on Home names it. +A useful workflow: ask the agent to inspect the flow, save a draft, and +validate it. Review the draft before asking it to publish. The agent has +a publish tool, so make your intent explicit. ## The assistant -The same tools drive the chat inside the dashboard. An operator sets a -provider under **Settings → Assistant** (anything OpenAI-compatible: OpenAI, -Anthropic, OpenRouter, a local Ollama), and an **Assistant** entry appears in -the sidebar for everyone. The panel opens on any screen and keeps its place -from screen to screen; on a flow, it knows which one you are looking at. +A superuser configures an OpenAI-compatible provider under +**Settings → Assistant**, with a base URL, API key, and model. This enables +the dashboard's Assistant entry for signed-in users. -The model calls the tools above, in-process, as **you**: every call carries -your own token, so the chat can do nothing you could not do yourself, and its -edits show up in the editor as drafts the moment they are saved. It is told to -publish only when asked, but it has the tool, so read what it did before -saying so. The trace under each answer lists every call it made. - -The conversation is one per person, kept on the instance beside the -configuration, and the bin in the panel's header starts it over. `MCP_ENABLED` -is not needed for it: the endpoint is for agents outside, the chat for the -one inside. +The assistant uses the same tools with your own permissions. Draft edits +appear in the editor, and each answer includes a tool trace. Conversations +are stored per user on the instance; the bin button clears yours. +`MCP_ENABLED` is only needed for external clients, so the built-in assistant +works without it. ## Control -Where a live flow has a [`wren` node](../reference/node-types.md#wren) with -its `prompt` and `actions` wired, the chat gains a **Control** tab, and Home a -chat button at the top right of the brain graph (in the header, on a phone) -that opens it. No provider is needed for this one: the model is the node. +A live [Wren node](../reference/node-types.md#wren) with both `prompt` and +`actions` wired adds a **Control** tab to the chat. It uses the node's local +model and needs no assistant provider. -A command is published to the node's `prompt` message as a value from the -chat, and the answer is the next value of its `actions`: each call, the -message it went to, whether it was applied, and how sure the model was. A -failing node answers with its error, and one that says nothing for fifteen -seconds is reported as such. With several nodes, a picker in the header -chooses. The conversation is kept in the browser only, and a reload clears -it. - -## A note on working this way - -The pattern that works is the same one that works with a person: let the agent -read the flow and its issues, let it write a **draft**, and publish yourself -once you have looked. `validate_flow` before `publish_flow` is cheap and tells -you what the canvas would have told you. - -## Limits - -- MCP is not currently reachable through a [portal](../interface/portal.md) - tunnel: the proxy forwards `/api/v1/` only. Connect an agent on the same - network as the instance. -- Secrets are never readable, by an agent or by anyone else. `list_secrets` - returns names. +Choose a node, type a command, and inspect the reported actions: target +message, value, confidence, and whether each action was applied. +The node's `apply` and `min_confidence` settings decide whether it writes +to devices. Control conversations stay in the browser and clear on reload. diff --git a/docs/code/api.md b/docs/code/api.md index ea08cec..e608e89 100644 --- a/docs/code/api.md +++ b/docs/code/api.md @@ -1,274 +1,241 @@ # The HTTP API -Everything the browser does, the API does first. The dashboard is a generated -client of this schema, not a privileged path into the engine, so anything you -can click, you can script. +The API covers the same operations as the browser. Use it for scripts, +integrations, or your own client. -Base URL: `https://api.${DOMAIN}/api/v1`, or `http://127.0.0.1:8000/api/v1` for -a `fluksio serve` instance. +Set the base URL for your instance: ```sh export FLUKSIO=http://127.0.0.1:8000/api/v1 ``` +Use the port printed by `fluksio serve`. For Docker, use +`http://api.localhost/api/v1`, `http://:8080/api/v1` on a LAN stack, +or `https://api./api/v1` for a deployment. +All paths below are relative to this base. + ## Authenticating -```sh -export TOKEN=$(curl -s -X POST $FLUKSIO/login/access-token \ - -d "username=admin@example.com&password=..." | jq -r .access_token) +`fluksio serve` signs the local CLI in automatically. To reuse its token +from a project instance, with `jq` installed: -curl -s $FLUKSIO/users/me -H "Authorization: Bearer $TOKEN" +```sh +export TOKEN=$(jq -r .token .fluksio/client.json) ``` -A bearer token, valid for eight days. `POST /login/test-token` checks one. -Password recovery and reset are `POST /password-recovery/{email}` and +Adjust the path for `--global` or `--data-dir`. For a remote instance, +use its login endpoint with the administrator credentials: + +```sh +export TOKEN=$(curl -s -X POST "$FLUKSIO/login/access-token" \ + --data-urlencode "username=admin@example.com" \ + --data-urlencode "password=" | jq -r .access_token) + +curl "$FLUKSIO/users/me" -H "Authorization: Bearer $TOKEN" +``` + +Session tokens last eight days by default. `POST /login/test-token` checks +a token. Password recovery uses `POST /password-recovery/{email}` and `POST /reset-password/`. -Agents authenticate differently; see [Agents over MCP](agents.md). +[Agents over MCP](agents.md) use separate OAuth tokens. -!!! tip "The interactive schema" +!!! tip "Interactive API reference" - When `ENVIRONMENT` is not `production`, the full OpenAPI schema is at - `/docs` (Swagger) and `/redoc`, and the raw document at - `/api/v1/openapi.json`, for a browser on the engine's own machine (or any, - with `API_DOCS_ENABLED=true`). That is the authoritative reference; this - page is the tour. - - It is closed in production: the schema enumerates every endpoint the - instance serves, including paths webhook nodes mounted at runtime. + Outside production, open `/docs` or `/redoc` on the API host. + The schema is at `/api/v1/openapi.json`. Access is limited to loopback + unless `API_DOCS_ENABLED=true`; the Docker development stack enables it. + Production disables these endpoints. ## Flows -| Method | Path | What | +| Method | Path | Action | |---|---|---| -| `GET` | `/flows/` | every flow, with node counts, draft and running state | -| `GET` | `/flows/{name}` | one flow — the draft if there is one — with node status and issues | -| `PUT` | `/flows/{name}` | save a draft (`version` must match, or 409) | -| `POST` | `/flows/{name}/publish` | put the draft live | -| `POST` | `/flows/{name}/discard-draft` | throw the draft away | -| `POST` | `/flows/{name}/rename` | rename it | -| `DELETE` | `/flows/{name}` | delete it. Its runs stay — a run is the record of what ran, and only `DELETE /runs` clears one. Refused with 409 while a run of it is `running` or `queued` | -| `GET` | `/flows/node-types` | every node type and its parameter schema | -| `GET` | `/flows/graph` | every flow as one graph — what Home draws | +| `GET` | `/flows/` | list flows and their state | +| `GET` | `/flows/{name}` | read the draft when present, with status and issues | +| `PUT` | `/flows/{name}` | save a draft; send the current `version` | +| `POST` | `/flows/{name}/publish` | publish; send `{"version": }` | +| `POST` | `/flows/{name}/discard-draft` | discard unpublished edits | +| `POST` | `/flows/{name}/rename` | rename | +| `DELETE` | `/flows/{name}` | delete; refused while a run is queued or running | +| `GET` | `/flows/node-types` | list types and parameter schemas | +| `GET` | `/flows/graph` | read the graph shown on Home | -Reading a flow returns the *draft* when one exists, along with a preview of how -it would run: node statuses and validation issues for the version you are -editing, not the one running underneath it. +Deleting a flow keeps its run history. Delete runs separately. ### Node source ```sh -curl -s $FLUKSIO/flows/house/nodes/decide/source -H "Authorization: Bearer $TOKEN" -curl -X PUT $FLUKSIO/flows/house/nodes/decide/source -H "Authorization: Bearer $TOKEN" \ - -H 'Content-Type: application/json' -d '{"code": "def process(x):\n return {\"y\": x * 2}\n"}' +curl "$FLUKSIO/flows/house/nodes/decide/source" \ + -H "Authorization: Bearer $TOKEN" + +curl -X PUT "$FLUKSIO/flows/house/nodes/decide/source" \ + -H "Authorization: Bearer $TOKEN" \ + -H 'Content-Type: application/json' \ + -d '{"code": "def process(x):\n return {\"y\": x * 2}\n"}' ``` -Writing a source is a superuser's, and so is any `PUT /flows/{name}` that -adds a Function or Command node, retypes or repoints one, changes a Command -node's settings, or changes any node's `resources.env`: each of them is code -the engine runs. Anyone else gets a 403 and keeps the rest — wiring, titles, -a Function node's settings and every other node. +Writing code requires a superuser. So does adding, retyping, or repointing +Function and Command nodes, changing Command settings, or changing +`resources.env`. Other users can edit wiring, titles, and ordinary settings. -`POST .../share` promotes a node's source to the shared library; -`POST .../unshare` gives it a private copy back. `GET /flows/library` lists the -shared sources and which flows use each. +Ordinary source edits stay in the draft until publishing. Shared library +source changes affect its users immediately. +`POST /flows/{name}/nodes/{id}/share` promotes a source to the library; +`/unshare` makes it private again. `GET /flows/library` lists shared sources. ### Running and controlling -| Method | Path | What | +| Method | Path | Action | |---|---|---| -| `POST` | `/flows/{name}/start` · `/stop` | activate or tear down its subscriptions and schedules | -| `POST` | `/flows/{name}/pause` · `/resume` | hold messages instead of running them | -| `POST` | `/flows/{name}/step` | release exactly one held message | -| `POST` | `/flows/{name}/validate` | the issues, without saving | -| `POST` | `/flows/{name}/run` | run every node once, from the values you pass | -| `POST` | `/flows/{name}/nodes/{id}/trigger` | feed values into one node | -| `POST` | `/flows/{name}/nodes/{id}/cancel` | kill the worker running it right now | -| `POST` | `/flows/{name}/nodes/{id}/acknowledge` | clear the failure the canvas is marking | +| `POST` | `/flows/{name}/start`, `/flows/{name}/stop` | activate or stop a live flow | +| `POST` | `/flows/{name}/pause`, `/flows/{name}/resume` | hold or release incoming work | +| `POST` | `/flows/{name}/step` | release one held message | +| `POST` | `/flows/{name}/validate` | validate the current draft | +| `POST` | `/flows/{name}/run` | run the current draft once with `{"inputs": {...}}` | +| `POST` | `/flows/{name}/nodes/{id}/trigger` | feed a node with `{"values": {...}}` | +| `POST` | `/flows/{name}/nodes/{id}/cancel` | stop active node code | +| `POST` | `/flows/{name}/nodes/{id}/acknowledge` | clear its failure marker | -`POST /flows/{name}/run` on a **batch** flow submits a run instead, because -that is what running one means: the parameters, the series and the result are -the point. +A preview run can have real side effects. On a batch flow, `/run` submits a +recorded run; use the run endpoints below for parameters and tracking. ## Messages ```sh -curl -s $FLUKSIO/messages/ -H "Authorization: Bearer $TOKEN" -curl -X POST $FLUKSIO/messages/house.setpoint -H "Authorization: Bearer $TOKEN" \ +curl "$FLUKSIO/messages/" -H "Authorization: Bearer $TOKEN" +curl -X POST "$FLUKSIO/messages/house.setpoint" \ + -H "Authorization: Bearer $TOKEN" \ -H 'Content-Type: application/json' -d '{"value": 22.5}' -curl -s $FLUKSIO/messages/house.temperature/history -H "Authorization: Bearer $TOKEN" +curl "$FLUKSIO/messages/house.temperature/history" \ + -H "Authorization: Bearer $TOKEN" ``` -`GET /messages/` is the catalogue: every message any published flow declares, -with its last value. Publishing puts a value into the graph exactly as a -dashboard control does, which means only a message some flow *declares* can be -published to. Flows own the namespace; everything else is a client of it. +The catalogue contains messages declared by published flows and their last +values. You can publish only to declared messages, just as dashboard controls do. ## Runs -| Method | Path | What | +| Method | Path | Action | |---|---|---| -| `POST` | `/runs/flows/{name}` | queue one run — `{"params": {...}, "seed": 7, "draft": false, "no_cache": false}`. `"cause"` says where it came from — `api` (the default), `cli` or `sdk` | -| `POST` | `/runs/flows/{name}/sweep` | queue many, sharing a `group_id` | -| `POST` | `/runs/import` | record runs executed somewhere else, up to 200 per request. Answers `[{"external_id", "id", "created"}]` | -| `GET` | `/runs` | the queryable history: `?flow=`, `?status=`, `?group=`, `?digest=`, `?since=`, `?before=`, `?limit=`, `?offset=` | -| `DELETE` | `/runs?flow=` | every run of one flow, and everything hanging off them. `flow` is required. Refused with 409 while one of them is still going | -| `GET` | `/runs/overview` | one row per flow that has runs, with how many are running or queued | -| `GET` | `/runs/export/metrics?…&name=&stride=&format=` | every selected run's series as one long table: `run, name, step, ts, value` | -| `GET` | `/runs/export/runs?…¶ms=&metrics=&format=` | one row per run: its inputs as columns, its final numbers, its status and provenance | -| `GET` | `/runs/metrics/names?…` | every metric name the selected runs recorded, distinct; takes the export's own filters | -| `GET` | `/runs/{id}` | one run in full: params, result, per-node record, artifacts | -| `POST` | `/runs/{id}/cancel` | stop it | -| `DELETE` | `/runs/{id}` | forget it, with its nodes, series and artifact rows. Cancel a live one first | -| `POST` | `/runs/{id}/retry` | run the same thing again, as a new run naming this one | -| `GET` | `/runs/{id}/metrics?name=&stride=` | one metric's series, in step order; every series of the run without `name` | -| `GET` | `/runs/series/compare?ids=a,b,c&metric=&x=` | that metric across several runs. `x` is what to plot against: nothing or `step`, `time` (seconds since each run's own first reading), or another metric's name, joined on the step the two share | +| `POST` | `/runs/flows/{name}` | queue a run with `{"params": {...}, "seed": 7}` | +| `POST` | `/runs/flows/{name}/sweep` | queue runs with a shared group | +| `POST` | `/runs/import` | import up to 200 completed runs | +| `GET` | `/runs` | query history | +| `GET` | `/runs/overview` | summarize runs per flow | +| `GET` | `/runs/{id}` | read parameters, results, nodes, and artifacts | +| `POST` | `/runs/{id}/cancel`, `/runs/{id}/retry` | cancel, or create a retry | +| `DELETE` | `/runs/{id}` | delete a finished run and its records | +| `DELETE` | `/runs?flow={name}` | delete a flow's runs; refused while any are active | +| `GET` | `/runs/{id}/metrics?name=&stride=` | read metric series | +| `GET` | `/runs/metrics/names` | list metric names across a selection | +| `GET` | `/runs/series/compare?ids=a,b,c&metric=&x=` | compare curves by step, time, or another metric | +| `GET` | `/runs/export/metrics`, `/runs/export/runs` | stream CSV or JSONL | -Submitting answers immediately with a `queued` run. Wrong parameters (an -undeclared name, a value of the wrong type) come back as a 422 naming the -problem, before anything executes. +Submissions return a queued run immediately. Invalid or undeclared parameters +return `422` before execution. Optional `draft` and `no_cache` flags select +unpublished code or skip stage caching. -`/runs/import` takes `{"runs": [...]}`. Each entry carries `flow`, -`external_id`, `params`, `seed`, `status` (`ok`, `error` or `cancelled`), -`created_at`, `started_at`, `finished_at`, `result`, `metrics` (`name`, -`step`, `ts`, `value`), `artifacts` (`name`, `filename`, `media_type`, -`content_base64`), `labels` and `group_id`. The parameters are completed and -checked as a submit's are, the run is stored with the cause `import` and an -empty commit and code digest, and nothing is queued. The request is checked -whole before anything is written, and an `external_id` already imported -answers with that run and `"created": false`. +History filters include `flow`, `status`, `group`, `digest`, `since`, +`before`, `limit`, and `offset`. Rows arrive newest first; use the last +row's `created_at` as `before` for stable pagination. -`?digest=` filters by the hash of the parameters, which is how you find "every -run that used exactly this configuration". +Exports also accept `ids` and `until`, with `format=csv` or `jsonl`. +The metrics export is a long table; the runs export is one row per run. +Use `params` and `metrics` to choose columns, including dotted paths into +record outputs. -`?before=` is how a long history is paged: rows come newest first, so handing -back the last row's `created_at` reads the next page whatever landed -meanwhile, which `?offset=` cannot, since a run submitted between two pages -shifts every row down one. `?since=` is inclusive and bounds the other end. - -Both exports stream `csv` (the default) or `jsonl`, and take the selection the -history takes plus `?ids=a,b,c`, `?since=` and `?until=`. `export/runs` puts -the inputs that *vary* across the selection in `param.` columns (the sweep -axis) unless `?params=` names them, and the run's numbers in `metric.` -columns. Both are dotted paths into whatever a node returned, so -`?metrics=final_metrics.train_loss` selects a field of a record and the -defaults reach every number inside one. A key with a dot in its own name, such -as `-1.0`, cannot be addressed that way and gets no column. The run id is on every row of both, which is what makes an exported -file a join back to the run rather than a loose table. - -`compare` answers in the same shape a chart widget draws, so three training -curves side by side is a widget binding rather than a screen of its own. - -See [Runs: pipelines that finish](../concepts/runs.md). +Import takes `{"runs": [...]}`, records completed results without executing +code, and deduplicates by `external_id`. See +[Runs](../concepts/runs.md) for examples, artifacts, caching, and exports. ## Artifacts ```sh -curl -X PUT $FLUKSIO/artifacts -H "Authorization: Bearer $TOKEN" \ - --data-binary @model.pt -curl -s $FLUKSIO/artifacts/sha256:abc... -H "Authorization: Bearer $TOKEN" -o model.pt +curl -X PUT "$FLUKSIO/artifacts?name=model.pt" \ + -H "Authorization: Bearer $TOKEN" --data-binary @model.pt +curl "$FLUKSIO/artifacts/" \ + -H "Authorization: Bearer $TOKEN" -o model.pt ``` -Content-addressed, so uploading the same bytes twice stores them once. Node -code normally reaches these through `fluksio.save_artifact` / -`fluksio.load_artifact` rather than here. +The upload returns a reference with the content digest. Identical bytes are +stored once. Node code uses `save_artifact` and `load_artifact` instead. ## Dashboards and panels -| Method | Path | What | +| Method | Path | Action | |---|---|---| -| `GET` | `/dashboards/` | every dashboard, without its contents | -| `GET` | `/dashboards/{name}?draft=true` | the published document, or the editor's copy | -| `POST` | `/dashboards/{name}` | create one | -| `POST` | `/dashboards/from-flow/{flow}` | build `{flow}_results` from a batch flow's declared ports — a chart per streamed metric, a stat per output. 409 if it already exists | +| `GET` | `/dashboards/` | list dashboards | +| `GET` | `/dashboards/{name}?draft=true` | read a draft; omit the query for published content | +| `POST` | `/dashboards/{name}` | create | +| `POST` | `/dashboards/from-flow/{flow}` | create a batch flow's results dashboard | | `PUT` | `/dashboards/{name}` | save a draft | -| `POST` | `/dashboards/{name}/publish` · `/discard` · `/rename` | | -| `GET` `PUT` | `/panels/` | which device shows which dashboards | -| `POST` | `/panels/pair` | start a pairing | -| `GET` | `/panels/pair/{code}` | what is holding a code | -| `POST` | `/panels/{id}/unpair` | drop this panel's credential, keep the panel | +| `POST` | `/dashboards/{name}/publish`, `/dashboards/{name}/discard`, `/dashboards/{name}/rename` | publish, discard, or rename | +| `GET`, `PUT` | `/panels/` | read or update panel assignments | +| `POST` | `/panels/pair` | start device pairing | +| `GET` | `/panels/pair/{code}` | poll pairing with its `secret` | +| `POST` | `/panels/{id}/pair` | approve the pairing | +| `POST` | `/panels/{id}/unpair` | revoke the panel credential | -## Secrets, modules, alerts +## Secrets, modules, and alerts -```sh -curl -s $FLUKSIO/secrets/ -H "Authorization: Bearer $TOKEN" # names only -curl -X PUT $FLUKSIO/secrets/influx-token -H "Authorization: Bearer $TOKEN" \ - -H 'Content-Type: application/json' -d '{"value": "..."}' - -curl -s $FLUKSIO/modules/ -H "Authorization: Bearer $TOKEN" -curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \ - -H 'Content-Type: application/json' -d '{"requirements": "numpy>=2\n"}' - -curl -s $FLUKSIO/alerts/config -H "Authorization: Bearer $TOKEN" -curl -X POST $FLUKSIO/alerts/test/my-phone -H "Authorization: Bearer $TOKEN" -``` - -Secrets are write-only over the API: you can list names and set values, never -read one back. Secrets and `POST /modules/apply` are a superuser's, and so is -what a connector keeps across restarts: `GET` or `DELETE -/connectors/{node_type}/state`, or `DELETE /connectors/{node_type}/state/{key}` -for one key (see [the connector contract](../reference/connector-contract.md#keeping-state-across-restarts)). - -## Observability - -| Path | What | -|---|---| -| `/observability/summary` | engine health — always 200, degraded or not — and the `version` it is running | -| `/observability/timeseries` | executions and failures over a window | -| `/observability/flows` | per-flow rollups with a 60-slice trend | -| `/observability/runs` | recent cascades, with `?flow=`, `?since=`, `?until=` | -| `/observability/events?kind=failure\|audit` | what went wrong, or who changed what; narrows by `?flow=`, `?run=`, `?since=`, `?until=` | -| `/observability/dead-letter` | work the engine gave up on | - -`GET /utils/health/` is the deep health check the container probe uses: it -fails when the event loop is wedged or the state backend is gone, not just when -the process is dead. - -## Workers - -| Method | Path | What | +| Method | Path | Action | |---|---|---| -| `GET` | `/workers` | what is attached, its labels and how busy it is | -| `POST` | `/workers/tokens` | mint a worker credential with its name and labels (superuser; shown once) | -| `GET` | `/workers/tokens` | the worker credentials minted, without the tokens (superuser) | -| `POST` | `/workers/tokens/{jti}/revoke` | refuse one from now on and drop a worker attached with it (superuser) | -| `GET` | `/workers/runtime` | the node runner's source, for a host without pip | -| `WS` | `/workers/attach` | where a worker dials in, with `Authorization: Bearer ` | +| `GET` | `/secrets/` | list names only | +| `PUT` | `/secrets/{name}` | set `{"value": "..."}` | +| `GET` | `/modules/` | read installed packages and the manifest | +| `POST` | `/modules/apply` | replace the manifest with `{"requirements": "numpy>=2\n"}` | +| `GET`, `PUT` | `/alerts/config` | read or update alert configuration | +| `POST` | `/alerts/test/{channel}` | test a channel | -See [Remote workers](workers.md). +Secrets and package changes require a superuser. Applying a manifest replaces +the whole managed environment's package list. Adopted environments are read-only +here; install their packages yourself. + +Connector state keys can also be listed and cleared by a superuser; see +[the connector contract](../reference/connector-contract.md#keeping-state-across-restarts). + +## Observability and workers + +| Method | Path | Action | +|---|---|---| +| `GET` | `/observability/summary` | health and engine version; always returns `200` | +| `GET` | `/observability/timeseries`, `/observability/flows` | execution metrics or flow rollups | +| `GET` | `/observability/runs`, `/observability/events`, `/observability/dead-letter` | cascades, events, or abandoned work | +| `GET` | `/utils/health/` | deep health check; fails on loop or state-backend problems | +| `GET` | `/workers`, `/workers/resources` | attached workers and resource availability | +| `GET`, `POST` | `/workers/tokens` | list or mint worker credentials; superuser only | +| `POST` | `/workers/tokens/{jti}/revoke` | revoke a worker | +| `GET` | `/workers/runtime` | read node-runner source | +| `WS` | `/workers/attach` | attach with a worker token | +| `GET`, `POST` | `/flavors/` | list or create resource sizes | +| `PATCH`, `DELETE` | `/flavors/{name}` | update or delete a size | + +Changing flavors requires a superuser. See [Remote workers](workers.md). ## Live events -```text -ws://127.0.0.1:8000/api/v1/flows/ws -``` +Connect to `ws://127.0.0.1:8000/api/v1/flows/ws` with an +`Authorization: Bearer ` header. -A client that can set headers sends `Authorization: Bearer ` on -the handshake. A browser cannot, so it first calls `POST /flows/ws-ticket` and -opens `/flows/ws?ticket=`: a ticket is good for one connect, within 30 -seconds, so the URL that access logs keep is worth nothing afterwards. -`?token=` is still accepted for one more release. The socket sends a snapshot -on connect and then every engine event: node executions, values published, -health changes, run started and finished. This is what the canvas and the -dashboards draw from. +Browsers first call `POST /flows/ws-ticket`, then connect with +`?ticket=`. Tickets are single-use and expire after 30 seconds. +The socket sends a snapshot followed by node, value, health, and run events. -## Errors +## Errors and client generation -| Code | Means | +| Code | Meaning | |---|---| -| `400` | the request was malformed, or the node refused it with a message | -| `401` / `403` | not signed in, or not allowed | -| `404` | no such flow, dashboard, run or message | -| `409` | someone else saved first — the body carries `current_version` | -| `422` | a parameter, port or binding did not typecheck | -| `503` | that subsystem is not available on this instance | +| `400` | malformed request or refused operation | +| `401` / `403` | missing authentication or permission | +| `404` | resource not found | +| `409` | version conflict, duplicate, or resource still in use | +| `413` | artifact upload exceeds the configured limit | +| `422` | invalid parameters, ports, or bindings | +| `503` | subsystem unavailable | -A 409 on a save or a publish is not an error to retry blindly: it means the -stored version moved past the one you were editing. Re-read, merge, save again. +For a version conflict, read `current_version`, reload, merge your edits, and +save again. Other `409` responses describe their own conflict. -## Generating a client - -The frontend's TypeScript client is generated from the OpenAPI schema -(`make generate-client`). Any OpenAPI generator will do the same for your -language: point it at `/api/v1/openapi.json` on a non-production instance, -from the machine it runs on. +`make generate-client` regenerates the frontend SDK. To generate a client in +another language, use the instance's `/api/v1/openapi.json` schema. diff --git a/docs/code/cli.md b/docs/code/cli.md index d78abaa..301d7cb 100644 --- a/docs/code/cli.md +++ b/docs/code/cli.md @@ -1,467 +1,261 @@ # The `fluksio` command +Install on Linux or macOS with Python 3.12 or newer: + ```sh pip install fluksio -``` - -Installs the engine and the `fluksio` command. Python 3.12 or newer, Linux or -macOS. - -The MQTT and InfluxDB connectors, outbound mail and error reporting are -`pip install 'fluksio[server]'`: a deployment talking to devices wants them, -and a laptop waiting on them to install does not. Everything else, the engine -and every python node included, is in the plain install; a node type whose -library is missing says which extra to add when one is actually built. - -There is a second, smaller distribution, `fluksio-worker`, for a machine that -should only *run nodes* for an engine elsewhere. It has none of the engine in -it. See [Remote workers](workers.md). - -The command is two things at once: `serve`, `enroll` and `worker` *are* an -instance, while `login`, `sync`, `run`, `runs`, `artifacts`, `sweep` and -`status` talk to one that may be anywhere. - -## Where an instance lives - -`.fluksio` beside your code, found the way `.git` is: from the working -directory, or any directory above it. Two repositories on one machine are -therefore two engines, with their own flows, runs and token. `fluksio serve` -makes one where there is none, and it ignores itself from within with a -`.gitignore` of `*`, so a database and a credential cannot be committed by -accident. - -`--global` uses `~/.fluksio` instead, shared by every directory. `--data-dir` -(or `FLUKSIO_HOME`) names any directory outright and wins over both. - -## `fluksio serve` - -Runs the engine. - -```sh fluksio serve ``` -On the first start it creates an admin account and prints its password **once**. -Nothing else has to be running: no database server, no message broker, no -Docker. +For the browser dashboard, sign up at [fluksio.com](https://fluksio.com), add +an instance, then run `fluksio enroll ` in the same directory. -At a terminal this opens a dashboard with the engine running under it; see -[below](#the-dashboard). `--plain` prints the log stream instead, which is -also what happens with no terminal: in a container, under systemd, or in CI. +Use `fluksio --help` or `fluksio COMMAND --help` for command options. +`pip install 'fluksio[server]'` adds MQTT, InfluxDB, outbound mail and error +reporting. A machine that only executes remote nodes can install +[`fluksio-worker`](workers.md). -The default port moves out of the way when something already has it (8001, -8002, and so on) and says which one it took; the URL written to -`client.json` is the one it is actually on. A port you *asked* for is never -moved off: `--port 9000` on a taken 9000 fails, because something else is -there and you named it. It fails before `client.json` is written, so every -client keeps talking to the engine that is actually up. +## Where an instance lives -What it will *not* do is start a second engine for the same instance. Before -anything else, `serve` looks for an engine already serving this directory — -the pidfile beside the data says where, and a token this directory's key -signed says whether what answers there is ours — and stops if it finds one, -whatever port the second was asked for. One SQLite database wants one engine, -and the second would repoint `client.json` at a port that dies with it. -Another instance's Fluksio on the port is named, and the move happens as usual. +By default, Fluksio finds the nearest `.fluksio/` directory by walking up from +your working directory. `serve` creates one in the current directory if needed. +Separate project directories can therefore hold separate instances. -| Option | Default | What it does | -|---|---|---| -| `--data-dir PATH` | `./.fluksio` (or `$FLUKSIO_HOME`) | where this instance keeps everything | -| `--host HOST` | `127.0.0.1` | what to bind | -| `--port PORT` | `8000`, or the next free one | what to listen on | -| `--plain` | off at a terminal | the log stream rather than the dashboard | -| `--log-level LEVEL` | `info` | uvicorn's log level | -| `--admin-email ADDR` | `admin@example.com` | the account created on first run | -| `--admin-password PW` | generated | set it instead of having one generated | -| `--enroll CODE` | — | pair with a portal as part of coming up | -| `--portal URL` | — | the portal `--enroll` redeems at | -| `--max-runs N` | 4 | batch runs driven at once (`FLOW_MAX_RUNS`) | -| `--max-cascades N` | 4 | cascades in flight at once (`FLOW_MAX_CASCADES`) | -| `--max-workers N` | 4 | python worker processes (`FLOW_MAX_WORKERS`) | -| `--gpus N` | counted | GPUs on this machine a node may be given (`FLOW_GPUS`) | -| `--ram MB` | worked out | memory on this machine a node may be given (`FLOW_RAM`) | -| `--worker-max-rss MB` | 0 (never) | retire a warm worker holding more when its node returns (`FLOW_WORKER_MAX_RSS_MB`) | +`--global` selects `~/.fluksio`. `--data-dir PATH` or `FLUKSIO_HOME` +selects a directory explicitly. The data directory contains a `.gitignore` +that excludes its database, credentials and other contents from your project. -Cards are counted from NVIDIA's device nodes (`/dev/nvidia0`, `/dev/nvidia1`, …), -which asks no vendor tool and so keeps the one dependency from becoming two. -Anything they do not cover — another vendor, or a card this process cannot see -— reports none until `--gpus` says otherwise, and a node asking for one is then -clamped to zero and runs alongside every other. `--gpus 1` is what serialises -them. Passing a number always wins over the count. +## `fluksio serve` -`--enroll` with `--portal` is the one-command setup: it pairs before the engine -starts, so the connection is dialled as part of coming up rather than needing a -restart. It is skipped if the instance is already enrolled. +Starts the API and flow engine. No external database, broker or Docker is +needed for the default setup. -!!! warning "One process" - - `fluksio serve` holds the flow engine. A second one is a *second engine*: - duplicated subscriptions, duplicated cron ticks, two webhooks answering the - same path. Run one, and distribute work with - [workers](workers.md) instead. - -!!! note "`$HOME` on a cluster" - - A login node's home directory is often NFS, where SQLite's write-ahead log - does not work, since the database would be locked or corrupt. `fluksio serve` - warns when it notices; point `--data-dir` at local disk. - -### What it prints - -```text -Created the admin account admin@example.com - password: example-password - Shown once. Change it from the dashboard. -Fluksio 0.1.0 — data in /home/you/.fluksio - API http://127.0.0.1:8000/api/v1 - No portal. Pair this instance with: - fluksio enroll +```sh +fluksio serve +fluksio serve --plain --port 9000 ``` -An enrolled instance says which portal it is on instead, and notes that the -dashboard is served from there rather than here. +The first start prints an admin password once. Local CLI commands receive a +saved token automatically; `login` is for connecting to another engine. + +The default port is 8000, advancing to the next free port if necessary. +An explicit `--port` fails when occupied. The startup banner and saved client +configuration show the selected URL. + +Run one engine per instance. At a terminal, a second `serve` reattaches to +that instance's running engine; plain mode refuses to start a duplicate. + +| Option | Default | Purpose | +|---|---|---| +| `--data-dir PATH` | Nearest `.fluksio`, or a new one here | Instance storage | +| `--global` | Off | Use `~/.fluksio` | +| `--host HOST` | `127.0.0.1` | Bind address | +| `--port PORT` | `8000`, or next free | Listening port | +| `--plain` | Off at a terminal | Print logs instead of opening the terminal dashboard | +| `--log-level LEVEL` | `info` | Uvicorn log level | +| `--admin-email ADDR` | `admin@example.com` | First admin account | +| `--admin-password PW` | Generated | First admin password | +| `--enroll CODE` | Unset | Enroll before starting, if not already paired | +| `--portal URL` | `https://hub.fluksio.com` | Portal used by `--enroll` | +| `--max-runs N` | `4` | Concurrent batch runs | +| `--max-cascades N` | `4` | Concurrent live cascades | +| `--max-workers N` | `4` | Local Python worker processes | +| `--gpus N` | Auto-detected | Available local GPUs | +| `--ram MB` | Auto-detected | Available local memory | +| `--worker-max-rss MB` | `0` (disabled) | Retire a worker above this resident memory after its call | + +Resource defaults can also come from the corresponding `FLOW_*` settings; +see [Configuration](../reference/configuration.md). GPU detection uses +NVIDIA device nodes. Set `--gpus` when your hardware is not detected. + +Use local disk for the data directory on a cluster: SQLite's write-ahead log +does not support NFS. ### The dashboard -At a terminal, `serve` opens three tabs: **Overview** is the health block and -the flows, **Runs** is the history, and **Logs** is the engine's own output. +At a terminal, `serve` opens **Overview**, **Runs** and **Logs**: -| Key | What it does | +| Key | Action | |---|---| -| `1` `2` `3` | the Overview, the Runs and the Logs | -| `q` | close the dashboard. **The engine keeps running**, and the pid is printed | -| `s` | stop the engine, or start it again | -| `r` | restart it | -| `e` | pair with a portal, without leaving the screen | +| `1`, `2`, `3` | Switch tabs | +| `q` | Close the dashboard; the engine keeps running | +| `s` | Stop or start the engine | +| `r` | Restart the engine | +| `e` | Enroll with a portal | -On the Runs tab the toolbar carries what a run is for: +On **Runs**, `space` selects runs, `enter` opens a chart or comparison, +`c` cancels a run, and `a` lists or downloads artifacts. Compare curves by +step, elapsed time or another metric. -| Key | What it does | -|---|---| -| `space` | tick the run under the cursor into a comparison | -| `enter` | compare the ticked runs, or draw the one under the cursor | -| `c` | cancel the run the cursor is on | -| `a` | list what the run left behind, and fetch it | - -The engine is a child process rather than a thread, which is what makes those -possible, and what makes `q` a way out of the screen rather than a way to -stop the engine. Running `fluksio serve` again reattaches to it. - -Its output goes to `serve.log` in the data directory rather than down a pipe, -which is what lets the screen be closed while the engine keeps running — a -pipe with nobody reading it breaks the next line the engine writes, and a -node's `print` is one of those. It also means the Logs tab has the output of -an engine this screen only adopted, and the scrollback of the one before it. -The engine cuts the file back to nothing once it passes 5 MB, so the bound -holds whether or not a screen is open. - -The screen takes its colours from the terminal rather than painting its own, -so it sits inside a light profile as readily as a dark one. - -An engine started elsewhere is adopted rather than duplicated, and can be -stopped from here only when it is this instance's own: both the pidfile -beside the data and a token this directory's key signed have to agree. Another -instance's engine is named and left alone. - -The screen subscribes to the engine's event bus over the same websocket a -browser uses, so a run appears the moment it starts. It reconnects on its own, -and an engine stopped from this screen is a normal state rather than an error. - -#### Comparing runs - -`enter` opens a comparison of the ticked runs: one metric's curve for each, -over a table of what differs between them. - -The curves are drawn in braille, five to a chart, in five distinct colours: a -departure from the browser's chart palette, where the five series -step one hue by lightness. Hue is what survives a terminal. Every curve is -named in the legend either way. - -Two pickers sit above the chart. The first is the metric, named as the run -records it (`train.loss` rather than `loss`, since a name is qualified by the -node that published it). The second is what it is plotted against: the step, -`time (s)` measured from each run's own first reading so runs started hours -apart lie on top of each other, or another metric of the same runs. - -The table below keeps only what actually differs (the inputs whose values are -not the same in every run, and the seed when it varies) beside each run's -status, duration and final numbers. A parameter every run shared is not a -column worth scanning. - -A run that is still going has its curve re-read once a second, and the whole -comparison refreshes when the engine says a run finished. +Run `fluksio serve` again to reopen the dashboard. Its engine writes to +`serve.log` in the data directory; the log is truncated after 5 MB. +Without a terminal, or with `--plain`, the command prints logs directly. ## `fluksio enroll` -Pairs an existing instance with a portal. +Connects this instance to the portal: ```sh fluksio enroll ABCD-1234 ``` -| Option | What it does | +Get a code from **Instances → Add instance** in the portal. Run the command +from the instance's project directory, or select it explicitly. + +| Option | Purpose | |---|---| -| `--portal URL` | a portal of your own, instead of `https://hub.fluksio.com` | -| `--as EMAIL` | the local account a portal session arrives as | -| `--data-dir PATH` | which instance, if not the one this directory is in | +| `--portal URL` | Use another portal instead of `https://hub.fluksio.com` | +| `--as EMAIL` | Choose the local superuser represented by portal sessions | +| `--data-dir PATH` | Select instance storage | +| `--global` | Select `~/.fluksio` | -Get the code from the portal under **Instances → Add instance**. It is -single-use and expires in fifteen minutes. `--as` matters when the instance -has several superusers. Without it, enrolment refuses rather than guessing. +With multiple local superusers, `--as` is required. A new code replaces an +existing enrollment only after the portal accepts it. A running engine picks +up the change and reconnects automatically. -An instance that is already paired is re-paired: a new claim code replaces the -connection, at the same portal or another one, and a running engine drops the -old tunnel within a few seconds. The code is redeemed before anything is -written, so one the portal rejects leaves the connection as it was. Moving to -a different portal clears the local accounts' mappings to the old one, since -nothing in a mapping says which portal issued it — the people who had access -are admitted again from the new portal. - -Afterwards, `fluksio serve` dials the portal as it comes up, and keeps dialling: -a portal that restarts, a wifi that changes, a laptop that suspends and wakes -somewhere else all end the same connection, and the link is put back up without -anybody noticing. A connection that stood up and then dropped is retried at -once; one that never stood up waits a little longer each time, up to half a -minute. See [Accounts and the portal](../interface/portal.md). +The instance keeps an outbound connection to the portal and reconnects after +network interruptions. See [Accounts and the portal](../interface/portal.md). ## `fluksio worker` -Runs nodes for an engine elsewhere. Everything after `worker` belongs to the -agent's own parser: it is the same program `fluksio-worker` installs, so the -two are interchangeable: +Starts the same agent as the standalone `fluksio-worker` command: ```sh -fluksio worker --url wss://api.${DOMAIN}/api/v1/workers/attach --labels gpu +FLUKSIO_WORKER_TOKEN=your-token fluksio worker \ + --url wss://api.example.com/api/v1/workers/attach --labels gpu ``` -See [Remote workers](workers.md). +Create the worker token on the engine first. See [Remote workers](workers.md) +for token creation, interpreter selection and labels. ## Talking to an engine -The commands below are the client half: they run wherever you work, and -address an engine over its API rather than being one, except under `--local`, -which boots one inside the command instead. +Client commands use the nearest `.fluksio/client.json`, or the global +configuration when there is no project instance. Legacy client configuration +is a fallback. +`FLUKSIO_URL` and `FLUKSIO_TOKEN` override saved values; command options +`--url` and `--token` override both. ### `fluksio login` ```sh -fluksio login --url https://api.${DOMAIN} +fluksio login --url https://api.example.com ``` -For an engine somewhere *else*. One you started yourself needs no login: -`fluksio serve` writes the token as it comes up and says where it put it. - -The token goes in this project's `.fluksio/client.json`, or with `--global` in -`~/.fluksio/client.json`. Every command below reads it from there, nearest -first walking up from the working directory, or from `FLUKSIO_URL` and -`FLUKSIO_TOKEN`, or from its own `--url` and `--token`. A token an older -version wrote to `~/.config/fluksio/client.json` is still read. +Prompts for credentials and saves an API token for a remote engine. +`--global` saves it in `~/.fluksio` instead of this project. Your locally +started engine already saves a token. ### `fluksio sync` ```sh -fluksio sync [PATH_OR_MODULE ...] # default: the current directory +fluksio sync study +fluksio sync examples/myresearch +fluksio sync --dry-run study ``` -Imports what you name, collects the flows the decorators declared, and uploads -each one with a generated import shim per node. A directory that is a package -is walked; a dotted name is imported as it stands; nothing is loaded from a -file path, because the shim has to import the same way. +Imports modules, packages or directories and uploads the declared flows. +With no target, it searches the current directory. Directory searches skip +hidden folders, virtual environments, `__pycache__` and `node_modules`. +File paths are not targets: use `study`, not `study.py`. -A plain directory is walked all the way down, so one folder per study -(`fluksio sync dev` over `dev/s1_baseline/study.py`) needs no naming. Hidden -directories, `__pycache__`, `node_modules` and virtualenvs are left alone. +The generated node bodies import your functions. Keep the project available +to each machine running them; sync does not copy your package to a remote +worker. Uploading source requires a superuser token. -Each file is imported under the name its path spells beneath the directory -being synced, so `dev/s1/study.py` and `dev/s2/study.py` are `s1.study` and -`s2.study` and a `study.py` per study collides with nothing. No `__init__.py` -is needed, since the directories in between are namespace packages, which leaves a -bare `from study import ...` in a test beside it working. A file at the top of -what is synced keeps its plain name. - -| Flag | What it does | +| Flag | Purpose | |---|---| -| `--dry-run` | print the flow documents and shims, upload nothing | -| `--no-publish` | leave the upload as a draft | -| `--force` | overwrite a flow, or a node body, that was edited on the canvas | +| `--dry-run` | Print flow documents and node bodies without uploading | +| `--no-publish` | Save changes as drafts | +| `--force` | Overwrite canvas edits that conflict with your local definitions | -Every sync retires the engine's workers, including one that had nothing to -upload: a worker holds your package in memory, so an edit to it is invisible -until the process goes. - -It also records, per node, which of your modules that node's function imports -its way to, and what they hash to. That is what the -[stage cache](../concepts/runs.md#stage-caching) keys on, so editing a -helper a node calls into is reported as that node changing (`train: updated -(flow, fit)`) and re-runs it, while editing something the node never reaches -is left alone. - -Uploading node code takes a superuser's token; anyone else's sync is refused. - -A sync that changed nothing says `unchanged`, which is the answer worth -having. `published` and `draft` are said only when there was something to -publish or a draft was genuinely left behind. An engine too old to store what -a node's code reaches says so in a line naming both versions; until it is -upgraded its cache is keyed on the whole repository, as it was before. See -[Getting started: data science](../getting-started/data-science.md). +Sync records source digests for [stage caching](../concepts/runs.md#stage-caching) +and refreshes local workers, so edited imports are picked up. An unchanged +flow reports `unchanged`. See the +[data science guide](../getting-started/data-science.md) for a complete example. ### `fluksio run` ```sh -fluksio run train --lr 0.05 --seed 7 [--wait] +fluksio run train --lr 0.05 --epochs 20 --follow ``` -Syncs the flow, then submits a run, so the command after an edit is this one -and nothing else, from the repository root as readily as from the study's own -folder. What it imports is the module that declared the flow, as the last -`sync` recorded it, under the same module name. Importing a module runs it, so -the other scripts in the repository are left alone. +Syncs local code, then submits a run. Flags such as `--lr` name the flow's +declared inputs and are converted to their declared types. Unknown inputs are +rejected. Use `--param name=a,b` with `sweep` for multiple values. -A flow the engine has no such record for (never synced, drawn on the canvas, -or on an engine older than the record) syncs the working directory *and -everything under it* instead, skipping whatever the repository's `.gitignore` -covers. A study there that will not import is a warning rather than a stopped -run. `--sync dev/s1_baseline` (repeatable) names what to sync outright, and -`--no-sync` skips it entirely. +After the first sync, `run` imports the module that declared the flow. If no +declaration is recorded, it searches the working directory and skips paths +covered by `.gitignore`. Import errors in that fallback search produce +warnings. Use `--sync PATH` to choose a target explicitly. -Flags that are not its own are the flow's inputs, typed by what the flow -declares them as, so a name the flow has not got is refused by name, and -`--param lr=0.002` is told that one value is `--lr 0.002` and several is a -[sweep](#fluksio-sweep). `--wait` blocks until the run -finishes and exits non-zero if it failed. `--follow` waits as well, and prints -the numbers the run reports as they arrive: +| Flag | Purpose | +|---|---| +| `--wait` | Wait for completion; exit non-zero if the run fails | +| `--follow` | Wait and print emitted metrics | +| `--timeout SECONDS` | Stop waiting after this long; leave the run active | +| `--seed N` | Record the seed and fill a declared input named `seed` | +| `--defaults` | Use declared input defaults without prompting | +| `--sync PATH` | Choose what to sync; repeat for multiple targets | +| `--no-sync` | Run the version already on the engine | +| `--no-cache` | Execute every node | +| `--local` | Start an engine in this command and wait for the result | -```text - train.loss[14] = 3.40295e-06 -``` +With no parameters at a terminal, the command prompts for each input. +Press Enter to keep its default. Scripted commands do not prompt. +Ctrl-C while waiting cancels the run and exits with status 130. -Ctrl-C while either is waiting cancels the run on the engine rather than only -stopping the watching, and exits 130. `--timeout SECONDS` gives up *waiting* -after that long and leaves the run going; it is nothing to do with a node's own -timeout. +A seed does not initialize your libraries' random number generators. Declare +a `seed` input and pass it to your code if it controls randomness. -`--seed N` is the experiment's seed, and it does three things. It is recorded -on the run, so what a result came from is answerable later. It goes into the -digest that identifies a run's inputs, so two runs of one configuration with -different seeds are different runs rather than a cache hit. And if the flow -declares an input named `seed`, that is what fills it, so the number the run -is labelled with is the number your code actually drew from, instead of merely -looking like it. A flow that declares no such input still records it, and -nothing reads it. Sweep over seeds with `--param seed=1,2,3`. - -Any input takes what a previous run produced, named rather than typed out: a -checkpoint, but equally a config object nobody wants to paste into a shell: +Inputs can reference earlier results: ```sh -fluksio run evaluate --dataset @run:1758042000123-9f2ab41c.dataset -fluksio run evaluate --dataset sha256:6dd1f0… -fluksio run train --meta @run:1758042000123-9f2ab41c.dataset_meta +fluksio run evaluate --dataset @run:.dataset +fluksio run evaluate --dataset sha256: ``` -`@run:.` is whatever that run's output was, whole and with its own -type; a bare digest is the content itself, resolved into a reference. Both -spellings are reserved on every input, `str` included, so an input that has to -carry one of them literally cannot. Passing the value as JSON still works and -is what a script that already holds one does, the same thing -`flow.submit(dataset=run.result["dataset"])` does from Python. +`@run:.` resolves a run output; `sha256:` resolves +stored bytes into an artifact reference. -Run a flow with no parameters at a terminal and it asks for them, one line per -declared input, with the declared value in brackets: - -```text -lr (float) [0.05]: 0.01 -epochs (int) [10]: -dataset (artifact): @run:1758042000123-9f2ab41c.dataset -``` - -Enter keeps what is in brackets, so pressing it through the lot runs the -defaults. Nothing changes for a scripted run: passing any parameter, or piping -the command, skips the questions, and `--defaults` skips them explicitly. - -`--no-sync` runs what is already on the engine. Worth it in a tight loop where -you know nothing changed, since syncing retires the workers and the next call -pays its imports again. A directory that declares no flows syncs nothing and -says nothing: a flow drawn on the canvas is run the same way. - -`--no-cache` executes every node, including one an earlier run already -answered. See [Stage caching](../concepts/runs.md#stage-caching). - -`--local` boots the engine inside this process instead of talking to a served -one, so there is no `fluksio serve` terminal to keep open. It is the same -instance either way (the same `.fluksio`, the same database, artifacts and run -history) so a run made this way and a run made through a served -engine cache against each other. It always waits, because the engine it starts -lives exactly as long as the command. Starting one costs a few seconds of -worker pool and module reconcile, against the ~15 ms of submitting to an -engine that is already up: `--local` is for "I just want to run it", not for a -loop you are iterating in. +`--local` uses the same data directory, history and artifacts as `serve`. +It starts and stops the engine for each command, so keep `serve` running +when iterating frequently. ### `fluksio status` ```sh -fluksio status [--watch] +fluksio status +fluksio status --watch ``` -The home screen's top half in a terminal: whether the engine is healthy and -what is wrong if not, whether it is paired with a portal and reaching it, then -every flow with its state, its node count and whether it has unpublished -changes, and the last few runs and failures under them. - -A `resources` line names each machine the engine can run a node on and how much -of it is in use, plus how many nodes are queued for one. It is absent on an -engine that accounts for nothing. - -The portal reads one of three ways. `no portal` means this instance was -never enrolled. `portal hub.fluksio.com` means the link is up. `portal -unreachable` names the error, and is the one worth acting on: the dashboard is -served from the other end, so nobody can reach it while that is showing. - -`--watch` keeps it on screen and refreshes every five seconds until Ctrl-C, the -cadence the dashboard polls at, since nothing here moves faster. It needs a -terminal; without one, run it without `--watch` and the output pipes cleanly. - -`--local` reads the flows and history out of this directory with no engine -served, the same way `runs --local` does. It cannot be watched: an in-process -engine is the command itself, so nothing changes under it. +Shows engine health, portal connectivity, flows, recent runs and available +resources. `--watch` refreshes every five seconds. `--local` reads the local +instance without a served engine. ### `fluksio runs` ```sh -fluksio runs [--flow train] [--limit 20] +fluksio runs --flow train --limit 10 ``` -The runs an engine has recorded, newest first: id, status, flow, duration, how -long ago it was submitted, the commit of the repository it came from, and the -inputs it was given. Statuses are coloured when a terminal is reading the -output: `ok` green, `error` red, `cached` cyan. - -Only the inputs that *differ from what the flow declares* are shown, and they -are clamped to what is left of the terminal's width. A run that took the -defaults lists none at all, and a flow taking a few kB of JSON does not push -everything else off the line. `Client.runs()` and -[`fluksio export runs`](#fluksio-export) are where the whole value is read. -`--local` reads the same history from an in-process engine, without one having -to be served. +Lists recent runs, newest first. Omit `--flow` for all flows. +`--local` reads the local instance. ### `fluksio retry` ```sh -fluksio retry [ ...] +fluksio retry fluksio retry --group ``` -Runs the same thing again as a run of its own, carrying the flow, the inputs, -the seed and the group of the one it repeats, and recording it as its parent. -What it is for is a run an engine restart interrupted: those are marked -`abandoned` once their lease goes stale, and `--group` retries every run of a -sweep that did not end `ok`, so the missing combinations are filled in rather -than the whole grid being submitted again. The stage cache is what makes it -cheap — the nodes that finished are restored rather than recomputed. +Submits a new run with the previous inputs, seed and group, linked to its +parent. `--group` retries finished runs in that group that did not end +`ok`. Active runs must be cancelled before retrying. -A run that has not finished is refused; cancel it first. - -Runs that were still *queued* when an engine stopped need none of this: the -next engine reads them back out of its own history and picks them up. +Queued runs survive engine restarts. Interrupted running runs become +`abandoned` after their lease expires; retry them to resume with any +available cached stages. ### `fluksio flavors` @@ -469,137 +263,96 @@ next engine reads them back out of its own history and picks them up. fluksio flavors ``` -The named sizes a node can ask for, as in `@node(resources="gpu-small")`, with the -cores, memory and cards each stands for. Editing them is the Workers screen or -`POST /api/v1/flavors`; this is the read. +Lists named resource sizes, usable as `@node(resources="gpu-small")`. +Edit them in **Workers** or through the flavors API. ### `fluksio sweep` ```sh -fluksio sweep train --param lr=0.1,0.01 --param epochs=10,50 --wait +fluksio sweep train --param lr=0.01,0.1 --param epochs=10,50 --wait ``` -Every combination of the parameter lists, submitted as one group: four runs -above, sharing a `group_id` and executing in parallel. Values are typed by the -flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`, -`--no-cache` and `--local` mean what they do there. `--wait` blocks until all -of them are finished and exits non-zero if any failed. +Submits every combination of the declared input values: four runs here, +sharing a `group_id`. Runs execute concurrently within the engine's limits. +To vary seeds, the flow must declare a `seed` input; use +`--param seed=1,2,3`. + +`--seed`, `--sync`, `--no-sync`, `--no-cache`, `--timeout` and +`--local` work as for `run`. `--wait` exits non-zero if any run fails. ### `fluksio export` ```sh fluksio export metrics --flow train --list -fluksio export metrics --flow train --name train.train_loss --stride 10 -o curves.csv -fluksio export runs --flow train --status ok > arms.csv +fluksio export metrics --flow train --name train.loss -o curves.csv +fluksio export runs --flow train --status ok -o results.csv ``` -The two tables an analysis reads. `export metrics` is the long one, a row per -run, metric and step, which is what a plotting library takes without -reshaping; `--name` keeps the metrics it lists and `--stride` keeps every Nth -point of *each* curve. `export runs` is the wide one: a row per run with its -inputs as columns, its final numbers, its status, its duration and the commit -and digest of the code it ran. +`metrics` exports a row per run, metric and step. `--list` lists metric +names; `--name A,B` selects them; `--stride N` keeps every Nth point. -Every input the selected runs recorded becomes a column, so the schema does -not move with the selection and a filter written against one export keeps -working on the next; `--params lr,seed` narrows it to the axis a comparison is -read along. `--metrics` narrows the final numbers the same way. +`runs` exports a row per run with inputs, final numeric results, status, +duration and code identifiers. Use `--params A,B` and `--metrics A,B` to +select columns, including dotted paths such as `final_metrics.train_loss`. -An input left out of a submit is recorded at the value the flow declares for -it, so every row says what it was actually run with rather than leaving the -cell blank. +Both support `--flow`, repeated `--run ID`, `--group`, `--status`, +`--since`, `--until` and `--local`. The run ID joins results to metrics. -A node usually returns a record rather than a scalar, so both sides take -dotted paths into one: `--metrics final_metrics.train_loss,test_metrics.known.perfect` -selects three fields rather than two blobs, and `--params model.ansatz` does -the same for an input. The defaults reach the same depth: every number a -result carries becomes a column wherever it sits, and a record's inputs are -taken leaf by leaf rather than as one blob. +Output defaults to CSV on stdout. `-o FILE` writes a file; `--format jsonl` +writes JSON Lines. Parquet requires `pip install 'fluksio[parquet]'` and +`--format parquet -o FILE`. -Metric names are flow-qualified (a node of `train` writing `train_loss` records -`train.train_loss`) so `--list` prints the names the selected runs -carry when the spelling is not obvious. The engine answers it over the whole -selection, so a name only an older run ever recorded is listed too. - -Both take `--flow`, `--run ID` (repeat it), `--group`, `--status`, `--since`, -`--until` and `--local`, and both put the run id on every row: it is the join -back to the run page and to what the run made. - -`--format` is `csv` (the default), `jsonl` or `parquet`; output goes to stdout -unless `-o FILE` names somewhere. Parquet keeps the types, needs pyarrow -(`pip install 'fluksio[parquet]'`) and needs a file to write, since it is not a -stream. In a notebook, `Client.export_metrics()` and `Client.export_runs()` -answer the same rows as a list of dicts, which `pandas.DataFrame` takes -directly. +In Python, `Client.export_metrics()` and `Client.export_runs()` return the +same rows as lists of dictionaries. ### `fluksio artifacts` ```sh -fluksio artifacts 1758042000123-9f2ab41c -fluksio artifacts 1758042000123-9f2ab41c weights -o model.npz +fluksio artifacts +fluksio artifacts train.weights -o weights.json ``` -The files a run produced: what `fluksio.save_artifact(...)` wrote, and any -artifact a node returned. Named alone it lists them: the message each left on, -its size, its media type and the filename the node gave it. Name one and it is -written here, under that filename unless `-o` says otherwise. - -The message name is the one to pass, since it is what addresses the bytes; -`--local` reads them from this directory without an engine served. +Lists a run's returned artifacts, including their message names, sizes and +filenames. Pass a name from that list to download it. Output uses the stored +filename unless `-o` selects another path. `--local` reads the local instance. ## What lives in the data directory -```text -.fluksio/ (or ~/.fluksio, with `--global`) -├── client.json the token `serve` wrote, mode 600 -├── serve.pid the engine serving this directory, while one is -├── serve.log what the engine under the dashboard printed -├── .gitignore `*`, so a database and a credential are ignored from within -├── fluksio.db SQLite: users, runs, metrics, observability, agents -├── flows/ a git repository, one directory per flow -│ ├── house/ -│ │ ├── flow.json the published structure -│ │ ├── nodes/*.py the published node code -│ │ ├── flow.draft.json unpublished edits, if any -│ │ └── nodes.draft/*.py -│ ├── _lib/ shared node sources -│ ├── _dashboards/ dashboards, drafts and all -│ └── requirements.txt what the Modules screen installs -├── artifacts/ content-addressed bytes, two levels deep -├── user-venv/ the interpreter your node code runs on -├── secrets.enc encrypted credentials, deliberately outside flows/ -├── alerts.json alert channels and rules -├── panels.json wall-panel pairings -├── oauth-key.pem signs agent tokens -├── cloud.json the portal enrolment, if there is one -├── secret_key signs sessions and derives the secrets key -└── env optional settings file -``` +The main files and directories are: -Two things follow from this layout and are worth internalising: +| Path | Contents | +|---|---| +| `client.json` | Local API URL and token | +| `fluksio.db` | SQLite database: accounts, runs, metrics and operations history | +| `flows/` | Git repository of flows, node sources, dashboards and module requirements | +| `artifacts/` | Files stored by content digest | +| `user-venv/` | Managed node environment, when used | +| `secrets.enc` | Encrypted credentials, outside the flow repository | +| `cloud.json` | Portal enrollment, when paired | +| `secret_key` | Session signing and encryption key | +| `serve.pid`, `serve.log` | Running engine information and terminal dashboard logs | +| `env` | Optional instance settings | -**`flows/` is a real git repository.** `git log` is the history of every change -anyone made to any flow. A run records the commit it ran at, so `git show` on -that hash is literally the code that produced the number. +The flow repository records edits in Git. Runs record its commit and code +digests; code-defined nodes also depend on your project files, so keep that +project's history too. -**Backing up the data directory backs up the instance.** Everything else is -rebuildable. Copy it while the engine is stopped, or use SQLite's online backup -for the database if it is not. +Back up the data directory while the engine is stopped. An adopted venv and +your project files live outside it and need their own backup. ## Settings -Settings come from the environment, or from an `env` file in the data -directory. The ones you are most likely to touch: +Settings come from environment variables or an `env` file in the data +directory. Common choices: -| Variable | Default | What it does | -|---|---|---| -| `DATA_DIR` | `./.fluksio` via the CLI; `~/.fluksio` with `--global` | everything below it derives from this | -| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL | -| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter | -| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart | -| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links | -| `ENVIRONMENT` | `local` | `production` closes the interactive API schema | -| `MCP_ENABLED` | `false` | opens the [agent endpoint](agents.md) | -| `SECRET_KEY` | generated once | signs sessions, derives the secrets key | +| Variable | Purpose | +|---|---| +| `FLUKSIO_HOME` | Select the CLI's data directory | +| `NODE_VENV` | `auto` adopts an existing venv; `managed` requests a separate one; a path selects an interpreter | +| `REDIS_HOST` | Use Redis for flow state and queues | +| `FRONTEND_HOST` | Browser address used in mail, OAuth metadata and panel links | +| `ENVIRONMENT` | `production` disables interactive API documentation | +| `MCP_ENABLED` | Enable the [agent endpoint](agents.md) | -The full list is in [Configuration](../reference/configuration.md). +See [Configuration](../reference/configuration.md) for defaults and the full +list. diff --git a/docs/code/connectors.md b/docs/code/connectors.md index d6b66f8..732565c 100644 --- a/docs/code/connectors.md +++ b/docs/code/connectors.md @@ -1,146 +1,130 @@ # Writing a connector -A walkthrough from an empty directory to a node on the canvas. The normative -surface is in [the connector contract](../reference/connector-contract.md); -this is how to use it. +A connector adds a device or service integration to the node palette. It is +a Python package with a `ConnectorNode` subclass and an entry point. +This example works without hardware or another repository. -## Start from the skeleton +## Create the package -A working connector, kept as small as a correct one can be, lives at -`skeleton/` in the [connectors repository](https://git.stroblme.de/Fluksio/connectors). -Copy it: +Make this directory beside your project: -```bash -git clone https://git.stroblme.de/Fluksio/connectors.git -cp -r connectors/skeleton fluksio-connector-mydevice -cd fluksio-connector-mydevice +```text +fluksio-connector-random/ + pyproject.toml + src/ + fluksio_connector_random/ + __init__.py ``` -Rename the package (`src/fluksio_connector_skeleton/` and the `[project]` name -in `pyproject.toml`), then point the entry point at your class: +Put this in `pyproject.toml`: ```toml +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "fluksio-connector-random" +version = "0.1.0" +requires-python = ">=3.12" +dependencies = ["fluksio>=0.1,<0.2"] + [project.entry-points."fluksio.node_types"] -my_device = "fluksio_connector_mydevice:MyDevice" +random_sensor = "fluksio_connector_random:RandomSensor" + +[tool.hatch.build.targets.wheel] +packages = ["src/fluksio_connector_random"] ``` -## Install it into the engine - -Connectors are found through installed package metadata, so the engine has to -have yours installed. During development, install it as an editable path -dependency of the backend: - -```bash -cd app/backend -uv add --editable ../../fluksio-connector-mydevice -``` - -Editing your connector's **code** then takes effect on the next engine restart, -with no reinstall. Changing its **metadata** (the entry point, the version, -the package name) needs `uv sync` to regenerate the installed metadata before -a restart picks it up. - -Two things to know about this: - -- `uv add` writes the path dependency into `backend/pyproject.toml` and - `uv.lock`. That path does not exist inside the backend's Docker image, so - **revert both files before committing**; the dependency belongs on your - machine, not in the repo. Distribution is a later milestone. -- For the same reason, develop against the **standalone backend** rather than - the Docker stack: - - ```bash - cd app && make dev-backend # FastAPI on :8000 - make dev-frontend # Vite on :5173 - ``` - -Restart the backend and your node type appears in the add-node palette (⌘P), -labelled with the package it came from. - -## Write the node +Put this in `src/fluksio_connector_random/__init__.py`: ```python -from typing import Any +import random from pydantic import Field + from fluksio.flow.connector import CONTRACT_VERSION, ConnectorNode -class MyDevice(ConnectorNode): +class RandomSensor(ConnectorNode): contract = CONTRACT_VERSION - title = "My device" - description = "Reads temperature and humidity over the device's HTTP API." + title = "Random sensor" + description = "Emits a random reading. No sensor required." class Params(ConnectorNode.Params): - host: str = "192.168.1.50" - api_key: str | None = Field( - default=None, json_schema_extra={"x-secret": True} - ) + ceiling: float = Field(default=1.0, gt=0) - async def poll(self) -> dict[str, Any] | None: - reading = await self._read() - return {"temperature": reading.temp, "humidity": reading.humidity} -``` - -Then, on the canvas: place the node, set `poll_interval` to how often the -device should be read, pick the API key from the secrets store, and add output -ports named `temperature` and `humidity`. The port names are the wiring: any -node consuming those messages is downstream of this one. - -## If the device is told rather than asked - -A connector that commands something implements `write()` instead of, or -alongside, `poll()`. Input ports carry the values in: - -```python - def write(self, **ports: Any) -> None: - self._device.set(ports["level"]) -``` - -Place the node with *input* ports named `level`, and whatever publishes that -message drives the device. Set `idempotent = False` on the class: a write is a -command, and a redelivery after a crash should not undo a newer one. - -Keep the first version off the wire. A boolean setting the code checks before -it sends (`artnet`'s `transmit` is the example) lets a flow be built and -watched in the logs before anything physically moves. - -## If the reading is bytes - -A camera or a microphone publishes a reference rather than the bytes: - -```python async def poll(self): - jpeg = await asyncio.to_thread(self._grab) - return {"frame": self.save_artifact(jpeg, "f.jpg", media_type="image/jpeg")} + return {"reading": random.random() * self.config.ceiling} ``` -Place the node with an `image`-typed output port and a Media widget draws each -frame as it lands. `fluksio-connector-test-media` publishes test frames and -tones this way, so the whole path can be wired up with no camera in the room. -Copy it if yours is a media device. +The entry point name, `random_sensor`, becomes the stored node type. +Choose a unique name; the engine skips names that are already registered. -## Try it without a device +## Install and try it -Give `poll` something predictable first and confirm the values reach the canvas -before pointing it at hardware: +Use the **engine's Python environment**, the one where you installed Fluksio: -```python -async def poll(self): - return {"temperature": 21.5} +```sh +pip install -e /path/to/fluksio-connector-random ``` -Press Run, or wait for the poll interval, and the value appears on the edge. -Anything the connector prints shows up in the logs panel in the dock, and a -raised exception shows there with its traceback and marks the node down. +Restart the engine from its original project directory so it keeps using +the same instance data: -## Checklist before sharing it +```sh +fluksio serve +``` -- `contract = CONTRACT_VERSION` declared on the class itself. -- Credentials marked `x-secret`, never read from the environment. -- `stop()` is safe to call twice, and after a failed `start()`. -- Blocking I/O wrapped in `asyncio.to_thread`. -- `poll()` returns `None` rather than raising when there is simply nothing new. -- `write()` returns quickly; the thread it runs on is the scheduler's. -- Health reported when the connection comes and goes. -- Anything that has to outlive a restart in `self.store`, keyed by device. +Restart an already running engine after installing. In the browser: + +1. Add a **Random sensor** node to a live flow. +2. Set `poll_interval` to `5` seconds. +3. Add an output port named `reading`, type `float`. +4. Publish the flow and watch the value change. + +The palette shows the package name and version. Code edits need an engine +restart; metadata changes also need another editable install. + +For Docker, install the package in the backend image and rebuild it. +The Modules screen installs packages for **node code**, so it does not +install connectors into the engine. + +## Connect a real device + +Replace `poll()` with your device read. Return a dictionary keyed by declared +output port, or `None` when there is nothing new. The base class polls at +`poll_interval`, publishes changed values, and reports failures. + +Keep blocking I/O off the event loop with `asyncio.to_thread`. For pushed +updates, manage the connection in `start()` and `stop()`, leave +`poll_interval=0`, and call `self.inject({...})` when data arrives. + +Mark credential parameters with +`Field(default=None, json_schema_extra={"x-secret": True})`. The editor +then stores a secret reference rather than the credential in the flow. + +To send commands, implement `write(self, **ports)`: it receives values keyed +by input port. Set `idempotent = False` for commands that must not be replayed +after a crash. + +## Media and persistent state + +Use `self.save_artifact(data, name, media_type=...)` for bytes such as camera +frames. Return the reference through an `image`, `audio`, or `video` port. +For disposable live frames, pass `volatile=True`. + +Use `self.store` for short strings that must survive a restart, such as a +pairing token. Include the device address in the key: all nodes from the +same top-level package share this store. + +## Before sharing + +- Declare `contract = CONTRACT_VERSION` on the class itself. +- Return only declared output ports, with values matching their types. +- Mark credentials as secrets. +- Keep `stop()` safe after a failed start and when called twice. +- Report connection health if you manage your own subscription. + +See [the connector contract](../reference/connector-contract.md) for lifecycle, +storage limits, and certificate pinning. diff --git a/docs/code/nodes.md b/docs/code/nodes.md index 3ef2a18..6dee16a 100644 --- a/docs/code/nodes.md +++ b/docs/code/nodes.md @@ -1,323 +1,237 @@ # Writing node code -A Function node is a Python file: no base class, no decorator, no framework -import unless you want one. +A Function node runs Python in a worker process. Write its source in the +canvas, or declare functions in your project with `@node` and +[`fluksio sync`](cli.md#fluksio-sync). ```python def process(temperature, setpoint=21.0): - """Ask for heat when the room is below the comfort point.""" return {"heat": temperature < setpoint} ``` +Declare `temperature` as an input port, `heat` as a Boolean output port, and +`setpoint` as a setting. Now you've got a tiny thermostat. + ## The rules -**One function called `process`.** If the file defines exactly one public -function under another name, that one is used instead. Two, and the node -refuses to load rather than guessing. +- **Define `process`.** Without it, the worker accepts exactly one public + function defined in the source. Multiple candidates produce an error. +- **Arguments arrive by name** from input ports and settings. A setting cannot + share a name with a port. See [Where values come from](../concepts/values.md). +- **Return a dictionary keyed by output port.** Undeclared keys and values of + the wrong type fail validation. With `@node(provides=Port("heat", "bool"))`, + the SDK also accepts a bare value and wraps it for that single port. +- **Use JSON-compatible values.** For files, use artifacts. For persistent + state, use [flow state](../concepts/state.md); worker globals are not a + reliable store. -**Arguments come from ports and settings, by name.** `temperature` above is an -input port; `setpoint` is a setting typed into the node's panel. Both arrive as -keyword arguments, which is why a setting may not share a name with a port. -See [Where a node's values come from](../concepts/values.md). - -**The return value is a dict keyed by output ports.** Every value is checked -against the port's declared type before it is published. A key that is not a -declared port is an error, not a silent drop. Nothing leaves a node except -through a port it declared. - -**Nothing else is importable from the engine.** Node code runs in a separate -process, on a separate interpreter, with none of Fluksio's own modules on its -path. What it can import is what the [Modules](../interface/operations.md) -screen installed, which can include [your own project](#your-own-code-as-a-package), -so a node need not be a self-contained file. - -**A node is a pure function of its inputs.** No context object, no global -store, no handle to reach for. A running total or a debounce timer has a -specific shape; see [Keeping state in a flow](../concepts/state.md). +Inside a node, `import fluksio` exposes reporting and artifact helpers. It +does not expose the engine's internal modules. Other imports come from the +node's Python environment; see [Packages](#packages). ## Producing values over time -A node that produces values *during* its execution is a generator. Every -`yield` is a dict keyed by output port, published the instant it happens: +Yield dictionaries to publish outputs before the node finishes: ```python +import math + + def process(lr, steps): loss = 1.0 - for _ in range(steps): - loss = train_one_step(lr) - yield {"loss": loss} # published now + for step in range(steps): + loss = math.exp(-lr * (step + 1)) + yield {"loss": loss} return {"final_loss": loss} ``` -Whatever the generator `return`s at the end is the node's result, which is what -downstream nodes read. If you never `return`, the last thing you yield is the -result instead. - -Mark the port so the flow says what it does: +Declare `loss` as a streaming float port: ```json {"name": "loss", "dtype": "float", "stream": true} ``` -Two consequences. In a [run](../concepts/runs.md), the whole series is kept as -that run's metrics, which is why there is no `log_metric()` anywhere in the -API. And **the node's timeout starts measuring silence rather than duration**: -each emission resets the deadline, so a node yielding every few seconds can run -for hours under a timeout of 300. +Each yield is type-checked and published immediately. The final return is +the node's result; without a return, the last yield is the result. A +[run](../concepts/runs.md) keeps numeric emissions as metric series and waits +for the result before running downstream nodes. A live flow can also trigger +downstream nodes from emissions. ### `fluksio.emit` -Where a `yield` cannot reach, because the value comes from inside somebody -else's callback and they call you rather than the other way round: +Use `emit` when a callback produces the value: + +```python +import fluksio + + +def on_epoch_end(epoch, logs): + fluksio.emit(loss=logs["loss"]) +``` + +Pass this callback to your training framework. It publishes through the same +declared ports as `yield`. Outside a node, `emit` does nothing, so ordinary +tests can still call your code. + +## Bytes: artifacts + +Save files or bytes with `save_artifact`, then return the reference through an +`artifact` port: ```python import fluksio def process(): - model.fit(callbacks=[LambdaCallback( - on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"]) - )]) - return {"weights": ...} -``` - -Same ports, same type checking, same publication. Prefer `yield` where you can -reach it; `emit` where you cannot. - -## Bytes: artifacts - -Messages are JSON, which is what lets the same value pass through Redis, the -work queue and the worker protocol unchanged. A checkpoint is not that. - -```python -import fluksio - - -def process(dataset): - path = fluksio.load_artifact(dataset) # → a local path to read - ... return { - "weights": fluksio.save_artifact("model.pt", media_type="application/octet-stream"), - "score": 0.94, + "report": fluksio.save_artifact( + b"Experiment complete\n", "report.txt", "text/plain" + ) } ``` -`save_artifact` takes bytes or a path, stores them by their SHA-256 digest, and -returns a small reference (digest, size, media type, name) which is what an -`artifact`-typed port carries. +Another node can read it with `fluksio.load_artifact(report)`, which returns +a local file path. These two helpers require a running node; for tests, +keep the underlying calculation in ordinary Python functions. -Because the address is the content's hash, a sweep whose fifty configs share -one preprocessed input stores it once, and a reference stays valid wherever the -store is reachable from, including on another machine. - -`fluksio artifacts RUN NAME` is how one comes back out at a terminal. - -### `fluksio.logger` - -A node's `print` is kept as that node's logs, and so is anything on -`fluksio.logger` is the same logger the SDK exports at top level, so code that -runs both inside a node and outside one says it the same way: - -```python -fluksio.logger.info("resuming from epoch %d", start) -``` +Artifacts are stored by their SHA-256 digest, so identical bytes share storage. +Use [`fluksio artifacts`](cli.md#fluksio-artifacts) to list and download a +run's files. ### Media -Say what the bytes are and the port can be typed for them: +`image`, `audio` and `video` ports carry artifact references with a matching +media type. For example, save JPEG bytes with `media_type="image/jpeg"` and +return the reference through an `image` port. See +[Payload types](../reference/payload-types.md#image-audio-video). -```python -def process(speech): # an `audio` port - clip = fluksio.load_artifact(speech) - words = transcribe(clip) - return {"transcript": words} # a `str` port -``` - -```python -def process(camera_url): - for index, jpeg in enumerate(grab(camera_url)): # a generator - frame = fluksio.save_artifact( - jpeg, f"frame-{index:05d}.jpg", - media_type="image/jpeg", volatile=True, - ) - frame["meta"] = {"seq": index} - yield {"frame": frame} # an `image` stream port -``` - -An `image`, `audio` or `video` port is an artifact reference whose media type -has to match, so a node declaring `audio` never receives a video by accident. -See [Payload types](../reference/payload-types.md#image-audio-video) for what -each carries and what rates are realistic. - -`volatile=True` is for a frame rather than a result. The bytes go to a ring in -memory instead of the data volume, and the engine pushes them down the -websocket to whichever screens are drawing that message — so a camera runs at -ten frames a second without writing anything to disk. They last as long as it -takes newer frames to need the room. Leave it off for a clip somebody asked to -keep. - -!!! warning "Emitted media is not kept; returned media is" - - Only what a node *returns* is recorded against its run. Frames yielded - along the way are replaced in state by the next one, and the artifact sweep - removes bytes nothing refers to any more, which is what stops a camera - filling the disk. If a particular frame matters, return it: a volatile one - is copied out of the ring when the run records it, so returning it is also - what makes it outlive the next few seconds. +For transient frames, `save_artifact(..., volatile=True)` uses an in-memory +ring and streams bytes to connected screens. New frames eventually replace +old ones. A volatile artifact returned as a run result is copied to persistent +storage; emitted frames alone are not recorded as run artifacts. ## Printing -`print` works and is captured. The first 16 KB per call is kept and shown in -the flow editor's log panel and on the run's per-node record; the rest is -dropped, so a node printing in a loop cannot fill anything up. - -Use it to debug, not to record results. A number worth keeping is an output -port, not a line of text. +`print` and `fluksio.logger.info(...)` are captured in the node logs. The +worker retains the first 16,384 characters per call. Use logs to debug and +output ports for numbers you want to plot or compare. ## Errors -An exception fails that node's execution, not the flow. The message you see is -one line from the frame in *your* code, not a stack through the engine. - -The node keeps its last error visible after it recovers, so a failure that -fired an alert at 03:00 still says what it was at 09:00. It can also be -acknowledged from the canvas. +Exceptions appear on the node, with a short message and a traceback. In a live +flow, one failed execution does not stop the whole flow. In a batch run, node +failure makes the run fail. The canvas keeps the last error visible after +recovery until you acknowledge it. ## Timeouts -`timeout` on a node is how many seconds its code may be *silent* before it is -stopped. A yield or an `emit` resets the clock, and the first call's imports are -not charged to it: a node importing torch is not being slow, it is loading. +`timeout` is the allowed number of seconds without an emission. A `yield` or +`emit` resets the deadline; initial compilation and imports are excluded. -There is no timeout by default. Training runs for hours and a node that reports -nothing is usually working, so the engine waits: what fails a call is the worker -dying, which arrives at once rather than after a deadline. Set a timeout on the -nodes where silence means stuck (an HTTP call that should answer in seconds, a -loop that can spin) either per node or, for all of them, -with `FLOW_NODE_TIMEOUT`. `timeout = 0` says explicitly that this node has none. +There is no timeout by default. Set one on a node where silence means stuck, +or use `FLOW_NODE_TIMEOUT` for the instance default. `timeout=0` explicitly +disables it for a node. A worker disappearing fails its active call without +waiting for that deadline. ## Running a node somewhere else -A node declares the label of the machine it needs: +Set a device label to select an attached worker: ```json {"id": "train", "device": "gpu", "device_policy": "require", "timeout": 7200} ``` -`require` (the default) waits for a worker carrying that label; `prefer` runs -locally when none is attached. A node bound to a device is compiled *on that -machine*: a node importing `torch` is correct on the GPU box and a missing -module on the engine, so checking it here would fail something that is fine. - -See [Remote workers](workers.md). +`require` is the default and waits for a worker with that label. `prefer` uses +the local engine when none is attached. Code is compiled on the selected +machine, which needs the packages your node imports. See +[Remote workers](workers.md) for setup and resource requests. ## Sharing code between flows -A node's source can be promoted to the shared library from its panel, and other -flows can then use it by reference. There is one copy and one place to edit, and -every flow using it runs the edit. +Promote a node's source to the shared library from its panel. Other flows can +reference that source, and edits apply to all of them. Shared sources live in +`_lib/` in the instance's flow repository. -Shared sources live in `_lib/` in the flow repository, so they are versioned -with everything else. +For code-defined flows, reuse the same decorated function. `use(fn, ...)` +changes its wiring or settings for one flow: + +```python +use(fit, wire={"dataset": "augmented"}, epochs=3) +``` ## Packages -Node code runs in a virtual environment of its own, on the instance's data -volume, separate from the one Fluksio itself runs on. +The startup banner shows the interpreter used for local nodes: -Declare what you import in [Modules](../interface/operations.md), or over the -API: +| Environment | How to add packages | +|---|---| +| Fluksio installed in an existing venv | Use pip or uv in that venv, then `fluksio sync` to refresh workers. Modules is read-only. | +| Managed node venv | Edit the manifest in **Modules** and apply it. | +| Remote worker | Install packages in the interpreter selected by `--python`. | + +With `NODE_VENV=auto`, Fluksio adopts its existing venv unless this instance +already has a managed one. Without an existing venv, it creates a managed +environment. `NODE_VENV=managed` always requests one; the Docker image uses +this setting. + +In managed mode, the Modules manifest is installed with `uv pip sync` and +versioned with the flows. Applying it refreshes local workers. The API accepts +the same manifest: ```sh -curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \ +curl -X POST "$FLUKSIO/modules/apply" -H "Authorization: Bearer $TOKEN" \ -H 'Content-Type: application/json' \ -d '{"requirements": "numpy>=2\npandas\n"}' ``` -It is a pip manifest installed with `uv pip sync`, versioned alongside your -flows. An install takes effect immediately; nothing restarts. +Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN` is an +access token; see [The HTTP API](api.md). ### Your own code as a package -!!! tip "If Fluksio is installed in the venv you work in, skip this" - - Node code then runs on that environment, so your project and everything it - imports are already importable; see - [Getting started: data science](../getting-started/data-science.md). What - follows is for a Fluksio with a venv of its own, which is what a container - always has. - -A manifest line can name a directory, so the project you already have is -installable like any other dependency: +`fluksio sync` generates an import shim pointing at your local project. A +managed environment can also install your project from a Modules manifest: ```text -e /home/you/my-research numpy>=2 ``` -A node body then imports it, and the logic stays where it already lives: in -your repository, under your own version control, importing its own siblings: +Then a canvas node can import ordinary helpers from it: ```python -from myresearch.train import fit +from myresearch.training import train_model def process(lr, epochs): - return fit(lr, epochs) + return {"score": train_model(lr, epochs)} ``` -That is the whole of it. The node is three lines, `myresearch` can be as many -modules as it likes, and nothing was copied. +Editable installs expose changed files, but warm local workers still hold +imports in memory. Apply the Modules manifest or run `fluksio sync` after +edits. Attached workers start a process per call and read the files afresh. -!!! tip "You can have those three lines written for you" - - Decorate `fit` with `@node(...)` where it is defined, say which nodes make - a flow with `Flow(...)`, and `fluksio sync` generates the body above, - along with the flow document, so there is nothing to PUT by hand. The - declaration lives beside the function it describes and is checked against - its signature. See - [Getting started: data science](../getting-started/data-science.md). - -!!! warning "Editable, but not live" - - `-e` means edits reach the venv without reinstalling, but a node's process - already holds the imported module in memory. The engine's workers are - long-lived, so a change to your code is picked up when they are retired, - which is what **Apply** does. Pressing it after an edit is the loop. - - A [worker](workers.md) you attach yourself is the exception: it starts a - process per call, so it reads your code fresh every run. If you are - iterating on the code many times an hour, point one at your own - interpreter (`fluksio-worker --python "$(which python)"`) and mark the - node with its label. - -!!! note "The path is a deployment detail" - - It is resolved on whichever machine runs the node, and the manifest is - committed to the flow repository, so an absolute path from your laptop - means nothing inside a container or on a GPU box. Those need their own - install of the same project; a VCS requirement - (`myresearch @ git+ssh://…@a1b2c3d`) travels where a path does not. +Paths are resolved on the machine running the node. A remote worker or +container needs its own copy or installation of your project; syncing a shim +does not ship the package. ## A worked example -The repository ships a small supervised fit as a seedable demo: three nodes, -a batch flow, streaming metrics, artifacts between stages, and a GPU-labelled -node that falls back to the engine when no worker is attached. It is the -shortest complete thing to read: +[Getting started: data science](../getting-started/data-science.md) contains a +complete runnable flow. The public repository also includes +`examples/myresearch/`, with separate preparation, training and evaluation +modules. From the repository root: -```text -prepare ──dataset(artifact)──▶ train ──weights(artifact)──▶ evaluate - │ - └── loss (streaming float) ──▶ chart +```sh +fluksio sync examples/myresearch +fluksio run train --lr 0.05 --wait ``` -The example is built by a seed script the maintainers run against a live -stack; the flow above is what it produces. +Start an instance first, as shown in the getting started guide. ## See also - [Where a node's values come from](../concepts/values.md) - [Keeping state in a flow](../concepts/state.md) - [Runs: pipelines that finish](../concepts/runs.md) -- [Node types](../reference/node-types.md) — the ones you do not have to write +- [Node types](../reference/node-types.md) diff --git a/docs/code/workers.md b/docs/code/workers.md index 89049c4..810c501 100644 --- a/docs/code/workers.md +++ b/docs/code/workers.md @@ -1,119 +1,77 @@ # Remote workers -The engine runs where the automations are. The GPU is somewhere else, the -Raspberry Pi with the relays is in a shed, and neither of them is on the same -network as the other. +A worker runs node code on another machine: a GPU host, a Raspberry Pi or a +batch allocation. It connects outward to the engine over an authenticated +WebSocket. Keep one engine; add workers where the hardware lives. -A **worker** is a process that runs the code of nodes marked for it. It dials -*out* to the engine over one authenticated websocket, so nothing on that -machine has to be reachable, and nothing has to expose the engine's state -backend across hosts, which it never should. +## Create a worker token -## Install and attach +On the engine, use a superuser access token. In these examples: + +- `API_URL` is the engine's API base, including `/api/v1`, such as + `https://api.example.com/api/v1` or `http://localhost:8080/api/v1` for a LAN stack. +- `TOKEN` is your superuser bearer token. The login endpoint + `POST /api/v1/login/access-token` accepts form fields `username` and `password`. + +Mint a token with the labels this machine should carry: ```sh -pip install fluksio-worker - -fluksio-worker \ - --url wss://api.${DOMAIN}/api/v1/workers/attach \ - --labels gpu,cuda12 \ - --python /opt/torch-venv/bin/python -``` - -`fluksio-worker` is its own distribution: the agent, the node runner, and -`websockets`. Nothing of the engine, so a GPU box does not install a database -driver in order to run a training step. An engine host already has it, and -`fluksio worker …` is the same program. - -| Option | Default | What it does | -|---|---|---| -| `--url` | **required** | `wss://…/api/v1/workers/attach` | -| `--token` | `$FLUKSIO_WORKER_TOKEN` | the credential, minted on the engine; on the command line any user of the machine reads it from `ps`, so prefer the variable | -| `--name` | this host's name | sent along; the worker list shows the token's name | -| `--labels` | the token's | comma-separated; must match the labels the token was minted with | -| `--python` | this interpreter | the interpreter node code runs on | -| `--parallel` | `1` | how many node calls it will take at once | -| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket | -| `--cpus` | what the job or the machine has | cores to advertise | -| `--gpus` | what the job says, else counted | GPUs to advertise | -| `--ram-mb` | what the job or the machine has | memory to advertise, in MB | -| `--max-idle` | never | stop after this many seconds with nothing running | -| `--insecure` | off | allow `ws://` and `http://` to an engine on another machine | - -The worker refuses a plain `ws://` or `http://` address unless the engine is on -this machine (`localhost`, `127.0.0.0/8`, `::1`): its token and every secret a -node is handed would cross the network in clear. `--insecure` is for a network -you trust, such as an engine on the same LAN without TLS. Node code gets this -host's environment less anything named like a credential (`*TOKEN*`, -`*SECRET*`, `*PASSWORD*`), the same as on the engine; hand a node a credential -as a [secret](../interface/operations.md#secrets) instead. - -`--python` is the important one. It is how this machine keeps its own wheels -(the CUDA build, the vendor SDK, the thing that will not install anywhere else) -without the engine ever installing them or knowing about them. - -### What it says it has - -A worker reports its inventory when it attaches (cores, GPUs and memory) and -the engine schedules against it: a node asking for two cores and a GPU goes to -a machine that has them free, not merely to one carrying the right label. - -Cores and memory are read off the machine, or off the batch job that started -this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **What the job says -it was given always wins for GPUs** (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`, -or `FLUKSIO_WORKER_GPUS`): a node with eight cards may have granted this job -one, and advertising eight would be a lie the scheduler acts on. With nothing -said, NVIDIA's device nodes are counted, the same as the engine does for its -own machine, and `--gpus` overrides either. No vendor tool is asked, which is -what keeps the one dependency from becoming two. A worker that reports nothing -still attaches and is scheduled by its label alone, as every worker was before -any of them reported anything. - -The engine tells each call what it may use: thread caps, and the devices it -may see. The worker starts a process per call, so it applies them at the only -moment a numerical library still reads them: before the import. - -`--max-idle` is for a worker something else started for one job, a batch -scheduler say. It exits when nothing has run for that long, so the allocation -goes back rather than idling until its walltime. - -## Mint the token - -On the engine, as a superuser: - -```sh -curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \ +curl -X POST "$API_URL/workers/tokens" \ + -H "Authorization: Bearer $TOKEN" \ -H 'Content-Type: application/json' \ -d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}' ``` -Shown once, valid for a year, since a worker is a machine somebody sets up and -leaves running. The name and labels are signed into it: labels decide which -nodes, with their source and secrets, are sent to a worker, so a worker cannot -choose its own. It shows up under the token's name, and a worker whose -`--labels` differ from the token's is refused. +The response contains `token`, `jti` and the expiry. Save the token as +`FLUKSIO_WORKER_TOKEN` on the worker host. It is returned once and is valid +for 365 days. The signed name and labels control which nodes and secrets the +engine sends to that worker. + +## Install and attach + +Use Python 3.12 or later on the worker host: ```sh -curl -s $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" | jq -curl -X POST $FLUKSIO/workers/tokens/$JTI/revoke -H "Authorization: Bearer $TOKEN" +pip install fluksio-worker +export FLUKSIO_WORKER_TOKEN='' + +fluksio-worker \ + --url wss://api.example.com/api/v1/workers/attach \ + --labels gpu,cuda12 \ + --python /opt/torch-venv/bin/python ``` -The first lists what was minted, by `jti` (the tokens themselves are not -kept); the second refuses one from then on, here and at the artifact -endpoint, and drops a worker attached with it. Rotating the keypair agent -tokens are signed with still revokes every worker at once. A token minted -before tokens carried their labels is refused: mint a new one. +Use your engine's hostname. `--python` selects the environment containing the +node's dependencies, such as PyTorch or a device SDK. If you install the worker +in that environment, you can omit the option. An engine installation also +provides the same agent as `fluksio worker`. -??? note "A host where pip is not an option" +| Option | Default | Purpose | +|---|---|---| +| `--url` | required | engine's `/api/v1/workers/attach` WebSocket | +| `--token` | `$FLUKSIO_WORKER_TOKEN` | credential; prefer the environment variable to keep it out of process arguments | +| `--name` | hostname | suggested name; the engine uses the token's name | +| `--labels` | token's labels | comma-separated; when supplied, must match the token | +| `--python` | worker's interpreter | Python used for node code | +| `--parallel` | `1` | concurrent calls | +| `--artifact-url` | derived from `--url` | artifact endpoint, if hosted separately | +| `--cpus` | detected | advertised CPU count | +| `--gpus` | detected | advertised GPU count; explicit `0` means none | +| `--ram-mb` | detected | advertised memory in MB | +| `--max-idle` | `0` | exit after this many idle seconds; `0` disables | +| `--insecure` | off | allow unencrypted connections to a non-loopback host | - The two files work copied into one directory and run with `python agent.py - …`. The engine serves the runner itself at `GET /api/v1/workers/runtime`. - It is the same module its own local workers run, deliberately standard - library only. +For a trusted LAN stack, the URL is +`ws://:8080/api/v1/workers/attach` and requires `--insecure`. +Outside loopback, the worker otherwise requires encrypted URLs (`wss://` and +`https://`): node secrets and the worker token travel over the connection. + +Node code receives a filtered host environment. Credential-like variables are +removed; pass credentials through [Secrets](../interface/operations.md#secrets). ## Send a node to it -A node declares the label of the machine it needs: +Set these fields on a node through `PUT /api/v1/flows/{name}`: ```json { @@ -124,94 +82,73 @@ A node declares the label of the machine it needs: } ``` -| `device_policy` | Behaviour when nothing carrying the label is attached | +This is a node fragment; keep the node's other fields and the rest of the flow +when updating it. `device` and `device_policy` are not in the node panel yet. + +| `device_policy` | If no matching worker is attached | |---|---| -| `require` (default) | the run stays `queued` and says what it is waiting for | -| `prefer` | it runs on the engine instead | +| `require` (default) | queues until a matching worker is available | +| `prefer` | falls back to the engine | -`prefer` is what makes a flow work before the GPU box exists. `require` is what -you want once it does. +Source code travels with each call and compiles on the worker, so remote-only +dependencies do not need to be installed on the engine. Calls to `emit`, +`save_artifact` and `load_artifact` work there too. -!!! note "Not in the panel yet" +Cancelling a run stops its active node processes. A disconnected worker fails +in-flight calls. While executing, the agent sends heartbeats every ten seconds; +90 seconds without contact fails the call. Heartbeats do not reset the node's +own silence timeout. - `device` and `device_policy`, which machine a node's code runs on, are set - through the API rather than the panel, with `PUT /flows/{name}`. +## Resources and artifacts -## What follows from this +Workers report CPU, GPU and memory inventory, and scheduling checks both labels +and available resources. CPU and memory detection uses Slurm allocations when +present, otherwise host figures. GPU detection uses `SLURM_GPUS_ON_NODE`, +`FLUKSIO_WORKER_GPUS` or `SLURM_JOB_GPUS` before counting NVIDIA device nodes. +Explicit flags override detection. -- **The node's source travels with every call.** Nothing has to be deployed to - the worker, and changing a node's code takes effect on the next execution. -- **A node bound to a device is compiled on that machine.** A node importing - `torch` is correct on the GPU box and a missing module on the engine, so - checking it here would fail something that is fine. -- **`import fluksio` inside a node is the worker's own reporter.** `emit`, - `save_artifact`, `load_artifact`, installed before your code runs, so an - installed `fluksio` package on that box never shadows it. -- **Cancelling a run kills what it is executing**, there or here, and leaves - other runs of the same node alone. -- **If the worker disappears mid-call**, the run fails in seconds with - `worker went away mid-call` rather than waiting out its timeout. -- **A worker sends a heartbeat every ten seconds while it executes**, so a long - node is distinguishable from a dead socket. Ninety seconds of nothing at all, - not even a heartbeat, fails the call as gone. A heartbeat says the *agent* is - alive and nothing about the node, so it never satisfies a node's own timeout: - one set to thirty seconds fires after thirty seconds of the node reporting - nothing, wherever it runs. +Artifacts are identified by content hash. Remote workers fetch and upload over +HTTP beside the WebSocket endpoint; node code uses the same artifact helpers +as on the engine. Downloads are cached by digest: -## Artifacts across machines +| Variable | Default | +|---|---| +| `FLUKSIO_ARTIFACT_CACHE` | `fluksio-artifacts` in the temporary directory | +| `FLUKSIO_ARTIFACT_CACHE_BYTES` | `1073741824` (1 GiB) | -An artifact reference names content by its hash, not a location, so it stays -valid wherever the store is reachable from. A worker that shares the engine's -filesystem writes to it directly; one that does not fetches and uploads over -HTTP, using the artifact endpoint beside the socket it already has. Either way -your node code is the same two calls. +Oldest files are removed when the cache exceeds its limit. Raise it if workers +repeatedly use large inputs. -A fetch is cached on the worker by digest, since content addressing means an -entry is never stale. Nothing expires on its own, so the cache is bounded by -size and the oldest fall out: `FLUKSIO_ARTIFACT_CACHE` says where it lives -(default a directory in the temporary directory) and -`FLUKSIO_ARTIFACT_CACHE_BYTES` how much it holds (default 1 GiB). Worth raising -where a worker reads the same large inputs repeatedly, and worth leaving alone -where it reads a media stream — those are chunks nothing asks for twice. - -## Seeing what is attached +The **Workers** screen shows attached hosts, available resources and queued +nodes. The API exposes the same information: ```sh -curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq +curl "$API_URL/workers" -H "Authorization: Bearer $TOKEN" +curl "$API_URL/workers/resources" -H "Authorization: Bearer $TOKEN" ``` -Name, labels, how many calls it will take at once, how many are in flight, when -it attached, when it was last seen, its Python version, a digest of its -environment, and what it says it has: cores, GPUs and memory. +## Revoke or upgrade -## Upgrading +List tokens and revoke one by its `jti`: -The engine and the worker speak a version-matched protocol, and a worker -announcing anything else is refused rather than half-understood. Protocol 2, -the one that carries inventory, is `fluksio-worker` 0.2.0. An older agent is -told so on the socket and stops, rather than retrying against an engine that -will never accept it; `pip install -U fluksio-worker` on that host is the whole -upgrade. Nothing changed in the runner served at `GET /api/v1/workers/runtime`, -so a host that copies its two files copies the same one as before. +```sh +curl "$API_URL/workers/tokens" -H "Authorization: Bearer $TOKEN" +curl -X POST "$API_URL/workers/tokens/$JTI/revoke" \ + -H "Authorization: Bearer $TOKEN" +``` -A token minted before tokens carried their labels and a `jti` is refused. Mint -each existing worker host a new one with `POST /workers/tokens`, naming the -labels it should carry, and swap it in for `FLUKSIO_WORKER_TOKEN`. +Revocation also disconnects attached workers and blocks artifact access. +Replacing the OAuth signing key revokes all worker tokens. -**Upgrade the engine before its workers.** A worker sends its token as an -`Authorization` header on the handshake, where it stays out of access logs, and -an engine from before that reads it only from the URL: it refuses the newer -worker at the handshake. The engine still takes the old `?token=` for one -release, so older workers keep attaching to a newer engine meanwhile. +Upgrade the engine before its workers, then run `pip install -U fluksio-worker` +on each host. Engine and worker must speak the same protocol; a mismatch is +reported and refused. Older tokens without signed labels and a `jti` need to +be replaced. -## Machines from a batch scheduler +## Slurm provisioning -A cluster is not a machine that attaches and stays; it is a queue somebody else -owns. So Fluksio does not submit *nodes* to Slurm. It submits a job whose payload -is an ordinary worker dialling back in, and from there everything works the way -it already does: the same protocol, the same artifacts, the same cancellation. - -Write the clusters into `provisioners.json` beside the flows: +The engine can submit a Slurm job that starts a worker. Put this configuration +in `$DATA_DIR/provisioners.json`, replacing the example addresses and paths: ```json [{ @@ -219,58 +156,34 @@ Write the clusters into `provisioners.json` beside the flows: "name": "hpc", "login": "me@login.cluster", "ssh_key": "/secrets/hpc_ed25519", - "engine_url": "wss://api.${DOMAIN}/api/v1/workers/attach", + "engine_url": "wss://api.example.com/api/v1/workers/attach", "max_idle_s": 300, "provision_timeout_s": 900, "profiles": [{ "name": "gpu-small", - "cpus": 8, "gpus": 1, "ram_mb": 65536, + "cpus": 8, + "gpus": 1, + "ram_mb": 65536, "labels": ["gpu"], "sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"], - "prerun": ["module load cuda/12", "source ~/venvs/flux/bin/activate"] + "prerun": ["module load cuda/12", "source ~/venvs/fluksio/bin/activate"] }] }] ``` -A **profile** is what the scheduler is asked for, where a flavor is what a node -asks for. They are separate on purpose, and agree when you set them up to. +The engine uses system SSH to submit `sbatch`. Prepare the environment named +by `prerun` with `fluksio-worker` and your node dependencies; the generated job +does not install packages. Compute nodes must be able to connect to the +engine. Each job receives its own worker token, valid for 24 hours. -When a node needs a machine nothing attached can give, and a profile fits, the -engine `sbatch`es one over the system `ssh`, and the run waits meanwhile, saying -so. `prerun` owns the environment: a `module load`, or a venv with -`fluksio-worker` already in it. The generated script runs no `pip install`. -It hands the worker a token of its own, good for 24 hours, as -`FLUKSIO_WORKER_TOKEN` rather than `--token`, so other users of a shared -compute node cannot read it from `ps`. - -One outstanding request per profile, however often it is asked for. A job that -never attaches within `provision_timeout_s` is `scancel`led, as is anything -outstanding when the engine stops. `--max-idle` is what ends the job at the -other end, so an allocation goes back rather than idling to its walltime. - -Both take `0` for "no limit": `provision_timeout_s: 0` waits for as long as the -queue does, which is what a cluster that queues overnight needs, and -`max_idle_s: 0` keeps the machine for the job's whole walltime. - -`GET /workers/resources` reports what is outstanding and what last went wrong. - -!!! note "It needs a route out" - - A compute node must be able to open a connection to the engine. That is - true of most clusters and false of air-gapped ones; there is no staging - path today. - -## What a worker is not - -It is not a second engine. Subscriptions, schedules, webhooks, the dashboards -and the run queue all stay in one process, which keeps a message having one -definition and a cron tick happening once. A worker executes node bodies. - -Running two engines against one data directory is not supported. Distribute -work with workers. +Only one request per profile is outstanding at a time. Jobs that fail to attach +within `provision_timeout_s` are cancelled, as are pending requests when the +engine shuts down. `max_idle_s` releases idle workers. Either setting accepts +`0` to disable its limit. Provisioning status appears under +`GET /api/v1/workers/resources`. ## See also -- [Runs: pipelines that finish](../concepts/runs.md#running-a-node-somewhere-else) -- [Writing node code](nodes.md) -- [Getting started: data science](../getting-started/data-science.md) +- [Runs](../concepts/runs.md#running-a-node-somewhere-else): remote execution. +- [Writing node code](nodes.md): node functions and artifact helpers. +- [Getting started: data science](../getting-started/data-science.md): resource declarations. diff --git a/docs/concepts/flows.md b/docs/concepts/flows.md index 36ec524..6056cc8 100644 --- a/docs/concepts/flows.md +++ b/docs/concepts/flows.md @@ -1,204 +1,151 @@ # Flows, nodes and messages -Three ideas hold the system up. Almost everything else follows from them. +A **flow** is a group of **nodes** that exchange **messages**. Nodes declare +what they consume (`requires`) and produce (`provides`); matching message names +connect them. No wires to draw. -## A flow is a graph you did not draw +## How nodes connect -A **flow** is a set of **nodes**. Each node declares the messages it needs -(`requires`) and the messages it produces (`provides`). The graph is whatever -those declarations imply: +For a Function node, declare input and output ports in its panel, then write +`process(...)`. Arguments use input port names; the returned dictionary uses +output port names: ```python -# node "read" +# Node "read": provides temperature def process(): - return {"temperature": read_sensor()} + return {"temperature": 18.5} -# node "decide" +# Node "decide": consumes temperature, provides heat def process(temperature, setpoint=21.0): return {"heat": temperature < setpoint} ``` -`decide` is downstream of `read` because it needs `temperature` and `read` -produces it. Nobody drew a wire. +Bind both temperature ports to the same message and `decide` follows `read`. +Here `setpoint` uses a Python default; you can also add it as a node +[setting](values.md#settings-constants-of-this-node). -The consequences: - -- **Fan-in is free.** Two nodes providing `temperature` are two producers of - one message. The consumer does not change. -- **A node runs when something it reads was published.** Not merely when - something upstream of it ran: a node that produced nothing this time (held - back by a rate limit, say) leaves what reads it on the value it already - has, and so does everything behind that. -- **A wire cannot be wrong.** There is no wire. There is a name that either - matches or does not, and the canvas tells you at edit time which it is. -- **Layout is not a document.** The canvas computes the arrangement, so a flow - has no stored positions to maintain, merge or fight over. -- **Flows stay small.** A graph nobody can hand-arrange is one worth keeping - small, which is the intent. Several atomic flows that name each other beat - one flow with sixty nodes in it. +- Several nodes can produce the same message. Consumers use the latest value. +- Publishing an input wakes its consumers once their required values are + available. An upstream node that publishes nothing leaves downstream nodes + waiting for a new publication. +- The editor arranges the graph automatically. Keep flows focused and share + messages between them when needed. ### Message names are namespaced -Inside flow `house`, a message named `temperature` is really -`house.temperature`. A bare name is qualified with its own flow; a dotted name -is used as written. That is how two flows share a value: +In flow `house`, `temperature` becomes `house.temperature`. A dotted name is +used as written, so another flow can consume it directly. Bind an input port +named `temperature` to `house.temperature`: ```python -# in flow `dashboard`, reading a message that flow `house` produces -def process(house_temperature): # port bound to "house.temperature" - ... +def process(temperature): + return {"shown": temperature} ``` -The canvas draws messages arriving from another flow as labelled endpoints, so -you can see where they come from without opening the other flow. +The canvas shows cross-flow messages as labelled endpoints. -## A node is a function with declared ports +## Node types and payloads -Most nodes are **Function** nodes: a Python file defining `process(...)`. Its -arguments are its input ports by name; its return value is a dict keyed by -output ports. - -```python -def process(reading, unit="C"): - return {"shown": reading if unit == "C" else reading * 1.8 + 32} -``` - -`reading` is a port. `unit` is a **setting**: a constant of this node's code, -typed into its panel and stored with the flow. Both arrive as arguments, which -is why a setting may not share a name with a port. See -[Where a node's values come from](values.md). - -The rest of the node types are the ones that would be tedious or unsafe to -write yourself: MQTT, HTTP, InfluxDB, schedules, switches, notifications. Each -one is configured by filling in a form the editor generates from its parameter -schema, so they all behave the same way. The full list is in +Function nodes run your Python code. Built-in nodes handle MQTT, HTTP, +InfluxDB, schedules, filtering and more through forms in their panels. See [Node types](../reference/node-types.md). -### Ports are typed +Every port declares a payload type (`dtype`). Values are checked against it, +and dashboard widgets offer compatible messages. Scalars, structured JSON and +media references are covered in [Payload types](../reference/payload-types.md). -A port declares a `dtype`: `float`, `int`, `str`, `bool`, `json`, `record`, -`list`, `series`, `artifact`, or one of the media types `image`, `audio` and -`video`. Every value that passes through is checked against it. - -Types are not decoration. They are what lets the dashboard editor offer you -only the messages a gauge can actually draw, and what lets the canvas refuse a -binding before anything runs. See [Payload types](../reference/payload-types.md). - -Everything on the wire is JSON. Bytes (a checkpoint, an image, a model) -travel as an `artifact`: the bytes go to a content-addressed store and the -message carries a small reference to them. The media types are that same -reference, saying what kind of bytes are behind it. +Messages carry JSON. For binary data, store the bytes as an **artifact** and +publish its reference. `image`, `audio` and `video` are artifact references +restricted to their media family. ### Streaming ports -A port marked `stream` produces repeatedly *during* one execution rather than -once at the end: a training loss, a progress fraction, a frame from a camera, a -second of speech. A node publishes on one by being a generator and yielding, or -by calling `fluksio.emit`. +Mark an output **streaming** when it publishes during an execution: progress, +training loss or media chunks. A Function node can yield values or call +`fluksio.emit`. -Each value is delivered to the nodes reading it, in the order it was produced, -so a recogniser slower than the microphone in front of it still sees every -chunk rather than only the newest. What is in state remains the latest value, -which is what everything else reads, and what a run keeps is the whole series. +Downstream nodes receive each publication in order; message state holds the +latest value. A batch run records the full series. An output `interval` limits +live publication without thinning the run's recorded series. -An `interval` on a streaming port thins what reaches the canvas without -thinning the run's record of it. +### Keeping node code predictable -### Nodes are pure +Write Function nodes as functions of their declared inputs and settings. +Fluksio provides no hidden `context` or global flow store. To remember a value +between executions, use a message the node reads and writes; see +[Keeping state in a flow](state.md). -A node is called with the values of the messages it declares and returns the -values of the messages it provides. There is no context object, no global -store, no handle to reach for. +## Live and batch flows -That is deliberate: a node with hidden state cannot run twice in parallel, -cannot be replayed, and cannot be moved to another machine. Plenty of real -automations do need to remember something, and there is a specific way to say -so; see [Keeping state in a flow](state.md). - -## Two shapes of flow - -Set `mode` on the flow: +Choose **Mode** in the flow settings: | | `live` (default) | `batch` | |---|---|---| -| Runs | continuously | once per run, on request | -| Started by | subscriptions, schedules, webhooks | `POST /runs/flows/{name}` | -| Ends | never | when the graph drains | -| Keeps | the last value of each message | a run record: params, result, metrics, artifacts | -| Is | a thermostat, an ETL job on a cron | an experiment, a CI-style job | +| Runs | reacts continuously | once per requested run | +| Started by | subscriptions, schedules, webhooks or manual execution | Run in the editor, SDK or API | +| Finishes | stays active until stopped | when its work finishes | +| Keeps | latest message values | parameters, results, metrics and artifacts | +| Example | thermostat | experiment or build job | -A batch flow is built and validated like any other, appears on the same canvas -and is type-checked the same way. It is simply never *activated*: no -subscriptions, no schedules, no webhooks. See -[Runs: pipelines that finish](runs.md). +Batch flows use the same editor and validation, but do not activate background +subscriptions, schedules or webhooks. See [Runs](runs.md). -## Editing is separate from running +## Flow inputs -Every flow has a published version and, while you are working, a draft. - -- **Saving** writes the draft. The engine keeps running the published version. -- **Publishing** promotes the draft. The engine reloads and picks it up. -- **Discarding** throws the draft away. - -The store is a git repository (`flow.json` for the structure, `nodes/*.py` for -the code) and each save is a commit. So a flow's history is readable with -ordinary git tooling, and copying a flow between instances is copying a -directory. - -Saving carries the version you last saw. If someone else saved in between, you -get a 409 instead of quietly overwriting their work. - -## What can be wrong, and when you find out - -The canvas validates continuously and names problems on the nodes they belong -to: - -| Issue | What it means | -|---|---| -| `unconnected_input` | a port needs a message nothing in reach provides | -| `missing_initial_value` | the message exists but has never held a value, and nothing will give it one. Not reported on a batch flow: its inputs arrive with the run | -| `cycle` | A waits for B and B waits for A — nothing could ever start | -| `self_loop_needs_initial` | a node reads a message it also writes, with no starting value | -| `node_error` | the node's code did not load: a syntax error, a missing import | -| `unauthenticated_hook` | advisory — a webhook with no shared secret is open to anyone | -| `node_unhealthy` | the node loaded but is not working: a connector that cannot reach its device, or whose last publication failed | - -A flow with any of these except the advisory one and `node_unhealthy` does not -run. A node reporting itself down is a live condition, not a build error, so -the rest of the flow keeps going and the issue clears by itself once the node -reports well again. The health summary on Home counts them, so "why is nothing -happening?" has an answer that does not involve reading logs. - -## What happens at runtime - -- **A flow can be started and stopped.** Stopped means its subscriptions and - schedules are torn down. -- **A flow can be paused and stepped.** Paused holds messages instead of - running them; step releases exactly one. This is how you test something - before it moves a relay. -- **A failing node does not take the flow down.** It reports an error, keeps - its last error visible after it recovers, and can fire an alert. -- **A flow whose background tasks keep crashing is quarantined.** The engine - stops restarting them and says so, rather than spinning. Publishing a change - gives it another chance. - -## Values that arrive from outside - -Some messages are not computed by any node: a dashboard control writes them, -the API publishes them, a batch run passes them in. Declare those as the flow's -**inputs**, with the value they start from: +Declare messages supplied by dashboard controls, the API or run parameters as +the flow's **Inputs**. Give live inputs a starting value: ```json {"inputs": [{"spec": {"name": "setpoint", "dtype": "float"}, "initial": 21.0}]} ``` -Without that, the node reading `setpoint` waits for something nothing provides, -and the canvas says so. With it, the flow starts at 21.0 and whatever writes -the message afterwards takes over. +This seeds `house.setpoint` in flow `house` if it has no value yet. Later +publications replace it. Batch inputs are run parameters, and batch outputs +select the run's result. See [Where a node's values come from](values.md). + +## Drafts and publishing + +- **Saving** stores a draft; the engine keeps running the published version. +- **Publishing** promotes the draft and reloads the flow. +- **Discarding** removes the draft. + +Shared node code is an exception: saving it updates every flow using it +immediately. See [The flow editor](../interface/flow-editor.md#code-and-reuse). + +Flows are stored in a git repository: `flow.json` describes the graph and +`nodes/*.py` holds code. Saves are versioned, and stale versions return a +conflict instead of overwriting another editor's changes. + +Python installations need Git on the host to record history. Without it, +flows still save as files; Docker includes Git. + +## When a flow needs attention + +The canvas reports issues on the affected nodes: + +| Issue | Meaning | +|---|---| +| `unconnected_input` | no node or declared flow input supplies the message | +| `missing_initial_value` | a live input has no starting value | +| `cycle` | nodes depend on each other in a loop | +| `self_loop_needs_initial` | a node reads and writes a message with no starting value | +| `missing_source` | a Function node has no saved code | +| `node_error` | code or configuration could not load | +| `unauthenticated_hook` | a webhook has no shared secret; advisory | +| `node_unhealthy` | a loaded node reports a runtime problem, such as a lost connection | + +Build issues prevent the affected flow from running. The last two are runtime +or advisory issues: other work continues, and node health clears when the +connection recovers. + +You can **stop** a live flow to remove its subscriptions and schedules, or +**pause** it to hold pending work while values still arrive. **Step** releases +one held item. Repeated background crashes can quarantine a flow; publishing +a change gives it another chance. ## Where to next -- [Where a node's values come from](values.md) — ports, settings, flow inputs -- [Keeping state in a flow](state.md) — the sanctioned way to remember something -- [Runs: pipelines that finish](runs.md) — the batch half of the engine -- [Writing node code](../code/nodes.md) — the practical side of `process()` +- [Where a node's values come from](values.md) — ports, settings and flow inputs +- [Keeping state in a flow](state.md) — remembering a value +- [Runs](runs.md) — batch flows +- [Writing node code](../code/nodes.md) — practical Python examples diff --git a/docs/concepts/runs.md b/docs/concepts/runs.md index e255728..bb51221 100644 --- a/docs/concepts/runs.md +++ b/docs/concepts/runs.md @@ -1,476 +1,265 @@ # Runs: pipelines that finish -A flow that automates a house never ends: a value arrives, nodes fire, and it -waits for the next one. A research pipeline is the other shape: parameters go -in, stages execute in order, and at some point it is *done* and has produced -something worth keeping. Fluksio calls the second one a **run**, and it is the -same engine either way. +A **live flow** reacts to events. A **batch flow** starts with parameters, +runs its graph once and returns a result. Each execution is a **run**, with +its own state, parameters, node records, metrics and artifacts. -This is what makes Fluksio usable where Kedro, MLflow or ClearML would be: a -run has parameters that identify it, a result, per-step metrics, artifacts and -a place in a queryable history, with no second server and no -project bootstrap on every execution. +Use runs for experiments, reports and other jobs with a finish line. For a +complete first example, start with the +[data science guide](../getting-started/data-science.md). ## A batch flow -Set `mode: "batch"` on the flow and name the messages its result should hold: +In Python, `Flow(...)` defaults to `mode="batch"`. Declare `inputs` for +run parameters and `outputs` for the final result. On the canvas, choose batch +mode and define the same inputs and outputs. -```json -{ - "name": "train_polymer_gnn", - "mode": "batch", - "outputs": ["final_loss", "report"], - "inputs": [ - {"spec": {"name": "lr", "dtype": "float"}, "initial": 0.1}, - {"spec": {"name": "steps", "dtype": "int"}, "initial": 3000} - ], - "nodes": [{"id": "train", "timeout": 7200, "device": "gpu", "...": "..."}] -} -``` +A batch flow does not need activation and does not open subscriptions, +schedules or webhooks. A run executes each node once after its dependencies +finish. Missing parameters use their declared initial values. -A batch flow is built and validated like any other (it appears on the canvas, -its ports are type-checked) but it is never *activated*: no subscriptions, no -schedules, no webhooks. It runs when a run asks it to, and not otherwise. - -Its `inputs` are its parameters. A run supplies values for them; anything it -does not supply keeps the declared initial value. - -One thing a batch flow may not do is rate-limit a port (`interval`). A rate -limit holds a value back for a timer to release, and a run has no timer, so the -value would be dropped rather than delayed, so submitting is refused instead. +Non-streaming ports cannot use `interval` in a batch flow: there is no timer +to deliver a delayed value. Streaming ports can use it to reduce display +updates while keeping every metric point. ## Submitting -**Runs → New run** in the app is the same call with a form in front of it: pick -a flow, fill in its declared inputs, press Run. A comma-separated list in a -number field runs every combination of them as a sweep. **Use draft** runs the -working copy rather than what is published, which is how a change is tried -before it is deployed, and each run keeps its own state, so nothing it computes -reaches the live flow. +In **Runs → New run**, choose a flow, enter its inputs and press **Run**. +**Use draft** runs the working copy without publishing it. -```bash -curl -X POST $FLUKSIO/runs/flows/train_polymer_gnn \ - -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ - -d '{"params": {"lr": 0.3, "steps": 4000}, "seed": 7}' +From a terminal: + +```sh +fluksio run train --lr 0.1 --epochs 20 --wait ``` -The answer is immediate and the run is `queued`; a training run is measured in -hours, so nothing waits for it. Poll `GET /api/v1/runs/{id}` for its status, -result, per-node record and artifacts. +Or submit through the API: -Wrong parameters are refused before anything executes: an undeclared name, or -a value of the wrong type, comes back as a 422 naming the problem. +```sh +curl -X POST "$FLUKSIO/runs/flows/train" \ + -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ + -d '{"params": {"lr": 0.1, "epochs": 20}, "seed": 7}' +``` -`"draft": true` on the body runs the unpublished working copy — the flow -document and the node sources the editor is showing. The copy is read again -when the run starts, so a draft published while it waited runs as the -published one. +Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN` +is an access token. See [The HTTP API](../code/api.md) for authentication. + +Submission returns immediately with status `queued`. Read +`GET /api/v1/runs/{id}` for status and results. Unknown parameters or wrong +types return 422 before execution. + +The API body also accepts `"draft": true`, `"no_cache": true` and an +`idempotency_key`. Reusing an idempotency key returns the original submission +instead of creating another run. Drafts are read when execution starts, so +edits made while a run is queued can affect it. + +A seed is recorded on the run and fills a declared input named `seed`. +Your node code must use that input to seed its random number generator. ### Sweeps -An ensemble is the same parameters at different seeds; a grid search is the -parameters spread out. Both are one call, and the caller builds the list: +A sweep submits every combination of input values as one group: -```bash -curl -X POST $FLUKSIO/runs/flows/train_polymer_gnn/sweep \ - -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{ - "runs": [{"params": {"lr": 0.1}, "seed": 1}, {"params": {"lr": 0.3}, "seed": 1}] -}' +```sh +fluksio sweep train --param lr=0.01,0.1 --param epochs=10,50 --wait ``` -From a terminal that is `fluksio sweep train_polymer_gnn --param lr=0.1,0.3`, -which builds the product of the lists you give it and posts the same call. +These four runs share a `group_id` and execute concurrently within the +engine's limits. Each has its own state, so one run cannot overwrite another's +messages. Vary seeds with `--param seed=1,2,3` if the flow declares that input. -They share a `group_id`, so `GET /api/v1/runs?group=…` is the sweep, and they -execute in parallel. That is safe because **each run has a state backend of its -own**: message names are global keys, so two runs of one flow would otherwise -overwrite each other's values. They do not. +The API equivalent is `POST /api/v1/runs/flows/{name}/sweep` with a +`runs` list of parameter dictionaries and optional seeds. Filter history +with `GET /api/v1/runs?group=`. ## Producing values before you are finished -A training loop has numbers worth keeping long before it has a result. Those -numbers are **outputs**, not logs: a node declares a port for them and produces -them over time, which in Python is a generator. +Training curves and progress values leave through output ports. A Python +generator publishes them with `yield`; callbacks can use `fluksio.emit`. +Declare the port with `stream: true`. See +[Writing node code](../code/nodes.md#producing-values-over-time) for examples. -```python -def process(lr, steps): - loss = 1.0 - for _ in range(steps): - loss = train_one_step(lr) - yield {"loss": loss} # published now, on the `loss` port - return { - "weights": fluksio.save_artifact(dump(model), "weights.npz"), - "final_loss": loss, - } +Every numeric emission is stored under its flow-qualified message name, +such as `train.loss`, with a step counter and timestamp. Retrieve it with: + +```text +GET /api/v1/runs/{id}/metrics?name=train.loss +GET /api/v1/runs/series/compare?ids=,&metric=train.loss ``` -Mark the port it streams on, so the flow says what it does: +A streaming port's `interval` reduces updates sent to the canvas, while the +run history keeps every point. In a run, intermediate emissions do not start +downstream nodes; those wait for the node's final result. -```json -{"name": "loss", "dtype": "float", "stream": true} -``` - -Every yield is published the instant it happens: same port, same type check, -same place on the canvas as any other value. Whatever the generator `return`s -is the node's result, and is what downstream nodes read. If you never -`return`, the last thing you yield is the result instead. - -There is no `log_metric()`. A metric leaves through a port, which makes it a -message: a chart binds to it directly, a downstream node can consume it, and the -run keeps its series. - -**Where a yield cannot reach** — the value comes from inside somebody else's -callback, and they call you rather than the other way round. `fluksio.emit` -writes the same ports the same way: - -```python -import fluksio - - -def process(): - model.fit(callbacks=[LambdaCallback( - on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"]) - )]) - return {"weights": ...} -``` - -### What a run does with them - -Every number a node emits is kept as the run's series, stepped by the count of -emissions on that message. Read one back with -`GET /api/v1/runs/{id}/metrics?name=.loss`, or leave `name` off for -every series the run kept, or compare runs: - -``` -GET /api/v1/runs/series/compare?ids=,,&metric=.loss -``` - -That answers in the `series` shape a chart widget already draws, so three -training curves side by side is a widget binding. During a run the values also -arrive live on the flow socket, so a chart bound to the port fills in as the -training goes. - -A streaming port may set `interval` to thin out what reaches the canvas. The -run's history still keeps every value, because the interval is asking for the -display not to be flooded, not for the curve to have holes in it. - -Emitting has a second effect: **a node's timeout measures silence, not -duration**. A node that yields every few seconds can run for hours under a -`timeout` of 300; one that says nothing for longer than its timeout is killed. -There is no timeout unless one is set, since a training node that reports -nothing is usually working. Set `timeout` where going quiet means stuck, at how long -the node may plausibly be. - -In a *live* flow, an emission also wakes whatever is downstream of it, exactly -as a subscriber publishing does. In a run it does not: a run's graph is -scheduled once, and three thousand mid-node cascades would leave "the run has -finished" with nothing to mean. +A node's `timeout` measures silence. Each yield or emit resets it, so a node +can run for hours while reporting progress. No timeout is imposed by default. ## Artifacts -Bytes never travel as a message. `save_artifact` writes them to a -content-addressed store and returns a small reference (digest, size, media -type, name) which is what an `artifact`-typed port carries: +Use `fluksio.save_artifact` for checkpoints, datasets, plots and other bytes. +Return its reference through an `artifact` port. Another node calls +`fluksio.load_artifact` to obtain a local path. -```python -def process(weights): # requires: weights, dtype "artifact" - path = fluksio.load_artifact(weights) - ... +Identical bytes share one stored copy. Returned artifacts are listed on the +run and can be downloaded with: + +```sh +fluksio artifacts +fluksio artifacts train.weights -o weights.json ``` -Because the address is the content's hash, a sweep whose fifty configs share -one preprocessed input stores it once, and a reference stays valid wherever -the store is reachable from. Artifacts a run produced are listed on it and -downloadable at `GET /api/v1/artifacts/{digest}`. +Use the exact message name shown by the list command. Emitted media frames +are transient; return a frame you need to keep as a run artifact. ## Stage caching -A run mostly does not redo what an earlier one already did. Before a node -executes it is fingerprinted with a sha256 over its source, its settings, the -ports it declares and the values it is about to read. If some earlier run of that -same fingerprint finished, what that one returned is restored into this run's -state and the node is skipped. It is recorded with the status `cached` and a -duration of zero, and its artifacts are listed on the new run as well, so they -stay downloadable from either. +Before executing a Python node, the engine checks for a successful previous +execution with matching source, settings, port declarations and input values. +On a hit, it restores the outputs and artifacts and records the node as +`cached`. -The settings go into the key raw, so a secret contributes its `{"$secret": -name}` reference and never its value. An artifact input counts as its content -digest: the same bytes under a different filename are the same input. Each node -on a run carries the `cache_key` it was looked up by. +For code-defined flows, sync also records digests of the project modules +reachable from each node. Editing those modules invalidates the corresponding +stage. Static import tracking does not follow computed import names or track +installed dependency versions; use `--no-cache` after changes it cannot see. -For a [code-defined flow](../getting-started/data-science.md), "its source" is -the generated shim, which imports the real function and does not change when -that function does. So the key carries one thing more: a digest of the project -modules that node's function reaches, worked out by `fluksio sync`, which is -the only side that imports your code and can see what it imports, and read -again from those files when the run starts. +Artifact inputs are compared by content digest. Secret settings contribute +their references rather than secret values. External state is not part of a +cache key, so disable caching on a node that reads changing data or performs +side effects: -Editing a helper three calls down from the node invalidates it, which is the -point: the alternative is a re-run answering with the previous code's numbers. -Editing something the node does not reach leaves the hit standing, which is -the other half: a notebook two directories away is not a reason to retrain. -The walk follows imports statically and stops at the standard library, at -anything installed, and at Fluksio itself; a module imported under a name the -code computes is not followed. An engine that cannot see the files keeps what -sync recorded instead of nothing, so a worker on another machine no longer -keys every run the same. +```python +@node(provides=Port("reading", "float"), cache=False) +def read_sensor(): + return get_current_reading() +``` -The run history *is* the cache; there is no second store. A node's returned -outputs are kept on its run record as canonical JSON, up to 256000 characters. -A node returning more than that is not cacheable that run. An entry -whose artifact bytes have since left the store is a miss, not an error. +Use `fluksio run ... --no-cache`, `fluksio sweep ... --no-cache`, or +`"no_cache": true` in an API submission to execute every node. -Only `python` nodes are cached, and by default all of them are. A built-in node -type or a connector node has side effects and no source to fingerprint, so -neither is ever a candidate. Turn it off for one node with -`@node(..., cache=False)` (the flow document carries it as `cache`, so the -canvas and the API can change it too) or for one run with -`fluksio run --no-cache`, `fluksio sweep --no-cache`, or `"no_cache": true` in -the submission body. +Only Python nodes are cached. Built-in and connector nodes are not. The cache +uses run history; outputs too large to store there are not cacheable, and +missing artifact bytes make an entry a miss. -A cached node replays no emissions, since those values were the story of an -execution that is not happening this time, so its series is not rewritten -either. The run it was restored from is recorded instead, and that is where the -curve is read back from: asking the reusing run for its metrics answers with -the same points, under its own flow's names. The one way to be left with a -result and no curve is for that earlier run to have been deleted. Deleting a -flow does not do that — its runs stay — so this only happens when the run -itself was deleted. +A cache hit does not replay emissions. Its metrics are read from the original +run, so comparisons still show the recorded curve. Deleting that original run +leaves cached results available but removes their curve. ## Objects that cannot be serialized -A live model, a `DataLoader`, a JAX-compiled function: none of these cross a -node boundary, and no framework flag will make them. There are exactly two -patterns, and they are both deliberate: - -* **Keep them in one node.** Stages that must share live memory are one node. - Building the model and training it is one stage; the fact that Kedro would - make them two nodes is Kedro's problem, not a structure worth reproducing. -* **Cross at a checkpoint.** Save what matters as an artifact and rebuild from - it on the other side. That is the boundary that also survives the next node - running on a different machine. +Live models, data loaders and compiled functions cannot cross a JSON message +boundary. Keep stages that share live objects in one node, or save a checkpoint +artifact and rebuild the object in the next node. ## Running a node somewhere else -A node that needs a GPU declares the label of the machine that has one: +Attach a [remote worker](../code/workers.md) to the engine and declare the +worker's label as the node's `device`. `device_policy="require"` waits for +that label; `"prefer"` uses the local engine when none is attached. -```json -{"id": "train", "device": "gpu", "device_policy": "require", "timeout": 7200} -``` - -A worker on that machine dials **out** to the engine, because the engine -generally cannot reach it (different network, no inbound route) and because -nothing should expose Redis across hosts. Install it on the box, mint it a -token, and start it: - -```bash -curl -X POST $FLUKSIO/workers/tokens -d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}' # once, as an admin - -pip install fluksio-worker - -fluksio-worker \ - --url wss://api.${DOMAIN}/api/v1/workers/attach \ - --labels gpu,cuda12 \ - --python /opt/torch-venv/bin/python -``` - -`fluksio-worker` is its own distribution: the agent, the node runner, and -`websockets`. Nothing of the engine, so a GPU box does not install a database -driver to run a training step. Where pip is not an option, the two files still -work copied into one directory and run with `python agent.py …`; the engine -serves the runner at `GET /api/v1/workers/runtime`. - -`--python` is the interpreter node code runs on, which is how the GPU box keeps -its CUDA wheels without the engine ever installing them. The node's source -travels with every call, so nothing has to be deployed there. - -A few consequences worth knowing: - -* `import fluksio` inside a node is the worker's own reporter — `emit`, - `save_artifact`, `load_artifact`, installed before the node's code runs, so - the installed `fluksio` package (if the box has one) never shadows it. -* A node bound to a device is **compiled on that machine**. A node importing - `torch` is correct on the GPU box and a missing module on the engine, so - checking it on the engine would fail a node that is fine. -* If nothing carrying the label is attached, the run stays `queued` and says - what it is waiting for. Submit first, switch the GPU box on later. -* Cancelling a run kills what it is executing, there or here, and leaves other - runs of the same node alone. -* If the worker disappears mid-call, the run fails in seconds with - `worker went away mid-call` rather than waiting out its timeout. -* `device_policy: "prefer"` runs locally when no such worker is attached; - `"require"` (the default) waits for one. +The selected machine needs your imported packages. Code-defined flows also +need your project installed or available there: generated import shims travel +with the call, but your package does not. Workers connect outward to the +engine; see the worker guide for tokens and resource requests. ## Durability -Submitting journals the run to a Redis stream of its own, separate from the -one the automations use: a burst of five hundred sweep runs must not stand -between a house and its heating. An engine that is down when a run is -submitted picks it up when it starts. +Queued runs are stored in the database and picked up when the engine restarts. +Running jobs refresh a lease. If the engine dies and the lease expires, those +runs become `abandoned` rather than being executed again automatically. -The row is what makes that true rather than the queue. A run is written before -the work item is added, so an engine reads its own history at startup and -wakes itself for anything still `queued` — which is what a run submitted -seconds before a restart is, and what an in-memory queue would otherwise have -lost with the process. +Retry a finished run or the unfinished results of a sweep: -From the moment a run is claimed, its database row is the record and the queue -is finished with it. Redelivering two hours of training because an -acknowledgement was late is not recovery; instead a running run refreshes a -lease, and one whose lease goes stale is marked `abandoned`, which is what a -run whose engine was killed mid-training becomes. Starting it over is a -decision rather than something that happens: [`fluksio -retry`](../code/cli.md#fluksio-retry) submits it again, and `--group` does -that for the runs of a sweep that did not finish. +```sh +fluksio retry +fluksio retry --group +``` + +A retry creates a new run with the previous inputs, seed and group, linked +to its parent. It uses the flow's current definition, and can reuse completed +cached stages. Cancel active runs before retrying them. ## Looking at what ran -**New run** is where one is started, and the **Runs** screen is the experiment -log: every run newest-first, filtered by flow, by status, or down to one -sweep. A sweep is worth filtering to, since the table then draws a column per -parameter that actually varied, which is what makes fifty runs of one flow -readable. +The **Runs** screen filters by flow, status or sweep. Each run shows its +parameters, per-node timings and logs, traceback, metrics, artifacts and +result. -Tick two or more and their curves go side by side. Shift-click to take a -range, or the header box to take everything on screen. That comparison is the -address, so a link to it is a link someone else can open. +Select two or more runs to compare curves and the parameters that differ. +Plot against step, elapsed seconds or another metric from the same runs. +Comparisons have shareable URLs. -The curves are drawn against the step by default. They can also be drawn -against elapsed seconds, which answers "which one got there sooner" and is -measured from each run's own first reading so that runs started hours apart -still lie on top of each other; or against another metric of the same runs (an -epoch, or samples seen) joined on the step the two share. - -One run in full is params, the per-node record with its logs and traceback, -the artifacts it made, its metrics and its result. **Retry** submits it again -as a run of its own, keeping the flow, the inputs, the seed and the sweep it -belonged to. - -A run outlives the flow it ran. Deleting a flow leaves its history standing — -being the record of what ran is the point of keeping one — so the Runs screen -still lists it, and opening it says **flow deleted** with Retry disabled, -because there is nothing left to run it against. Clearing that history is a -decision of its own: a run at a time on the screen, or -`DELETE /api/v1/runs?flow=` for a whole flow's worth, which is what a -reseed wants. +Deleting a flow keeps its run history. You can still inspect those runs, but +cannot retry them until the flow exists again. Delete history separately, +after active runs finish or are cancelled. ### Taking it into a dataframe -An analysis wants a table rather than a screen, and there are two it usually -wants. `fluksio export metrics` is the long one, a row per run, metric and -step; `fluksio export runs` is the wide one, a row per run with the -inputs that varied as columns beside its final numbers: +Export metrics as a long table and results as a row per run: + +```sh +fluksio export metrics --flow train -o curves.csv +fluksio export runs --flow train --status ok -o results.csv +``` + +Both include run IDs. Result rows also include input values, final numeric +outputs, status, duration and code identifiers. Nested values use dotted paths, +such as `metric.final_metrics.train_loss`. + +Python returns the same rows directly: ```python -import pandas as pd from fluksio.sdk.client import Client client = Client() -curves = pd.DataFrame(client.export_metrics(flow="train")) -arms = pd.DataFrame(client.export_runs(flow="train", status="ok")) +curves = client.export_metrics(flow="train") +results = client.export_runs(flow="train", status="ok") ``` -The run id is on every row of both, so a curve joins to the arm it came from -and to the run page it was recorded on, and the wide table carries the commit -and the code digest, so an exported file says what produced its numbers. -[The CLI](../code/cli.md#fluksio-export) writes the same rows as csv, jsonl or -parquet, which is where an export belongs: in the script beside the analysis. - -A node's numbers usually arrive inside a record, and the wide table follows -them in: every number a result carries is a column of its own, named by its -path (`metric.final_metrics.train_loss`) and `metrics="final_metrics.train_loss"` -selects one. Inputs work the same way, and are compared leaf by leaf, so two -configurations differing in one field give that field as the axis. +Pass either list to `pandas.DataFrame`. The +[CLI reference](../code/cli.md#fluksio-export) covers selection filters, +JSON Lines and Parquet. ### Bringing in runs made elsewhere -Runs are made by running a flow, so a history recorded before the move (an -MLflow experiment, say) has to be imported instead. Each entry names the flow, -the id the run had where it ran, and what it ended with: +Import historical runs without executing them: ```python -Client().import_runs([ - { - "flow": "train", - "external_id": "mlflow:study-1/3f2a…", - "params": {"lr": 0.01}, - "seed": 1001, - "status": "ok", - "created_at": "2026-03-01T10:00:00Z", - "started_at": "2026-03-01T10:00:01Z", - "finished_at": "2026-03-01T10:40:12Z", - "result": {"final_metrics": {"train_loss": 0.08}}, - "metrics": [{"name": "train.train_loss", "step": 0, "ts": 1772359201.0, "value": 2.3}], - "artifacts": [{"name": "weights", "filename": "model.txt", - "media_type": "text/plain", "content_base64": "…"}], - "labels": ["imported"], - "group_id": "mlflow:study-1", - }, -]) +from fluksio.sdk.client import Client + +Client().import_runs([{ + "flow": "train", + "external_id": "previous-study:run-42", + "params": {"lr": 0.05, "epochs": 10}, + "status": "ok", + "created_at": "2026-03-01T10:00:00Z", + "result": {"score": 0.94}, +}]) ``` -The parameters are completed with the flow's declared values and checked the -way a submit checks them. The run is stored with the cause `import` and no -commit or code digest, since this engine never saw the code it ran. Importing -the same `external_id` again adds nothing, and `fluksio export runs` carries -it as a column, so an exported row joins back to where it came from. +The flow must exist, and parameters are checked against its inputs. Repeated +imports of the same `external_id` reuse the existing record. Imported runs +have no code commit or digest because the engine did not execute them. +Entries may also carry metric points and base64-encoded artifacts; see the +API schema for their fields. ### A dashboard, read against runs -A run records values under the same names a dashboard binds to (a run of -`study` writes `study.loss`) so a dashboard is already a way of looking at -one. **Open in dashboard** from a run or a comparison opens any dashboard with -`?runs=a,b,c` on it, and the widgets resolve from those runs instead of from -the live engine: each chart draws a line per run, the tiles that show one -number show the first, and the controls go quiet because there is nothing left -to publish to. +Choose **Open in dashboard** from a run or comparison to view those results +with a dashboard's widgets. Charts draw one line per run, scalar widgets use +the first run, and controls are disabled. -Nothing about the dashboard is specific to runs. The page built to watch a -training run happen is the page that shows the finished ones. +If the flow has no dashboard, the same menu can create one from its streaming +ports and outputs. The API equivalent is +`POST /api/v1/dashboards/from-flow/{name}`. The generated dashboard is +published and can be edited like any other dashboard. -If a flow has no dashboard yet, the same menu offers to build one from what -the flow declares: a chart per streaming port, a tile per output: - -``` -POST /api/v1/dashboards/from-flow/study -``` - -It publishes `study_results` and is an ordinary dashboard afterwards; editing -it is how it stops being generic. - -### A tile that always shows the last few - -The other direction is a chart pinned to runs rather than a page opened -against them, which is what a panel over a bench wants. Set a chart's source -to **Runs**, name the metric, and pick either the latest N of a flow, one -sweep, or specific runs. It re-reads on its own and whenever a run finishes. - -### When a run draws nothing - -A node restored from the [stage cache](#stage-caching) has its curve read back -from the run that recorded it. Delete that run and the reusing run is left with -a result and an empty curve, and the chart says so rather than looking broken. - -## What this costs, compared - -The repository ships a benchmark that measures submitting a run against a -Kedro project doing the same nothing: - -``` -fluksio — submit accepted median 15.3 ms - submit -> result median 60.8 ms -kedro — kedro run median 1109.5 ms -``` - -The difference is not the orchestration; it is that Fluksio does not boot a -project per run. The engine is already up, and the workers already have the -node's code compiled. On a 510-run sweep, that gap is about nine minutes of -pure startup that never happens. +To keep a chart showing recent experiments, set its source to **Runs** and +select the latest runs of a flow, a sweep or specific run IDs. ## See also -- [Getting started: data science](../getting-started/data-science.md) — the same - material as a first setup -- [Writing node code](../code/nodes.md) — generators, `fluksio.emit`, artifacts -- [Remote workers](../code/workers.md) — attaching the machine with the GPU -- [The HTTP API](../code/api.md) — every endpoint used above +- [Getting started: data science](../getting-started/data-science.md) +- [Writing node code](../code/nodes.md) +- [Remote workers](../code/workers.md) +- [The HTTP API](../code/api.md) diff --git a/docs/concepts/state.md b/docs/concepts/state.md index d80fb30..48f983f 100644 --- a/docs/concepts/state.md +++ b/docs/concepts/state.md @@ -1,75 +1,56 @@ # Keeping state in a flow -Logic nodes are pure functions of their inputs. There is no `context`, no -`global`, no handle to a store. A node is called with the values of the -messages it declares and returns the values of the messages it provides. +Need a running total or a last-seen value? Make it a message. Function nodes +have no hidden `context` or global flow store. -That is deliberate: a node with hidden state cannot be run twice in parallel, -cannot be replayed, and cannot be moved to another worker. But plenty of real -automations need to remember something. This is how. +## Read and write the same message -## State is a message the node both reads and writes - -A running total, a debounce timer, a last-seen reading: each is a value that -survives between runs. Give it a message name, declare it as both an input and -an output, and it is state: +Declare `reading` and `total` as inputs, and `total` as an output: ```python -def process(reading, total=0.0): +def process(reading, total): return {"total": total + reading} ``` -with `reading` and `total` as inputs and `total` as the output. Each run reads -the total the previous run wrote. +Add `total` to the flow's **Inputs** with a starting value of `0.0`. Each +execution reads the previous total and publishes the next one. -Two things make this work rather than loop forever: +- A node is never downstream of itself, so its own publication does not run + it again. +- A triggering input needs a starting value. Without one, validation reports + `self_loop_needs_initial`; a Python default alone does not supply the message. -- **A node never depends on itself.** The graph is built from message names, but - a node providing a message it also requires is not placed downstream of - itself. Publishing `total` does not re-run the node that wrote it. -- **The value must start somewhere.** The first run has nothing to read. Declare - the message as a flow input with a starting value; otherwise the flow reports - `self_loop_needs_initial` at edit time and refuses to publish. +## Feedback between two nodes -The port needs a default in the function signature (`total=0.0` above) so the -first call is well-formed even if the value is missing. - -## Feeding a value back between two nodes - -Sometimes the value comes from a different node, such as a controller reading back what -an estimator computed from its own last output. Written plainly that is a cycle, -and the validator rejects it, because a graph where A waits for B and B waits -for A can never start. - -What is actually meant is that the fed-back value is *state*, not a trigger. -Say so, by marking the input non-triggering: +A controller may read an estimate that depends on its own previous output. +Ordinary triggering inputs would form a cycle. Mark the feedback input +**non-triggering** in the flow definition instead: ```json {"name": "estimate", "port": "estimate", "dtype": "float", "trigger": false} ``` -A non-triggering input: +It creates no dependency and does not wake the node. Whenever another input +runs the node, it reads the latest estimate. If no estimate exists yet, that +argument is omitted, so give it a Python default: -- creates no dependency, so it cannot form a cycle; -- never makes the node wait. If the message has no value yet, the port is - simply left out of the call, and the function's default applies; -- is read fresh from state whenever the node does run, for whatever reason. +```python +def process(reading, estimate=0.0): + return {"command": reading - estimate} +``` -Use it for the back edge only. An input that should wake the node is an ordinary -input, and marking it non-triggering would mean the node never runs at all. +Use this on the feedback input; keep the inputs that should wake the node +triggering. Configure `trigger` through the API or flow JSON; the node panel +does not currently expose it. -## What still holds engine-side state +## Built-in state -Built-in nodes that are *about* time or change (rate limiting, filter-on-change, -delay, cron) keep what they need in the engine's own state backend, under keys -that never appear as messages. That is the engine's business, not a flow's: the -line is that node *code* you write never reaches for state, while node *types* -the engine ships may. - -Storage and direct-I/O nodes are the other sanctioned exception, since talking -to a device or a database is stateful by nature. +Built-in nodes such as delay, rate limit and filter-on-change keep their own +timing or comparison state in the engine. Storage and device nodes also manage +the state needed for their connections. Your Function code still exchanges +state through declared messages. ## See also -- [Where a node's values come from](values.md) — ports, settings and flow inputs -- [Node types](../reference/node-types.md) — the built-ins that do keep engine-side state +- [Where a node's values come from](values.md) +- [Node types](../reference/node-types.md) diff --git a/docs/concepts/values.md b/docs/concepts/values.md index 48bcd9e..1628f71 100644 --- a/docs/concepts/values.md +++ b/docs/concepts/values.md @@ -1,67 +1,60 @@ # Where a node's values come from -A node function is called with one argument per thing it needs, and every one -of them is declared somewhere you can see. There are two kinds. +Function nodes receive values as named arguments. Each comes from a message +port, a node setting or a Python default. ## Ports: what the graph carries -A port binds to a message name. Whatever last published that message is the -value the node is called with, and the wiring follows from the name rather than -from a line somebody drew. +A port binds to a message name and reads its current value. Declare inputs +`reading` and `setpoint`, and output `heat`: ```python def process(reading, setpoint): return {"heat": reading < setpoint} ``` -with `reading` and `setpoint` as inputs and `heat` as an output. +Matching message names connect nodes. A port can have a different name from +its message: bind port `reading` to `house.temperature`, for example. ## Settings: constants of this node -A setting is a value that belongs to this node's code rather than to the graph: -how many retries, which unit, how long to wait. It is typed into the node's -**Settings** section, stored with the flow, and arrives as an argument like a -port: +Use settings for values that stay fixed while the flow runs: units, retries, +thresholds. Add them in the Function node's **Settings** section; they are +stored with the flow and passed beside the ports: ```python def process(reading, unit="C"): return {"shown": reading if unit == "C" else reading * 1.8 + 32} ``` -Because both arrive by name, a setting cannot share a name with a port. The -node reports it rather than picking one. +With a `unit` setting, its configured value is passed to the function. Without +one, the Python default applies. A setting cannot share a name with a port. -Settings are part of the flow document, so changing one is an edit that gets -published, not something that happens at runtime. **A value that should change -while the flow runs is a message, not a setting.** - -There is no `global` and no `flow` context. A constant several nodes need is a -node that provides it: one place it is set, an ordinary message out, and every -consumer visibly downstream of it. +Changing a setting edits the flow draft, so publish it to apply the change. +For a value that should change at runtime, use a message. To share a constant +between nodes, publish it from one node and let the others consume it. ## Flow inputs: what arrives from outside -Some messages are not computed by any node in the flow, such as a dashboard control -writes them, the API publishes them, a batch run passes them in. Declare those -as the flow's **inputs**, with the value they start from: +Dashboard controls, the API and batch runs can supply messages without a +producer node. Declare them as **Inputs** in the flow settings: ```json {"inputs": [{"spec": {"name": "setpoint", "dtype": "float"}, "initial": 21.0}]} ``` -Without that, the node reading `setpoint` waits for something nothing provides, -and the canvas says so. With it, the flow starts from 21.0 and whatever writes -the message afterwards takes over. +The initial value seeds an unset live message. Later publications take over; +changing the starting value does not overwrite a value already in state. +Use **Publish this value now** in the input row to send it immediately. -The canvas draws each one as a labelled endpoint feeding the nodes that read -it, the same way it draws a dashboard tile or another flow, so a value never -appears from nowhere. They are edited in the flow's own panel. +The canvas draws flow inputs as labelled endpoints. Without a producer or +declared input, a consuming node reports `unconnected_input`. -A batch flow's inputs are also its run parameters, and its **outputs** name what -a run reports as its result; see [Runs](runs.md). +A batch flow's inputs are run parameters. Its **Result** section selects the +messages returned when the run finishes. See [Runs](runs.md). ## See also -- [Flows, nodes and messages](flows.md) — how the graph is built from the names -- [Keeping state in a flow](state.md) — a value a node reads *and* writes -- [Writing node code](../code/nodes.md) — the practical side +- [Flows, nodes and messages](flows.md) +- [Keeping state in a flow](state.md) +- [Writing node code](../code/nodes.md) diff --git a/docs/getting-started/data-science.md b/docs/getting-started/data-science.md index 9aa0eb7..bd36357 100644 --- a/docs/getting-started/data-science.md +++ b/docs/getting-started/data-science.md @@ -1,800 +1,196 @@ # Getting started: data science -You have a training script. It works. What you do not have is any reliable -answer to "what was the learning rate on the run that got 94%?", and the -`results_final_v3_ACTUAL.csv` in your home directory is not helping. +Turn a Python experiment into a flow with recorded parameters, training curves +and files you can find again. Goodbye, `results_final_v3.csv`. -This page adds Fluksio to what you already have. It takes about five minutes, -installs one Python package, and does not ask you to restructure anything. -## Install +This example uses a toy training curve, so you can try the whole workflow +without a dataset or GPU. You need Python 3.12 or newer on Linux or macOS. +Install Git too if you want flow changes recorded in version history. + +## 1. Start your instance + +Create a project directory and install Fluksio in a virtual environment: ```sh -cd my-research +mkdir research-flow +cd research-flow +python3 -m venv .venv +. .venv/bin/activate pip install fluksio fluksio serve ``` -That is the whole instance. No Docker, no database server, no ports to -open, and no login. The first run prints something like: +Fluksio creates `.fluksio/` for this project's flows, run history and artifacts. +It prints the initial admin password once and saves a token for local CLI +commands. Keep the password somewhere safe. -```text -Created the admin account admin@example.com - password: example-password - Shown once. Change it from the dashboard. -Fluksio 0.1.0 — data in /home/you/my-research/.fluksio - API http://127.0.0.1:8000/api/v1 - Nodes /home/you/my-research/.venv/bin/python - your environment, adopted. Add packages with pip. - No portal. Pair this instance with: - fluksio enroll - Signed in as admin@example.com - token in /home/you/my-research/.fluksio/client.json -``` - -Read the last two lines: you are already signed in. Signing in to your own -machine is a formality, since the password was printed by the same process that -would have checked it, so `serve` writes the token itself and every command -below just works. `fluksio login` is for an engine somewhere *else*. - -**Write that password down** anyway. It is shown once, and it is what the -dashboard asks for. - -### One instance per project - -`.fluksio` sits beside your code, and is found the way `.git` is: from the -directory you are standing in, or any directory above it. So two repositories -on one machine are two engines: separate flows, separate run history, -separate token, and no chance of one experiment's graph turning up in the -other's. - -```text -~/research/protein-fold/ - .venv/ torch, fluksio - .fluksio/ its own database, flows, artifacts, token - myresearch/ - -~/research/climate-sim/ - .venv/ jax, fluksio - .fluksio/ its own everything - climate/ -``` - -It holds a database and a credential, so it ignores itself from within with a -`.gitignore` of `*`, the same thing `uv` writes into `.venv`. Nothing to add -to your project's own. - -Give them different ports (`--port`) if you want two running at once. - -!!! tip "One engine for the machine instead" - - `fluksio serve --global` uses `~/.fluksio`, shared by every directory, - which is what you want for a personal server rather than a project. When - both exist, the banner says which one you are looking at and how to reach - the other. `--data-dir` still names any directory outright, which is worth - doing on a cluster where `$HOME` is a network filesystem: SQLite's - write-ahead log does not work on NFS, and `fluksio serve` warns you when - it notices. - -!!! tip "Keep it running" - - The engine is meant to be resident, which is what makes submitting a run - cost about 15 ms. Leave it in a `tmux` window, or write a small - `systemd --user` unit for it. - -## Talking to it over HTTP - -Everything the commands do is the HTTP API, and some of this page shows it -directly. For that, take the token `serve` already wrote: +At a terminal, `serve` opens a dashboard. Leave it running, then open a second +terminal in `research-flow` and activate the same environment: ```sh -export FLUKSIO=http://127.0.0.1:8000/api/v1 -export TOKEN=$(jq -r .token .fluksio/client.json) +. .venv/bin/activate ``` -While you are experimenting, the interactive schema at - is the fastest way to see what is available. - -For an engine on another machine, `fluksio login --url https://…` asks for a -password and stores the token the same way: in this project's `.fluksio`, or -with `--global` in `~/.fluksio`. - -## Your packages are already there - -If you installed Fluksio into the environment you work in (the venv that -already has torch or numpy in it) that is the environment your nodes run on. -Nothing to declare, nothing to install twice: +For the browser dashboard, sign up at [fluksio.com](https://fluksio.com), add +an instance, and enter the code it gives you: ```sh -python -m venv .venv && . .venv/bin/activate -pip install torch numpy pandas # what you were going to install anyway -pip install fluksio # and then this -fluksio serve +fluksio enroll ``` -`fluksio serve` says which interpreter it settled on: +Select the instance in the portal. The CLI also works without enrolling. -```text - Nodes /home/you/research/.venv/bin/python - your environment, adopted. Add packages with pip. -``` +!!! tip "Use your existing research environment" -That venv is yours. Add a package the way you added the rest (`pip install -scikit-learn`) and `fluksio sync` (or a restart) retires the workers so they -pick it up. The Modules screen lists what is installed and stays read-only, -because the alternative would be Fluksio deciding what belongs in an -environment it did not make. + You can install Fluksio in the virtual environment that already holds your + research packages. Nodes use that environment by default. The startup + banner shows the interpreter; see [Packages](../code/nodes.md#packages) + for other setups. -!!! note "Your pins and ours share a site-packages" +## 2. Write a small flow - The cost of not having two environments: a package the engine depends on - is one you can now upgrade out from under it. In practice this is what - everybody does with every other tool in the venv, and the answer when it - bites is the same: pin it back, or keep Fluksio somewhere separate with - the venv of its own below. - -### A venv of Fluksio's own - -Sometimes you want the isolation instead: a shared instance, a container, -or an environment too precious to let a node's dependency near. Set -`NODE_VENV=managed` and Fluksio builds and owns one under the data directory: - -```text - Nodes /home/you/.fluksio/user-venv/bin/python - a venv of its own; the Modules screen installs into it. -``` - -Then the Modules screen is how packages get in: a pip manifest, installed -with `uv pip sync` and versioned alongside your flows, so what a run imported -is recorded with what it ran: - -```sh -curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \ - -H 'Content-Type: application/json' \ - -d "{\"requirements\": $(jq -Rs . < requirements.txt)}" -``` - -Adding a package takes effect immediately; nothing restarts. The Docker image -sets `NODE_VENV=managed` for itself, because the venv in it holds the app and -nothing of yours, so a container is always this case. - -!!! tip "A GPU box works the same way" - - `pip install fluksio-worker` into the environment the training code runs - in, and node code runs on it: `--python` defaults to the interpreter the - agent was started with. It reads its token from `FLUKSIO_WORKER_TOKEN`. - - ```sh - fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach --labels gpu - ``` - - Then give the node `device="gpu"`; see [Remote workers](../code/workers.md). - -## Say which functions are nodes - -A **flow** is a graph of nodes. A **batch flow** is one that runs on demand -from parameters to a result, which is what an experiment is. A node is one of -your own functions. It stays in your repository, imported by its siblings as -it always was. - -Say your project looks roughly like this: +Save this as `study.py` in `research-flow`: ```python -# myresearch/train.py -def fit(dataset, lr, epochs=25): - model = build_model() - for epoch in range(epochs): - loss = step(model, lr) - print(f"epoch {epoch}: {loss}") # ← this is what you are losing - torch.save(model.state_dict(), "weights.pt") - return loss -``` +import json +import math +from pathlib import Path -Two changes turn it into a node, and neither of them moves it: - -```python -# myresearch/train.py import fluksio -from fluksio import Port, node +from fluksio import Flow, Port, node @node( - requires=["dataset", Port("lr", "float")], + requires=[Port("lr", "float"), Port("epochs", "int")], provides=[ Port("loss", "float", stream=True), Port("weights", "artifact"), Port("final_loss", "float"), ], - timeout=600, ) -def fit(dataset, lr, epochs=25): - """Fit the model. A generator, so numbers escape while it is running.""" - model = build_model(fluksio.load_artifact(dataset)) +def fit(lr, epochs): + loss = 1.0 for epoch in range(epochs): - loss = step(model, lr) - yield {"loss": loss} # ← published now, on the loss port - torch.save(model.state_dict(), "weights.pt") + loss = math.exp(-lr * (epoch + 1)) + yield {"loss": loss} + checkpoint = json.dumps({"loss": loss}).encode() return { - "weights": fluksio.save_artifact("weights.pt"), + "weights": fluksio.save_artifact( + checkpoint, "weights.json", "application/json" + ), "final_loss": loss, } -``` -`yield` replaces `print`. Each one publishes on the node's `loss` port the -instant it happens, and the run keeps every value as a series, which is why -Fluksio has no `log_metric()` call. A metric that leaves through a port is a -message like any other: a chart can bind to it, a downstream node can consume -it, and it shows up on the canvas. A metric that escapes through a logging -function is invisible to all three. -`fluksio.save_artifact` handles the things too big to be messages: a -checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a -small reference. Nothing changes about how you write the file. +@node(requires=["weights"], provides=Port("score", "float")) +def evaluate(weights): + checkpoint = json.loads(Path(fluksio.load_artifact(weights)).read_text()) + return 1.0 - checkpoint["loss"] -The decorator returns the function untouched, so `fit(dataset, 0.05)` in a -test, a notebook or a debugger is exactly what it was before. Outside a run -`fluksio.emit` does nothing rather than failing, so a progress report never -stops your own code from running. - -### What is declared, and what is not - -- **Ports are declared, never inferred.** `requires` names the messages the - node reads; each arrives as the parameter of the same name. A bare string is - the shorthand: `"dataset"` takes its type from whoever provides it, so a - message's type is written down exactly once. -- **Parameters with a default that are not ports become settings.** `epochs` - above is stored with the flow and tunable on the canvas without touching - this file. -- **The declaration is checked against the function.** A port with no matching - parameter, or a parameter that is neither port nor setting, is an error when - the module is imported, not when the node is first called. -- **The decorator hands the function back untouched.** `fit(dataset, lr=0.05)` - is an ordinary call, and the module is an ordinary script, so a - `if __name__ == "__main__":` check beside the nodes runs. Syncing one - defined in a script *run as* `__main__` is what cannot work, since the - generated body would have no name to import it by. - -!!! note "Where a `yield` cannot reach" - - If the number comes from inside somebody else's callback (Keras, Lightning, - HuggingFace `Trainer`) `fluksio.emit(loss=...)` writes the same port the - same way: - - ```python - model.fit(callbacks=[LambdaCallback( - on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"]) - )]) - ``` - -### Declaring what a node needs - -A training node is not like the rest of a flow. NumPy, JAX and PyTorch each -size their thread pool to every core they can see, so a few of them at once -oversubscribe the machine badly enough to starve the engine itself: the API -stops answering and every client waiting on a run dies with it. On a GPU the -same shape deadlocks: two processes each preallocating most of the card sit at -zero utilisation with nothing failing and nothing to read. - -Say what one execution takes, and the engine holds it: - -```python -@node(..., resources={"cpus": 2, "env": {"XLA_FLAGS": "--xla_cpu_multi_thread_eigen=false"}}) -def fit(dataset, lr, epochs=25): - ... - -@node(..., resources={"gpus": 1, "env": {"XLA_PYTHON_CLIENT_MEM_FRACTION": "0.9"}}) -def finetune(checkpoint): - ... -``` - -Two things follow. The node waits its turn rather than starting alongside more -of itself than the machine has room for, the same waiting it already does for -a worker. And the worker it runs in is *started* with thread limits matching -what it was given, because that is the only moment a numerical library reads -them; a GPU node is told which card is its through `CUDA_VISIBLE_DEVICES`, and -nothing else is given that card while it runs. - -`env` is for the tuning the engine must not invent. `XLA_FLAGS` is one composed -string whose contents depend on the version you have installed, so writing it -for you would silently replace whatever you had put there. - -The engine counts NVIDIA's device nodes to find its cards, which asks no -vendor tool. A card those do not cover has to be declared (`fluksio serve ---gpus 1`, or `FLOW_GPUS`); until it is, a node asking for one is quietly -given zero and runs alongside every other, and the log says so the first time -it happens. - -Workers are kept warm on purpose, so a library that takes most of the card at -import would hold it after the run finished. The workers that ran on a card -are retired when the run ends, which gives the memory back; what it costs is -the next GPU run paying for its imports again. - -**The declaration is what buys that**, not touching the card. A node that -imports jax without `resources={"gpus": 1}` runs on the shared pool, is never -given `CUDA_VISIBLE_DEVICES`, and leaves a warm worker holding whatever it -preallocated. Declaring the card is what makes it a worker the engine knows -to retire, and what stops two such nodes running at once in the first place. - -Host memory is held the same way, and there the declaration is not needed. A -worker that fitted a model keeps what it allocated, and on a laptop a few warm -ones are the machine. `fluksio serve --worker-max-rss 1024` retires any worker -holding more than that when its node returns, whichever pool it is in. A -node's `ram` is counted against this machine's memory, so declaring it is what -stops three large fits from starting at once. - -Declaring nothing is the default and is right for most nodes: a poll, a -threshold, a message on its way somewhere. Those share the engine's worker pool -and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops -several of them at once from each sizing to the whole box. - -### Ask for a size by name - -Cores and gigabytes are a property of the machines you have, and those change. -A node that names a **flavor** keeps meaning something afterwards: - -```python -@node(..., resources="gpu-small") -def finetune(checkpoint): - ... - -@node(..., resources={"flavor": "medium", "duration_s": "2h"}) -def fit(dataset, lr, epochs=25): - ... -``` - -The flavor is read every time the node is built, so editing it changes what the -next run gets. `fluksio flavors` lists them; the Workers screen edits them. - -`ram` takes `"2G"` and `duration_s` takes `"30m"`, and a flavor already says how -much, so pass one or the numbers, not both. `duration_s` is a statement about the -node for whoever is planning around it, not a limit; the limit is `timeout`. - -### Where it runs - -This is one decision, not two. Of every machine attached (this engine and each -worker) the node goes to one that can grant what it asked for and has it free. -So a node asking for a card finds the box that has one, without naming it: - -```python -@node(..., resources={"gpus": 1}) # wherever there is a card -@node(..., device="gpu", resources="gpu-small") # that box, and this much of it -``` - -If nothing attached can ever grant the ask, it is cut down to what is here and -runs anyway, since a flow written on a cluster still has to work on a laptop. If -something *could* but is busy, the node waits and says so. - -Ask what is free, and what is waiting for it, at -`GET /api/v1/workers/resources`, on the Workers screen, or in `fluksio status`. - -!!! note "Accounted, not enforced" - - Nothing stops a node that ignores its declaration; the numbers are - bookkeeping plus the environment its libraries read. Real enforcement - (cgroups, rlimits) is a next step, not this one. - -## Say which nodes make a flow - -Membership is a list, not a directory layout: the functions can live wherever -they already do. - -```python -# myresearch/pipeline.py -from fluksio import Flow, Port, use - -from myresearch.data import augment, prepare -from myresearch.evaluate import evaluate -from myresearch.train import fit train = Flow( "train", - nodes=[prepare, fit, evaluate], - inputs=[Port("lr", "float", initial=0.01)], - outputs=["score", "final_loss"], + nodes=[fit, evaluate], + inputs=[Port("lr", "float", initial=0.05), Port("epochs", "int", initial=10)], + outputs=["score", "final_loss", "weights"], ) ``` -`inputs` are the run's parameters, with the value a run gets when it names -none; `outputs` are what a run reports as its result. +`@node` declares inputs and outputs. Matching message names connect the nodes: +`fit` produces `weights`, and `evaluate` reads it. `Flow` lists the nodes, +run parameters and final results. -Nodes are connected because one **provides** a message another **requires**, -never because one imported the other. Importing `fit` into a second flow means -"the same code", not "wired to it". +Each `yield` records a loss value. The final `return` sends the checkpoint to +the next node. Artifact ports carry a small reference; the file stays in the +artifact store. -Which is how the same function serves two flows, rewired and reconfigured for -each: - -```python -finetune = Flow( - "finetune", - nodes=[ - prepare, - augment, - use(fit, wire={"dataset": "augmented"}, epochs=3), - evaluate, - ], - inputs=[Port("lr", "float", initial=1e-4)], - outputs=["score"], -) -``` - -`use(fn, ...)` is one use of a node in one flow: `wire` binds a port to a -different message, keyword arguments override settings, and `id=` tells two -uses of one function apart. `train` above is unaffected by any of it. - -A test can check the declarations without an engine. Importing the module -registers its flows in `fluksio.FLOWS`, and `fluksio.build(flow)` returns the -document `sync` would upload, refusing what the engine would refuse. - -## Sync it +## 3. Sync and run ```sh -fluksio sync myresearch +fluksio sync study +fluksio run train --lr 0.1 --epochs 20 --follow ``` -That imports the package, checks every declaration, and uploads each flow with -a generated body per node: +`sync` uploads the flow and generates node code that imports your functions. +Keep the project files available to the engine. Later `run` commands sync +edits before submitting; `--follow` waits and prints metrics as they arrive. +This run should finish with `final_loss` around `0.1353` and `score` around +`0.8647`. -```python -# generated by fluksio sync from myresearch/train.py — edit that file instead -import sys - -_REPO = '/home/you/my-research' -if _REPO not in sys.path: - sys.path.insert(0, _REPO) - -from myresearch.train import fit - - -def process(dataset, lr, **settings): - return (yield from fit(dataset=dataset, lr=lr, **settings)) -``` - -So the flow store still holds a complete, runnable, git-versioned definition; -the body simply happens to import rather than duplicate. Your code stays in -your repository, under your version control. - -The upload is stamped with that repository's commit, and every run records it -alongside the store's own. `--dry-run` prints all of this and uploads nothing; -`--no-publish` leaves it as a draft. - -!!! warning "Sync after you edit" - - The engine's workers are long-lived and hold your imported modules in - memory, so an edit to `myresearch` is invisible until they are retired, - which is what every `fluksio sync` does, including one that has nothing to - upload. If you are editing many times an hour, attach your own interpreter - as a worker instead: it starts a process per call and therefore reads your - code fresh every run. - - ```sh - fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \ - --labels local --python "$(which python)" - ``` - - Then give the node `device="local"`. - -!!! note "The repository path is a deployment detail" - - The generated body puts your repository on `sys.path` by absolute path, - which is right for the machine you synced from and meaningless in a - container. For anything else, install the package where the node runs: - `-e /home/you/my-research` in the module manifest, or - `myresearch @ git+ssh://…@a1b2c3d`, which travels where a path does not. - -### Which hardware a node runs on - -`device` picks the worker, the same way an executor does elsewhere: - -```python -@node(requires=["dataset"], provides=[...], device="gpu", device_policy="prefer") -def fit(dataset, lr, epochs=25): - ... -``` - -`device_policy="require"` (the default) waits for a worker carrying that label; -`"prefer"` runs it locally when none is attached, which is what you want while -the GPU box is not switched on. See [Remote workers](../code/workers.md). - -### What the canvas does with a synced flow - -Its node bodies are generated, so the editor shows them read-only and says -where the real code is. Everything else behaves as usual, but ports and -settings changed there are overwritten by the next sync, which is the point of -the repository being the source of truth. A node whose code you edit on the -canvas makes the next sync stop and say so rather than discarding your edit; -`--force` overrides that. - -## Run it +In the portal, open **Flows → train** to see the graph, or **Runs** to inspect +the parameters, metrics and results. From the CLI: ```sh -fluksio run train --lr 0.003 --seed 7 fluksio runs --flow train +fluksio artifacts +fluksio artifacts train.weights -o weights.json ``` -`run` syncs first, so after an edit the command is just `fluksio run`, with no -step to forget. `--no-sync` skips it for a tight loop where nothing -changed, since syncing retires the workers and the next call pays its imports -again. +Use the ID printed by `run`. Artifact names are flow-qualified; the list +command shows the exact names to download. -`--lr` is typed by the flow's own inputs, so `0.003` arrives as a float. A -parameter you did not declare, or one of the wrong type, is refused before -anything executes. It answers immediately with a queued run, since training is -measured in hours, so nothing waits for it unless you pass `--wait`, or -`--follow`, which waits and prints the numbers as the run reports them. +## 4. Try a few learning rates -No engine has to be up for any of this: `fluksio run --local` boots the -engine inside the command instead, on the same `.fluksio`: the same database, -artifacts and history a served one would use. It costs a few seconds of -startup per invocation against the ~15 ms of submitting to an engine that is -already up, so it is for the run you want now rather than the loop you are -iterating in. +```sh +fluksio sweep train --param lr=0.01,0.05,0.1 --wait +fluksio export metrics --flow train -o curves.csv +fluksio export runs --flow train --status ok -o results.csv +``` -A second run of a flow you did not change mostly does not execute. Each of -your nodes is fingerprinted by its settings, the ports it declares, the values -it reads, and a digest of your repository's Python files, so editing anything -your node calls into invalidates it, not only the decorated function itself. -One an earlier run already answered is restored from that run rather than run -again, reported as `cached`, so a flow with nothing left to do finishes as -`(3/3 cached)`. Change `--lr` and only the nodes downstream of it run. The -digest is blunt on purpose: an edit anywhere in the repository re-runs -everything. `--no-cache` turns caching off for one run, `@node(..., -cache=False)` for one node; the caveat and the details are in -[Stage caching](../concepts/runs.md#stage-caching). +The sweep creates three runs in one group. Select them on **Runs** to compare +their curves. The exports give you a row per metric point and a row per run, +ready for your analysis. -From Python, the flow you declared is also the handle to its runs: +Python can read the same tables: ```python -from myresearch.pipeline import train +from fluksio.sdk.client import Client -run = train.submit(lr=0.003, seed=7).wait() -print(run.status, run.result) -print(run.metrics("train.loss")[-1]) # the whole series is kept -open("weights.pt", "wb").write(run.download("train.weights")) +client = Client() +results = client.export_runs(flow="train", status="ok") +curves = client.export_metrics(flow="train", name="train.loss") ``` -Or over HTTP, which is what both of those are: +Pass either list to `pandas.DataFrame` if you use pandas. -```sh -curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \ - -H 'Content-Type: application/json' \ - -d '{"params": {"lr": 0.003}, "seed": 7}' +## Bring your own experiment -curl -s $FLUKSIO/runs/ -H "Authorization: Bearer $TOKEN" | jq -curl -s "$FLUKSIO/runs//metrics?name=train.loss" -H "Authorization: Bearer $TOKEN" | jq -``` +Replace the toy calculation in `fit` with your training code. Keep ordinary +Python helpers separate from artifact storage so you can still test them +without an engine: `save_artifact` and `load_artifact` work only inside nodes. +The public repository's `examples/myresearch/` package shows this pattern +across several files. -The run carries its parameters, a digest of them, the seed, its result, how -long each node took, what it logged, every artifact it produced, and both -commits: the flow store's and your repository's. That is the answer to "what -was the learning rate on the run that got 94%?". When a run failed, -`run.failures` is the node that did it, with its traceback and its logs. +- Defaulted function arguments that are not input ports become node settings. + Use flow inputs for values you want to vary between runs. +- Use `fluksio.emit(loss=value)` when metrics arrive through a framework + callback instead of a loop you can `yield` from. +- Runs can reuse unchanged stages. Use `--no-cache` when you want every node + to execute; see [Stage caching](../concepts/runs.md#stage-caching). +- `--seed` records a seed and fills an input named `seed` when declared. Your + code still needs to pass that value to its random number generator. -Beside your commit is `code_digest`, and `fluksio runs` prints the pair as -`a1b2c3d-dirty+9f0e1a2`. The commit alone cannot identify what ran: your node -bodies are imports, so the engine executes whatever is on disk when the worker -starts, and an uncommitted tree stamps `-dirty` for every run it ever produces. -The digest is over the modules this flow's nodes actually reach, not the -whole repository, so it moves when the code behind a number moves and stays -put when a notebook beside it changes. It is read at the moment the run -starts, so in a sweep whose runs queue for hours each one records the code -that actually executed it rather than the code that was there when you -submitted. It is what makes the column worth joining an -[exported table](../concepts/runs.md#taking-it-into-a-dataframe) on. +### Declaring what a node needs -### When the engine is busy - -A driver script outlives the engine being slow, because a sweep is hours long -and a blip is seconds: - -- **Reads are repeated.** Every GET, and the writes that mean the same thing - twice, are tried up to three more times on a transport error or a `502`, - `503` or `504`, waiting 1 s, 2 s, then 4 s. `Client(retries=0)` turns that - off. -- **Submitting is safe to repeat.** Each `submit()` mints a key the engine - stores with the run, so an attempt that timed out on the way back is - answered with the run it already made rather than starting a second one. A - sweep keys every entry, so retrying a half-created one creates only what is - missing. -- **Waiting tolerates a gap.** `wait()` and `--follow` survive five failed - polls in a row before giving up. An answer is not a gap: a `404` for a run - that is gone stops immediately. -- **Timeouts are split** — 5 s to connect, 120 s to read, so a wrong URL fails - at once while a busy engine is given longer than the slowest thing it does - on purpose. `Client(timeout=…)` takes a number or an `httpx.Timeout`. - -A command that still cannot reach the engine says -`fluksio: engine not answering (…)` and names the run it started, which is -running on the engine regardless, and `fluksio runs` finds it again. - -## Sweep it - -A grid search and an ensemble are the same submission, run in parallel: - -```sh -fluksio sweep train --param lr=0.001,0.003,0.01 --wait -``` - -Every combination of the lists you give, so a second `--param` is a grid -rather than a second sweep. Where the set you want is not a product, build the -list yourself and post it: - -```sh -curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \ - -H 'Content-Type: application/json' -d '{"runs": [ - {"params": {"lr": 0.001}, "seed": 1}, - {"params": {"lr": 0.003}, "seed": 1}, - {"params": {"lr": 0.010}, "seed": 1} - ]}' -``` - -They share a `group_id`, so the sweep is `GET /runs?group=…`. Running them -concurrently is safe because each run gets a state backend of its own: two -runs of one flow cannot overwrite each other's values. - -Compare the curves in one call: - -```sh -curl -s "$FLUKSIO/runs/series/compare?ids=$A,$B,$C&metric=train.loss" \ - -H "Authorization: Bearer $TOKEN" -``` - -which answers in exactly the shape a chart widget draws. - -## Take it into a dataframe - -The analysis itself wants a table, and `export` writes the two an analysis -asks for: the curves long, and one row per run with the parameters that -varied beside its final numbers: - -```sh -fluksio export metrics --flow train --name train.loss -o curves.csv -fluksio export runs --flow train --status ok -o arms.csv -``` - -Both carry the run id on every row, and `arms.csv` carries the commit and the -code digest beside it, so an exported file still says what produced its -numbers. Numbers inside a record are columns of their own -(`metric.final_metrics.train_loss`) and `--metrics` and `--params` take those -dotted paths to narrow the table. Metric names are flow-qualified, so -`--list` prints every one a selection recorded. `Client.export_metrics()` and -`Client.export_runs()` answer the same rows to a notebook, ready for -`pandas.DataFrame`. - -## Looking at them in the portal - -None of the above needs a browser, but there is one. **Runs** is the table of -everything that has run: filter it to a flow, to a status, or to one sweep, -where it grows a column per parameter that actually varied, and tick two runs -to put their curves side by side. - -A run records its metrics under the same names a dashboard binds to, so a -dashboard is another way of reading one. **Open in dashboard** carries the -runs you picked along in the URL, and every widget then answers for those runs -instead of for the live engine. If the flow has no dashboard yet, the same -menu builds one from the ports the flow declares. - -See [Runs → Looking at what ran](../concepts/runs.md#looking-at-what-ran). - -## What sync does, at the API level - -Nothing here is privileged: `sync` is a client, and a flow is a document you -can PUT yourself. The whole of it, for the one-node case: +For heavy computation, declare resources on the node: ```python -"""Create the `train` flow by hand. What `fluksio sync` automates.""" - -import httpx - -API = "http://127.0.0.1:8000/api/v1" -api = httpx.Client(base_url=API, timeout=60) -token = api.post( - "/login/access-token", - data={"username": "admin@example.com", "password": "example-password"}, -).json()["access_token"] -api.headers["Authorization"] = f"Bearer {token}" - -api.put("/flows/train", json={ - "name": "train", - "title": "Model training", - # Batch: nothing is activated, nothing fires until a run asks. - "mode": "batch", - "inputs": [ - {"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01}, - {"spec": {"name": "epochs", "dtype": "int"}, "initial": 50}, - ], - "outputs": ["final_loss", "weights"], - "nodes": [{ - "id": "train", - "type": "python", - "title": "Fit the model", - # An *idle* timeout once the node streams: this is how long it may go - # quiet, not how long it may run. - "timeout": 600, - "requires": [ - {"name": "lr", "dtype": "float"}, - {"name": "epochs", "dtype": "int"}, - ], - "provides": [ - # `stream` says this port publishes repeatedly during one execution. - {"name": "loss", "dtype": "float", "stream": True}, - {"name": "final_loss", "dtype": "float"}, - {"name": "weights", "dtype": "artifact"}, - ], - }], -}).raise_for_status() - -api.put("/flows/train/nodes/train/source", - json={"code": open("train_node.py").read()}).raise_for_status() - -version = api.get("/flows/train").json()["definition"]["version"] -api.post("/flows/train/publish", json={"version": version}).raise_for_status() +@node(requires=["dataset"], provides=Port("score", "float"), + resources={"cpus": 2, "ram": "2G"}) +def score_dataset(dataset): + return {"score": calculate_score(dataset)} ``` -Two things worth noticing, because the decorators only move where they are -said. Ports are declared, not inferred: `process(lr, epochs)` gets its -arguments from the ports of the same name, and the types are checked on every -value. And saving writes a *draft*; `publish` is what the engine picks up. -That separation is what lets you edit a flow that is running. +The engine accounts for these resources and sets worker thread limits. GPU +nodes can request `resources={"gpus": 1}`; check that the Workers screen +reports your GPU, or set `fluksio serve --gpus 1` when automatic detection +does not cover it. Resource accounting does not impose a hard memory limit. -A flow uploaded this way carries no `origin`, which is what tells the canvas, -and the next `fluksio sync`, that it was not generated. See -[The API](../code/api.md). +To run on another machine, attach a [remote worker](../code/workers.md) and +install your project and its dependencies there. -## Small scripts you are just playing with +## Next steps -The same machinery, minus the ceremony. If what you want is "keep a record of -every version of this thing I keep tweaking", note that **your flows are -already a git repository**: - -```sh -cd .fluksio/flows -git log --oneline -``` - -Every save is a commit: the node source, the parameters, the graph. A run -records the commit it ran at, so `git show` on that hash is literally the code -that produced the number. You get the versioning without adopting anything. - -For quick iteration, keep the flow small (one node is fine), keep the engine -running, and submit from wherever you are working: - -```python -from myresearch.pipeline import train - -for lr in (0.001, 0.003, 0.01): - train.submit(lr=lr) -``` - -A submit is around 15 ms, so calling that in a loop is a reasonable thing to do. - -## Get a browser onto it - -The pip install gives you the engine and the API, not a web interface, since a -machine with no inbound route cannot serve one usefully. To see the -canvas, the run history and live loss curves, pair the instance with a -portal, which serves the dashboard from its side: - -```sh -fluksio enroll --portal https://hub.fluksio.com -``` - -Get the claim code from the portal under **Instances → Add instance**. -Nothing needs to be exposed: your machine dials out and holds the connection -open. See [Accounts and the portal](../interface/portal.md). - -If you would rather stay entirely offline, run the dashboard SPA yourself from -the app's Docker image; see the [facility path](facility-automation.md), which -is the same stack. - -## Where to go next - -- [Runs: pipelines that finish](../concepts/runs.md) — the full picture: - artifacts, sweeps, durability, what happens when your engine dies mid-training -- [Writing node code](../code/nodes.md) — generators, settings, what a node may - and may not do -- [The command line](../code/cli.md) — `login`, `sync`, `run`, `runs`, `sweep` - in full -- [Remote workers](../code/workers.md) — send the training node to the GPU box - and keep the rest on your laptop -- [The flow editor](../interface/flow-editor.md) — once you have a portal, this - is where the graph gets easier to change than the script did +- [Writing node code](../code/nodes.md) — ports, streaming, artifacts and packages. +- [Runs](../concepts/runs.md) — caching, comparisons and retries. +- [CLI reference](../code/cli.md) — command options and instance settings. diff --git a/docs/getting-started/facility-automation.md b/docs/getting-started/facility-automation.md index 86db8bb..e0b850b 100644 --- a/docs/getting-started/facility-automation.md +++ b/docs/getting-started/facility-automation.md @@ -1,325 +1,164 @@ # Getting started: facility automation -You have a box in a cupboard, a handful of sensors that already publish -somewhere, and an ambition to make the house do something about them. This -page brings up a Fluksio instance you can leave running for years, then wires -the first sensor through to a dashboard. +Build a small temperature flow, then connect a real sensor when you're ready. +No hardware required for the first lap. -Budget an afternoon. Most of it is the browser, which is the point: you will be -in this interface a lot, so it is worth learning it properly. +## Start the Docker stack -## What you are standing up - -```text - sensors ──MQTT──▶ ┌──────────────┐ ──▶ InfluxDB (history) - │ Fluksio │ - HTTP / webhooks ──▶│ flow engine │ ──▶ dashboards (what you look at) - │ │ - schedules ────────▶└──────────────┘ ──▶ MQTT / HTTP (what you control) - │ - └── workers on other boxes (optional) -``` - -One engine holds every flow. Nothing here is a plugin you install separately: -the broker client, the time-series writer, the dashboards and the alerting are -all part of the same process, editing the same graph. - -## Prerequisites - -- Docker and Compose v2 on the host -- The address you will type into a browser. On a home network that is the - server's own address, `192.168.1.50`, and nothing else is needed. A - hostname is optional, and covered below -- Optionally: an MQTT broker and an InfluxDB you already run. If not, the stack - can start both for you - -## Set it up - -Clone the app repository and try it on the machine you are sitting at: +You need Git, Docker with Compose v2, `make`, and `openssl`. Port 80 must be +free. Clone the public app repository and run its commands from the repo root: ```sh -git clone https://git.stroblme.de/Fluksio/app.git ~/fluksio -cd ~/fluksio -make init # once: .env with generated secrets, `proxy` network, prints the admin login -make dev-local # full stack at http://app.localhost (API at http://api.localhost) -make down # stop it +git clone https://git.stroblme.de/Fluksio/app.git fluksio +cd fluksio +make init +make dev-local ``` -`make dev-local` needs port 80 free and only answers on `localhost`; the -sections below make it reachable from other machines. `make init` is safe to -run again: it only fills secrets still set to `changethis`, and `.env` -otherwise stays as you left it. +`make init` creates `.env`, generates secrets, creates the `proxy` network, +and prints your admin login. It is safe to run again: existing secrets stay +as they are. -For an instance you leave running, edit `.env` first (`FIRST_SUPERUSER`, -`ENVIRONMENT=production`). It is the whole configuration. The four settings that matter on day one: +Give the containers a moment to start on the first build, then open +[http://app.localhost](http://app.localhost). Sign in with the admin login: +adding Python code requires a superuser account. The API is at +[http://api.localhost](http://api.localhost), and the docs are at +[http://docs.localhost](http://docs.localhost). -| Setting | What it does | -|---|---| -| `DOMAIN` | the hostname everything is served under, if you use one. Left alone it only names the issuer that agents authenticate against | -| `FIRST_SUPERUSER` | the account you sign in with | -| `FIRST_SUPERUSER_PASSWORD` | generated by `make init`; it prints it, and it stays in `.env` | -| `ENVIRONMENT` | `production` closes the interactive API schema; `local` leaves it open | - -Everything the instance owns (the database, your flows, secrets, artifacts, -the packages your node code imports) is on one Docker volume. -Backing that volume up is backing up the instance. - -??? note "Even smaller: no Docker at all" - - `pip install fluksio && fluksio serve` gives you the same engine with no - containers, keeping its data in `~/.fluksio`. What it does not give you is - the web interface, which the SPA container serves, so you would drive it - from the API, or [pair it with a portal](../interface/portal.md) that - serves the dashboard for you. Good for a Raspberry Pi that only runs flows; - less good as your main instance. - -## Reach it from another machine - -Start it one of two ways, depending on whether you have a hostname to point at -it. Neither is more supported than the other. - -### On your own network, by address - -The usual case: the box is in a cupboard at `192.168.1.50` and you open it from -a laptop. Layer `docker/compose.lan.yml` on, which publishes the interface on a -port and serves the API from that same origin: +For a Docker instance reached from another machine on your network, use: ```sh -APP_PORT=8080 docker compose -p fluksio-app --env-file .env \ - -f docker/compose.yml -f docker/compose.lan.yml up -d --build +make dev-lan ``` -Open `http://192.168.1.50:8080`. That one address is the whole application: -the interface, the API under `/api`, and the websocket that puts live values on -the canvas. +Open `http://:8080`. Choose another port with +`make dev-lan APP_PORT=8081`. This is a development stack; see +[running it long term](#run-it-long-term) before using it for a deployment. -Nothing in that URL is configured anywhere. The interface is built to address -the API relative to whatever origin served the page, so the same containers -answer to the machine's address, to a hostname you add later, and through an -ssh tunnel, without being rebuilt. It is also why no CORS list needs -maintaining: the browser is talking to one origin. - -!!! danger "This is your network, not the internet" - - A published port is plain HTTP with no certificate. Between your own - machines that is fine, and it is not something to forward from a router. To - reach the house from outside, put a reverse proxy with TLS in front of it, - or [pair it with a portal](../interface/portal.md), which needs no inbound - route at all. - -### With a hostname, behind a reverse proxy - -If you have DNS, or want certificates, set `DOMAIN` and run `make up` -instead. The interface is then served on `app.${DOMAIN}` and the API on -`api.${DOMAIN}`, and nothing is published on a host port. - -The stack emits Traefik labels and ships a Traefik you can bring up alongside -it (`docker/compose.traefik.yml`). With Nginx Proxy Manager or Caddy instead, -attach it to the `proxy` network and forward `app.${DOMAIN}` → -`fluksio-app:80` and `api.${DOMAIN}` → `fluksio-api:8000`. - -!!! tip "Developing on the same machine" - - `make dev-lan` is `make dev-local` plus the published port: the app on - `http://:8080`, `APP_PORT` to move it. `make dev-local` recreates - the stack without that port, so re-run `make dev-lan` after one. - -## Sign in - -Open whichever address you chose (`http://app.localhost` for `make dev-local`) -and sign in with the login `make init` printed. You should be looking at Home: -an empty brain graph, a health summary, and a flow list with nothing in it. +When you are done, stop the stack with `make down`. Your data volumes stay +in place for the next start. ## Your first flow -Go to **Flows → New flow** and call it `house`. You land on the canvas. +Open **Flows**, create a flow named `house`, and use **Add node** to add an +**Inject** node. In its panel: -A flow is a set of nodes that talk to each other through **named messages**. -You do not draw wires: a node says which messages it needs and which it -produces, and the canvas draws the graph that follows. Renaming is safe, fan-in -is free, and two flows share a value by naming it. +- Under **Provides**, add `living_temperature` with type `float`. +- Under **Settings**, set the `living_temperature` value to `18.0` and + **Interval** to `5` seconds. -### Read a sensor +This gives you a pretend sensor. Add a **Function** node with: -Press **Add node** (or ⌘P / Ctrl-P, which opens the command palette) and pick -**MQTT**. In its panel on the right: +- **Consumes**: `living_temperature`, type `float`. +- **Provides**: `heat_wanted`, type `bool`. -- **Broker host** — your broker's hostname, `mosquitto` if you are using the - one the stack can start -- **Topic** — map each output to a topic: - `{"living_temperature": "zigbee2mqtt/living/temperature"}` -- **Provides** — add one output port named `living_temperature`, type `float` - -That is a working node. Press **Publish** (⌘S) and the engine picks it up. - -The canvas now draws your node with a live value on its output as soon as the -broker sends one. Click the wire to see the last payload and its history. - -### Do something with it - -Add a **Function** node. This is a Python node, and the code editor opens in -its panel: +In the code editor, enter: ```python -def process(living_temperature, comfortable=21.0): - """Ask for heat when the room is below the comfort point.""" - return {"heat_wanted": living_temperature < comfortable} +def process(living_temperature): + return {"heat_wanted": living_temperature < 21.0} ``` -Declare `living_temperature` as an input (type `float`) and `heat_wanted` as an -output (type `bool`). `comfortable` is not a port but a **setting**, -because it is a constant of this node rather than something the graph carries. -It shows up as a field in the node's Settings section. +Message names connect the nodes automatically; there are no wires to draw. +Save the code with ⌘S / Ctrl-S, then click **Publish** in the dock. Open +**Flow settings** from the dock's pencil button and make sure **Running** is on. -The canvas now draws MQTT → your function, because the message names line up. -Nothing else was needed. +Every five seconds, `18.0` should produce `true`. Select the Function and check +the value under its `heat_wanted` output. Change the Inject value to `22.0`, +publish again, and check that the result becomes `false`. Click the edge to +inspect the temperature readings. Your first bit of automation is alive. -### Act on it +If nothing happens, open **Issues** and **Logs** in the dock. They show missing +inputs, code errors and connection problems. See [The flow editor](../interface/flow-editor.md) +for the controls and shortcuts. -Add a second **MQTT** node, this time with `heat_wanted` as an *input*, and a -topic mapping to whatever your relay listens on. A node with inputs publishes; -a node with outputs subscribes. +## Connect a real sensor -Publish the flow. You have a thermostat. +Replace the Inject node with an **MQTT** node: -!!! warning "Test before it touches a relay" +| Field | Value | +|---|---| +| **Provides** | `living_temperature`, type `float` | +| `broker_host` | your broker's hostname or IP, reachable from the engine | +| `topic` | the sensor's temperature topic | - A flow can be **paused** (it holds messages instead of running them) and - **stepped** (release exactly one). Together with the run button, which - injects a value by hand, that is how you convince yourself the logic is - right before the contactor finds out. Both live on the dock at the bottom - of the canvas. +Publish the change. The Function now responds to sensor readings instead of +the test value. If your sensor sends a JSON object such as +`{"temperature": 18.0}`, set `json_key` to `temperature`. -## Store the history +An MQTT node with input ports publishes to its configured topic; one with +output ports subscribes. Add a publishing node only after checking the payload +format your device expects. A relay may need `ON` and `OFF` rather than a +boolean. -A live value is enough to control something and useless for answering "was last -February colder?". That is what the **InfluxDB** node is for. +!!! tip "Keep the first test on screen" -Add one, and configure it to write the message you already have: + Check both warm and cold readings before connecting an actuator. **Pause** + and **Step** let you inspect queued messages one at a time. A real heating + controller also needs appropriate limits and hysteresis. -```json -{ - "url": "http://influxdb:8086", - "token": {"$secret": "influx-token"}, - "org": "home", - "bucket": "sensors", - "writes": { - "living_temperature": { - "measurement": "environment", - "field": "temp_c", - "tags": {"room": "living"} - } - } -} -``` +Docker development stacks include a test broker at `mosquitto:1883`; it has +no published host port, so your physical sensors cannot reach it directly. +For real sensors, use your own broker. The Docker image includes the MQTT +and InfluxDB connector dependencies. -Give it `living_temperature` as an input and every value that passes gets a -point. +## Show it on a dashboard -Note the token. Credentials never sit in a flow: `{"$secret": "influx-token"}` -is a reference into an encrypted store, and the editor renders those fields as -a secret picker. Add the actual value once under **Secrets**. Flows are a git -repository you might well push somewhere, so this keeps a password out of it. +Create a dashboard and add a gauge bound to `house.living_temperature`. +Publish the dashboard to see the current reading. Dashboard controls can also +publish messages, so choose their bindings with care. -Reading back is the same node with `queries` instead of `writes`, or, for -anything a chart asks for, a pair of small Python nodes on either side that -build a Flux query and shape its rows. That indirection is deliberate: the -database node holds the connection and nothing else, so a dashboard widget -never learns which database answered it. +See [Dashboards and panels](../interface/dashboards.md) for widgets and pairing +a wall tablet. -## Put it on a screen +For longer-term history, add an **InfluxDB** node that consumes +`living_temperature`. Configure your server URL, organization, bucket, token +and `writes` mapping. Store the token under **Secrets** and select it in the +node's credential field. [Node types](../reference/node-types.md#influxdb) +has the write and query formats. -**Dashboards → New dashboard**, then drag widgets onto the grid and bind each -one to a message. A gauge on `house.living_temperature`, a switch on -`house.heat_wanted`, a chart on the history. +The Docker development stack includes an InfluxDB test service at +`http://influxdb:8086`, with organization and bucket `fluksio` and token +`fluksio-dev-token`. It is for testing; configure a persistent InfluxDB server +for data you want to keep. The production stack includes neither InfluxDB nor +an MQTT broker. -Widgets are typed the same way ports are: a switch binds to a `bool`, a gauge -to a number, an agenda to a `list`. Bind it wrong and the editor says so rather -than drawing nothing. +## Run it long term -Controls work in the other direction: a switch on a dashboard publishes the -message it is bound to, exactly as a node would. The canvas draws it as a -labelled endpoint feeding the nodes that read it, so a value never appears from -nowhere. +For a Docker deployment, run `make init`, then edit the repository's `.env`: -For a tablet on the wall, see [Dashboards and panels](../interface/dashboards.md): -a panel is a named device, it pairs with a six-character code instead of a -login, and it can only reach the dashboards you gave it. +| Setting | Example | +|---|---| +| `DOMAIN` | `example.com` | +| `ENVIRONMENT` | `production` | +| `FRONTEND_HOST` | `https://app.example.com` | +| `VITE_API_URL` | `https://api.example.com` | +| `BACKEND_CORS_ORIGINS` | `https://app.example.com` | +| `FIRST_SUPERUSER` | your administrator email | +| `TZ` | your timezone, such as `Europe/Berlin` | -## Spread it across machines +Keep the generated secrets. Stop any development stack with `make down`. +Set up DNS and a reverse proxy with HTTPS and WebSocket support before running +`make up`. That target starts the application services; it requires a proxy +on the Docker `proxy` network. Route +`app.example.com` to `fluksio-app:80`, `api.example.com` to `fluksio-api:8000` +and, optionally, `docs.example.com` to `fluksio-docs:80`. -You now have one box doing everything. Two reasons to change that: something -lives on a different network, or something needs hardware the engine's host -does not have. +`docker/compose.traefik.yml` is an optional proxy configuration. It also needs +`EMAIL`, `USERNAME` and `HASHED_PASSWORD` for certificates and dashboard +authentication; `make init` does not set these up. -The unit of distribution is the **worker**. It runs the code of nodes you mark -for it, and it dials *out* to the engine, so the Pi in the shed does not need -an inbound route, and the engine does not need to reach it. +The published port from `make dev-lan` uses HTTP. For access outside your +network, use HTTPS or [enroll with the portal](../interface/portal.md). -On the engine, mint a token: +Back up `.env` and both persistent volumes: `app-flow-data` holds the database, +flows, secrets and artifacts; `app-redis-data` holds message values, node state +and queued work. Compose prefixes volume names with the project name +`fluksio-app`. Also back up any external MQTT or InfluxDB data you need. -```sh -curl -X POST https://api.${DOMAIN}/api/v1/workers/tokens \ - -H "Authorization: Bearer $TOKEN" -d '{"name": "shed-pi", "labels": ["shed", "gpio"]}' -``` +## Next steps -The labels are signed into it, so the worker can repeat them with `--labels` -but not change them. - -On the other machine, with that token in `FLUKSIO_WORKER_TOKEN`: - -```sh -pip install fluksio-worker -fluksio-worker \ - --url wss://api.${DOMAIN}/api/v1/workers/attach \ - --labels shed,gpio \ - --parallel 2 -``` - -Then mark the node that talks to the shed's GPIO with `device: shed`, and it -runs there. Everything else stays where it is. A node bound to a label no -attached worker carries simply waits rather than failing, so you can write the -flow before the hardware arrives. - -!!! note "What a worker is not" - - It is not a second engine. Subscriptions, schedules, webhooks and the - dashboards all stay in one process, which keeps a value having one - definition. A worker executes node bodies, nothing else. Scaling *the - engine* to several processes is not supported: run one. - -## Make it tell you when something breaks - -Under **Alerts**, add a channel and a rule. Channels are ntfy (a push -notification on your phone), SMTP, a webhook, or a message a dashboard -notification widget reads. - -What you probably want on day one is everything, to ntfy: - -- add an ntfy channel with your server and topic -- add a rule with no events ticked, which means *all of them* - -The engine deduplicates aggressively, so the same node failing every second is -one alert rather than thirty-six thousand, and caps the total at ten an hour -however bad it gets. See [Secrets, modules and alerts](../interface/operations.md). - -## Keep it alive - -- **Back up the data volume.** That is the database, the flows, the secrets and - the artifacts. Everything else is rebuildable. -- **Watch Home.** The health summary names what is wrong — a quarantined flow, a - node that will not load, a stalled queue, a flow that cannot run because its - graph does not validate. -- **Flows are git.** `git log` inside the flow store is the history of every - change anyone made, and reverting one is a revert. - -## Where to go next - -- [Flows, nodes and messages](../concepts/flows.md) — the model, properly -- [Node types](../reference/node-types.md) — everything you can put on a canvas - without writing Python -- [The flow editor](../interface/flow-editor.md) — the canvas, in detail -- [Keeping state in a flow](../concepts/state.md) — running totals, debounces, - and the one rule that makes them safe -- [Accounts and the portal](../interface/portal.md) — reach the instance - from outside the house without opening a port +- [Flows, nodes and messages](../concepts/flows.md): how the graph works. +- [Secrets, modules and alerts](../interface/operations.md): credentials and notifications. +- [Remote workers](../code/workers.md): run node code on another machine. +- [Configuration](../reference/configuration.md): settings and storage paths. diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md index 668ce84..3a55bd5 100644 --- a/docs/getting-started/index.md +++ b/docs/getting-started/index.md @@ -1,9 +1,8 @@ # Pick your starting point -Setup differs depending on what you are here for. Pick the path that matches. - -Pick the one that sounds like you. Everything past this section is the same for -both. +Choose the setup that fits your work. +Both paths lead to the same Fluksio engine, with a first flow to try along the way. +This is really just about a decision what you're planning to do with Fluksio.
@@ -11,13 +10,13 @@ both. ### Data science -*"I have a training script. I want to stop losing track of what I ran."* +*"I have some Python code. I want to keep track of what I ran."* -One `pip install`, one command, and you are writing Python again. No Docker, no -database, no ports to open. Flows are files, runs are rows, and the metrics are -just the numbers your loop already produces. +Install Fluksio with pip in your research environment. Turn a function into a +node, record metrics and artifacts, and compare experiments. Connect to +[fluksio.com](https://fluksio.com) for the browser dashboard. -[Set up for experiments →](data-science.md) +[Start with Python →](data-science.md)
@@ -25,46 +24,20 @@ just the numbers your loop already produces. ### Facility automation -*"I have a homelab and a pile of sensors. I want them to do something."* +*"I have a bunch of sensors. I want them to do something."* -A stack you bring up once and leave running: the engine, a broker, a -time-series database, dashboards, alerting. Most of the work happens in the -browser, and it is worth doing properly because you will live in it. +Start the Docker stack for a local engine and browser interface. Build a live +flow, connect your devices, and put their values on a dashboard. -[Set up a homelab instance →](facility-automation.md) +[Start with Docker →](facility-automation.md) -## Not sure? +## Next steps -Some rough tells: - -| | Data science | Facility automation | -|---|---|---| -| **The flow** | starts, finishes, has a result | never ends | -| **You mostly** | write Python | wire nodes in the browser | -| **Time to first result** | a few minutes | an afternoon | -| **Runs on** | your laptop, or a login node | a box in a cupboard | -| **Data lives in** | SQLite beside the flows | InfluxDB, usually | -| **The thing you look at** | run history and loss curves | a dashboard, maybe on a wall | - -If both describe you (a lab with instruments to drive *and* models to fit), -start with the data-science path. It is the smaller instance, and it -grows into the other one without being reinstalled: the same engine, the same -flows, just more of them running all the time. - -## What is the same either way - -Whichever door you came in: - -- **Flows are files in a git repository.** Every save is a commit. You can read - the history with ordinary git, and you can copy a flow between instances - by copying a directory. -- **Editing is separate from running.** You edit a draft; the engine keeps - running what was published until you publish. -- **Nodes are typed.** A port declares what it carries, and a mismatch is - caught at edit time rather than at three in the morning. -- **Everything the browser does is an API call.** The dashboard is a client of - the same REST API you can script against. +- [Flows, nodes and messages](../concepts/flows.md) explains the graph. +- [The flow editor](../interface/flow-editor.md) covers editing and publishing. +- [The command line](../code/cli.md) covers local and remote use. +- [Configuration](../reference/configuration.md) lists settings and data paths. diff --git a/docs/index.md b/docs/index.md index 3f82d33..19add5c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,53 +1,33 @@ # Fluksio -Fluksio is a node-based automation engine. You describe what should happen as a -graph of small pieces of logic, and it keeps that graph running: reacting to -what arrives, or executing once from parameters to a result. +Fluksio is a node-based automation engine for experiments, buildings, and +homelabs. Connect small pieces of logic through typed messages, then let the +engine do the running. -It runs two kinds of graph: +- **Live flows** react to sensors, webhooks, and schedules. +- **Batch runs** take parameters, finish with a result, and keep metrics, + artifacts, and a history you can compare. -- A **live flow** produces values forever. A sensor publishes, a rule fires, a - relay closes, a dashboard on the wall shows what happened. It never ends. -- A **run** produces a value once. Parameters go in, stages execute, and it - finishes, leaving metrics, artifacts and a record to compare against. - -Both use the same nodes, the same type checking, the same editor and the same -API. - -## What you get - -- A **flow engine** that owns your graph, checks the types on every edge, and - keeps running when a node fails. -- A **canvas** that lays flows out for you and an editor for the Python inside - each node, with the running values drawn on the wires while you work. -- **Dashboards** built next to the logic that feeds them, including ones you - can hang on a wall tablet that has no keyboard. -- **Runs**: parameters, metrics, artifacts, sweeps and a queryable history, with - no second server and no project bootstrap per execution. -- **Distributed workers**: a node marked `device: gpu` runs on the machine that - has one, which dials out to the engine rather than needing to be reachable. -- An **HTTP API** covering everything the browser does, plus an MCP endpoint for - agents. +Both use the same nodes, editor, and API. Flows are versioned in git, so your +next bright idea comes with an undo trail. ## Start here -[Getting started](getting-started/index.md) has two paths: one installs a Python -package, the other stands up a server. Everything after that is shared. +[Pick your starting point](getting-started/index.md): -To look before installing, the hosted demo at -[fluksio.com](https://fluksio.com) runs a real instance: a small-house panel, a -media screen, and a lab where a training pipeline runs and its runs are -compared, four screens on one panel. +- [Data science](getting-started/data-science.md): install with pip in your + research environment and run your first experiment. +- [Facility automation](getting-started/facility-automation.md): start the + Docker stack and build your first live flow in the browser. -## Where things are +## Explore -| If you want to… | Read | +| To… | Read | |---|---| -| Understand what a flow, a node and a message are | [Concepts](concepts/flows.md) | -| Drive Fluksio from the browser | [The interface](interface/index.md) | -| Drive it from Python, a shell or CI | [Code and the CLI](code/cli.md) | -| Look up a node type or a payload type | [Reference](reference/node-types.md) | +| Understand flows, nodes, and messages | [Concepts](concepts/flows.md) | +| Edit flows and build dashboards | [The interface](interface/index.md) | +| Use Python, the CLI, or the API | [Code and the CLI](code/cli.md) | +| Look up nodes, payloads, and settings | [Reference](reference/node-types.md) | -Fluksio is self-hosted by default. An instance runs offline, keeps its data -on its own disk, and never contacts anything unless you -[connect it to a portal](interface/portal.md) yourself. +Your instance keeps its data on its own disk. Connecting it to a +[portal](interface/portal.md) is optional; local flows and runs work offline. diff --git a/docs/interface/dashboards.md b/docs/interface/dashboards.md index 6660a78..446e8e1 100644 --- a/docs/interface/dashboards.md +++ b/docs/interface/dashboards.md @@ -1,324 +1,217 @@ # Dashboards and panels -A dashboard is a grid of widgets bound to message names, the same names that -wire the graph. It is its own document rather than a set of nodes, so it reads -across flows without belonging to any of them, and a flow stays the logic it -was. - -There is no second tool and no separate deployment. You build the panel next to -the thing feeding it. +A **dashboard** is a grid of widgets bound to message names. It can show values +and send controls across several flows. A **panel** assigns published dashboards +to a paired screen, such as a wall tablet. ## Building one -**Dashboards → New dashboard**, then drag widgets onto the grid from the widget -picker and bind each to a message. +1. Open **Dashboards → New dashboard**. +2. Enter edit mode, choose **Add widget** and select a type. +3. Bind the widget to a message in its settings. On desktop, drag and resize + tiles on the grid. +4. **Publish** when the dashboard is ready. -Like flows, dashboards are drafts until you publish. A wall panel reads only -the published version, so a half-arranged page never reaches the wall. +Edits stay in a draft; paired screens show the published version. On phones, +tiles stack in one column and dragging is disabled, but settings remain editable. -On a phone the dashboard is read, not arranged: the grid stacks to one column -and dragging is off. Picking a widget and editing its settings still works. +Controls can publish real values while you edit, so choose their targets with +care. Your new Switch may already be flipping something! ## The widgets ### Display -| Widget | Binds to | Notes | +| Widget | Binds to | Shows | |---|---|---| -| **Value** | anything | a formatted reading with a unit and a precision | -| **Gauge** | `float`, `int` | min, max, unit | -| **Chart** | `float`, `int` | up to five series; see *Two kinds of chart* below | -| **Bar** | `float`, `int` | up to eight readings, a row each, with a scale per row | -| **Icon** | numbers, booleans, weather strings | maps a value onto a glyph | -| **Text** | — | markdown you write; a label, a note, an instruction | -| **Agenda** | `list` | upcoming items, e.g. from a calendar connector | -| **Forecast** | `list` | a short outlook strip | -| **Notification** | `record` | title, body and severity — what an alert channel writes | -| **Media** | `image`, `audio`, `video` | a camera frame, a clip; see *Media tiles* below | -| **Player** | `record` | what a streamer is playing, with its transport; see *Player tiles* below | -| **Clock** | — | the time, in a size a wall can read | -| **Website** | — | another site's page in the tile; see *Website tiles* below | +| **Value** | anything | formatted reading, unit and precision | +| **Gauge** | `float`, `int` | reading within a min/max range | +| **Chart** | numeric messages, a query response or run metrics | up to five lines | +| **Bar** | `float`, `int` | up to eight readings with separate scales | +| **Icon** | numbers, booleans or strings | a glyph selected from the value | +| **Text** | — | Markdown | +| **Agenda** | `list` | upcoming items | +| **Forecast** | `list` | an outlook strip | +| **Notification** | `record` | title, body and severity | +| **Media** | `image`, `audio`, `video`, `artifact` | image, audio or video | +| **Player** | `record` | playback status and transport controls | +| **Clock** | — | current time | +| **Website** | — | embedded page or image | -Every widget carries a **title**, and **Show title** decides whether the panel -draws it. Turned off, the tile is just the reading, which is what a row of -gauges under one heading wants. The title is still the widget's name: what a -screen reader calls its controls, and what a published value is labelled with. +Every widget has a title. **Show title** controls its visibility; the title +still names the widget for accessibility and published values. ### Input -| Widget | Publishes | Notes | +| Widget | Publishes | Settings | |---|---|---| -| **Button** | a fixed value | one-shot: run it, open it, reset it | +| **Button** | fixed value | one-shot action | +| **Buttons** | a value per button | grid of actions | | **Switch** | `bool` | on/off | -| **Slider** | `float`, `int` | min, max, step; **Drawn as** makes it a vertical fader | -| **Input** | text or a number | free entry | -| **Selector** | one of a list | a mode, a scene, a preset — as a menu, or as a row of choices with the active one held | -| **Colour** | `[h, s, v]`, `[r, g, b]` or `"#rrggbb"` | a hue wheel with saturation and brightness, for an RGB fixture | +| **Slider** | `float`, `int` | min, max, step; horizontal or vertical | +| **Input** | text or number | free entry | +| **Selector** | selected value | menu or row of choices | +| **Colour** | list or string | HSV, RGB or hex | -The colour wheel sends `[h, s, v]` by default (hue 0-360 degrees, saturation -and value 0-100 percent, which is what a DMX encoder expects) and its -**Sends** setting switches that to `[r, g, b]` (0-255 each) or to a -`"#rrggbb"` string, because fixtures differ. The first two bind a `list` -message, the third a `str`. +Colour defaults to `[h, s, v]`: hue 0–360°, saturation and brightness 0–100%. +**Sends** can select RGB (`[r, g, b]`, 0–255) or hex (`"#rrggbb"`). +HSV and RGB bind to `list`; hex binds to `str`. -A control publishes the message it is bound to, exactly as a node would. On the -flow canvas it is drawn as a labelled endpoint feeding the nodes that read it, -so nobody has to guess where the value came from. +Controls publish to their bound messages. The flow canvas shows them as +endpoints feeding the nodes that consume those messages. -### Typed bindings +Bindings are checked in the editor and on the server. See +[Payload types](../reference/payload-types.md). -Widgets are type-checked against the message the same way ports are: a switch -takes a `bool`, a gauge takes a number, an agenda takes a `list`, a -notification takes a `record`. Bind one wrong and the editor says so rather -than drawing nothing. The same table is enforced on the server. +## Charts -Charts draw their series in one fixed order of colours: the primary first, then -hues picked to stay apart, colour blindness included. A chart with more than -one series still always draws a legend, so no line is told by colour alone. +Choose the chart's data source: -## Two kinds of chart +- **Live:** recent numeric message history, updated as values arrive. +- **Query:** publishes a `record` request containing `range_s` and + `interval_s`, then draws a flow's `series` response. Include those fields in + the answer so the chart can reject responses for a different window. +- **Runs:** compares a numeric metric from named runs, a sweep or the latest + runs of a flow. See [Runs](../concepts/runs.md). -**Live.** Bind up to five messages and the chart draws the engine's own ring -buffer of recent values, extended over the socket as new ones land. Nothing -else is involved. This is what you want for "the last few hours of the living -room". +Multi-line charts show a legend as well as different colours. -**Querying.** The chart publishes a *request* (the window and resolution it -wants) exactly as a slider publishes a value, and draws the `series` some flow -answers with. What serves that request is the flow's business: typically a -small Python node that builds a Flux query, an InfluxDB node that runs it, and -another Python node that shapes the rows. +!!! tip "Skip the unfinished bucket" -The widget never learns which database answered it, so swapping the store -changes one flow and nothing else. An answer states what it was computed for, -so an answer to a different question is ignored. - -!!! note "Drop the bucket that is still filling" - - The request carries a window and an interval, and the binning is the flow's - own work, so the newest bucket only ever holds the part of an interval - that has elapsed. Drawn, it reads as a fall that never happened. The - engine's own [Activity charts](index.md#activity) end on the last closed - bin for that reason; a flow answering a chart has to drop or hold back its - newest bucket the same way. + If your query aggregates into time buckets, leave out the bucket still + filling. Otherwise an incomplete total can look like a sudden drop. ## Media tiles -A media widget draws what its message points at: a picture, a clip with -controls, a video. Media does not travel as a message; a reference to it does. -The tile takes the bytes behind whichever reference the message holds, and -redraws when a new one arrives. +Media messages contain artifact references; the widget loads the bytes they +name. **Crop or fit** controls image sizing. **Play as it arrives** enables +autoplay, subject to the browser's rules for sound. -**Crop or fit** decides how a picture fills the tile. **Play as it arrives** -starts a clip by itself, though a browser only plays sound once somebody has -touched the page, so a screen nobody has tapped stays silent. +For local streams, `volatile=True` artifacts can be pushed over the websocket +with their message. Stored artifacts are fetched individually, as are frames +viewed through a portal. Use occasional snapshots for remote screens. -Rate is the thing to get right, and it is decided by the node publishing rather -than by the tile. A frame the engine is holding in memory is pushed down the -same websocket that carries the value, so ten a second is a real view on the -local network; a stored one is fetched, a round trip each, which suits a glance -at a door every second or two. Through the portal everything is fetched, so -make it every few seconds there. Above that, put the camera's own address in -**Live stream** and the browser plays it from source, leaving the messages to -carry the occasional still that a flow can actually react to. - -Panels see media the same way, and only their own: a screen is sent, and may -fetch, the bytes its own tiles are showing and nothing else. +For continuous video, set **Live stream** to a browser-compatible camera URL. +The browser must be able to reach that URL; enrollment does not expose the +camera itself. See [Media payloads](../reference/payload-types.md#image-audio-video). ## Player tiles -A player is the one tile that both reads and publishes, so it has two bindings. -It **shows** a `record` describing what is playing and **publishes to** a `str` -carrying what to do about it: +A Player reads a status `record` and publishes commands to a separate `str` +message. -| Field of the record | Means | +| Status field | Meaning | |---|---| -| `title`, `artist`, `album` | what is playing | -| `status` | `play`, `pause`, `stop`, `load`, or `off` for a streamer with no power | -| `position`, `duration` | seconds, both | +| `title`, `artist`, `album` | track details | +| `status` | `play`, `pause`, `stop`, `load` or `off` | +| `position`, `duration` | seconds | -The buttons and the bar publish words: `toggle`, `next`, `prev` and -`seek:`. Those are a streamer's own vocabulary rather than this app's, -which is what lets one tile drive whatever is on the other end: the node that -receives them decides what they mean for its device. - -The position counts forward in the browser between readings, so the bar moves -at one second while the device is polled at whatever rate suits it. Every -reading that arrives is taken as the truth and the count restarts from it. +Controls send `toggle`, `next`, `prev` and `seek:`. Your receiving +node translates those into device actions. The position advances in the +browser between readings and resets to each fresh reading. ## Website tiles -A website tile frames whatever page its **Address** points at: a weather map, a -camera's own web interface, a timetable. A site that refuses to be framed shows -its own refusal in the tile. +Set **Address** to a page to embed, or choose **Image** for a snapshot or MJPEG +URL. A website may refuse embedding. -The page runs sandboxed. By default it gets an origin of its own that matches -no site, so it can run scripts but has no cookies or storage, and many -interactive pages will not load like that. **Trusted site** lets it keep its -real origin, with its cookies and storage, which is what visiting the site -would give it. Turn it on only for a site you trust. A trusted tile pointing at -this app's own address, or the hub's when the app is reached through one, shows -a notice instead: on that origin the page could reach into the app and read -your session. - -**Image** draws the address as a picture scaled to fit the tile, rather than -as a page. Use it for a camera's MJPEG stream (`http:///stream`) or a -snapshot URL, which a page frame would show at the camera's native size. +Embedded pages run in a sandbox without cookies or storage by default. +**Trusted site** permits the site's own origin, cookies and storage. +Use it only for a trusted destination. The app blocks trusted embeds of its +own origin, including the portal origin, to protect your session. ## Dashboard settings -Most of what a dashboard carries is a widget: a tile bound to a message. A -handful of things are not, because they belong to the whole surface rather than -to any tile on it, and a screen bolted to a wall has nobody standing at it to -set them. +Open **Dashboard settings** from the dock. Each setting can hold a fixed value +or follow a message chosen in **Driven by**. The fixed value is the fallback +when no usable message value is available. -| Setting | Is | Driven by | +| Setting | Choices | Message type | |---|---|---| -| **Look** | `Fluksio`, `Material` or `Glass` | a `str` message | -| **Theme** | `System`, `Light` or `Dark` | a `str` message | -| **Palette** | the dashboard's colours, in order | a `list` message | -| **Background** | the URL of an image | a `str` message | -| **Lock** | read-only on or off | a `bool` message | +| **Look** | `fluksio`, `material`, `glass` | `str` | +| **Theme** | `system`, `light`, `dark` | `str` | +| **Palette** | ordered hex colours | `list` | +| **Background** | image URL | `str` | +| **Lock** | read-only on/off | `bool` | -They all work the same way, and both halves are optional: +To change one on a schedule, have a scheduled node publish its bound message. -- **Just a value.** Set Theme to `Dark` and that dashboard is dark wherever it - is shown, whatever the device or the browser prefers. This costs no flow at - all, and it is what most wall panels want. -- **Driven by a message.** Pick one in **Driven by** and a flow takes the - setting over, exactly as it drives a tile. The value you set stays the - fallback: what the dashboard uses before the first message arrives, and - whenever the message is silent. +### Look and theme -A bound setting is type-checked like a widget binding, so a Theme pointed at a -`float` is refused by the editor and by the server. The panel's credential -is extended to it, so a paired screen may read its own theme message and -nothing further. +**Fluksio** is the default look. **Material** uses flat tonal cards; **Glass** +uses translucent tiles. Widgets work the same in all three. -There is no schedule field. **A schedule is a node publishing to the bound -message**: an `inject` with a cron expression feeding a `change` node that maps -the hour onto a palette is the whole of "warmer after sunset". The same channel -drives anything else, including locking a panel down remotely. - -### Look - -**Fluksio** is the app's own design (`--card` surfaces separated by a hairline -and a low shadow, pill controls, one slate-blue accent) and is what a dashboard -wears until you choose otherwise. **Material** lays flat, tonal cards on a plain -ground. **Glass** floats translucent tiles over a soft, slowly moving one. - -They are three complete sets of components, not three stylesheets, but they are -the same dashboard: every widget, every control and every keystroke behaves -identically, so switching look never changes what a panel can do. +Theme follows the chosen preference unless a palette sets the background +colour. ### Palette -A palette is an ordered list of colours, and **position is the role**: +Paste a colour list or a [Coolors](https://coolors.co) palette link. Colours +have roles by position: -| # | Role | +| Position | Role | |---|---| -| 1 | the ground the dashboard sits on | -| 2 | the surface a widget is | -| 3 | the primary — fills, active controls, the first chart line | -| 4 | the accent | +| 1 | background | +| 2 | widget surface | +| 3 | primary controls and first chart line | +| 4 | accent | | 5 | text | -| 6+ | the next chart lines, before the app's own chart colours | +| 6+ | additional chart lines | -Paste a [coolors.co](https://coolors.co) link (or a list of hex colours) and the -whole dashboard is recoloured: every widget, the rail and the charts. Leave the -later roles off and they are worked out from the ones you gave, so **three -colours are a whole dashboard**. **Rotate** turns the list when the roles landed -in the wrong order. Text that could not be read on a surface is replaced with -black or white there, so no palette can produce a line nobody can see. +Missing roles are derived, so three colours are enough to get started. +**Rotate** shifts their order. Text contrast is corrected automatically; +failure and success indicators keep their status colours. -While a palette is set, **Theme is idle**: the first colour is the ground, so -whether the panel reads light or dark is already decided by the palette itself. -Fault and success keep their own colours in every palette, so a failure is never -paintable as a reading. +### Background and lock -### Background - -An image drawn under the widgets, covering the canvas. It replaces the ground -the Glass look brings with it. Bound to a message, a flow decides the picture. - -### Lock - -Lock is a read-only *surface*, not a permission. The controls stay visible, -stop publishing and read as disabled, and the panel says **Read-only** in the -corner. What a paired screen is allowed to reach is still decided by its own -credential, below. +Background draws an image beneath the widgets. Lock disables publishing from +controls and shows **Read-only** when viewing the dashboard. It does not +restrict API permissions and does not disable controls in edit mode. ## Showing one -- `/view/{name}` — a browser tab pointed at one dashboard. Needs an ordinary - session. -- **Panels** — a named device that pairs instead of logging in. Below. +Open `/view/{name}` for a dashboard-only view with your normal session. +For a dedicated screen, pair a panel. ## Panels: hanging a screen on a wall -A wall tablet has no keyboard, so it pairs. +As a superuser: -1. **Dashboards → Panels**, add a panel named after where it hangs, and tick - the dashboards it shows. More than one and the screen draws a rail to switch - between them, with each dashboard's own icon on it. On a phone the rail is a - row floating along the bottom; the app shows it there too, above its own - buttons, for a dashboard that hangs on such a panel. -2. Tick **Touch friendly** if the screen is touched rather than pointed at. - Controls and the rail grow to a finger's size — the rail gets taller without - taking a wider column, so the arrangement does not move. It sits here rather - than on a dashboard because it describes the screen: the same dashboard may - also be open in a browser with a mouse. A phone gets it anyway, from its own - width. -3. Point the device's browser at the link the dialog shows. The device then - displays a six-character code. -4. Type that code into the same panel's **Pair device** field. The line under - it names what is holding the code. Check it is the screen you just hung, - because approving adopts whatever answered. The screen picks the credential - up within a few seconds and never asks again. +1. Open **Dashboards → Panels**, add a panel and select its dashboards. +2. Enable **Touch friendly** for larger controls. +3. Open the displayed pairing link on the device. It shows a six-character code. +4. Enter that code for the panel, check the device description, then press + **Pair device**. -What the screen holds is not a login. It reaches that panel's published -dashboards and the messages its own widgets read or publish, and nothing else. -A message no tile on it draws is refused in both directions. +The screen collects its credential and shows the assigned published dashboards. +Several dashboards get a navigation rail. -The *credential* cannot be made strictly read-only, and that is honest rather -than an oversight: a querying chart publishes its request, and a control on a -panel is the reason you put one there. The dashboard's own **Lock** setting -above stops its controls publishing and can be driven by a flow, which is how -you quieten a screen remotely, but that is the surface behaving, not the -credential being narrowed. +A panel credential permits its assigned dashboards, their bound messages and +the artifacts they show. It also permits publishing from its controls and query +charts. **Lock** changes the viewing surface; it does not narrow the credential. -Deleting the panel revokes the credential and the assignment together, which is -how you retire a device and what it showed. Unpairing revokes only the -credential: the panel, its dashboards and their arrangement stay exactly where -they are, and the screen falls back to asking for a new code, which is how you -swap the device out without rebuilding what hangs there. +**Unpair screen** revokes a locally paired credential and keeps the panel's +dashboard assignments. Deleting the panel removes the assignment and its +access. -!!! note "If the link is wrong" +!!! note "If the local link is wrong" - The pairing link is built from the instance's `FRONTEND_HOST`. If that - is not the address devices on your network actually reach, fix the setting - rather than the link: it is the same one password-reset mails and the OAuth - metadata are built from. + Pairing links use `FRONTEND_HOST`. Set it to the app address reachable + from the screen's network. ### A screen somewhere you cannot reach -Another building, someone else's network, no route in. An instance -[enrolled with a portal](portal.md) shows a second link, -`https://hub.${DOMAIN}/i/{instance-id}/panel`, and the same three steps -work through it: the portal serves that one page without a session, forwards -the pairing calls down the tunnel, and mints the credential when you approve -the code. The pairing line then reads *via portal*. +An [enrolled instance](portal.md) also shows a portal pairing link. Use that +link on the remote screen, then approve its code in the same dialog. The +device description says *via portal*. -The portal names the panel and nothing else. What the panel may read is decided -on the instance, on every call, by the same check a locally paired screen -passes. Two differences: it acts as the account the instance was enrolled -with rather than as whoever approved it, and deleting the panel stops it here -immediately while the portal's copy of the token expires on its own, which is -also why unpairing, which works on a credential this instance signed, does -not reach a remote screen. Revoke that one at the hub. +The instance enforces the same dashboard and message limits. Remote +credentials are issued by the portal, so **Unpair screen** does not revoke +them; revoke them at the portal. Deleting the panel removes access on the +instance. ## See also -- [Payload types](../reference/payload-types.md) — what a widget can bind to -- [Flows, nodes and messages](../concepts/flows.md) — where the names come from -- [Accounts and the portal](portal.md) — reaching all of this from outside +- [Payload types](../reference/payload-types.md) +- [Flows, nodes and messages](../concepts/flows.md) +- [Accounts and the portal](portal.md) diff --git a/docs/interface/flow-editor.md b/docs/interface/flow-editor.md index 2a6c178..5ab659e 100644 --- a/docs/interface/flow-editor.md +++ b/docs/interface/flow-editor.md @@ -1,177 +1,149 @@ # The flow editor -The canvas is where most of the work happens. It floats its chrome over a -full-bleed graph, so what you are looking at is always the flow. +Build a flow by naming its messages and configuring its nodes. Fluksio draws +the connections and arranges the canvas for you. ## The canvas -Nodes are laid out for you, left to right on a desktop and top to bottom on a -phone, and the arrangement is recomputed as the graph changes. There is -nothing to drag and nothing to tidy. +The graph runs left to right on desktop and top to bottom on phones. -The edges come from message names, so there is no layout to store or to merge -when two people edit the same flow. +- **Nodes** show their execution status. +- **Edges** pulse when values pass through. Click one to inspect its payload. +- **Endpoints** show flow inputs, dashboard widgets and nodes in other flows + that exchange messages with this flow. -What is drawn: - -- **Your nodes**, with a status dot — running, succeeded, failed, or nothing at - all when idle. Status is always named in words too, never colour alone. -- **Edges**, which pulse when a value goes down them. Click one to inspect it. -- **Endpoints** — anything wired into this flow that is not a node in it: a - dashboard control writing a message, a tile reading one, a node in another - flow on the far side of a dotted name. Drawn as labelled boxes at the - boundary, so a value never appears to come from nowhere. - -Values are live. The editor holds a websocket to the engine, and the connection -banner tells you when it is not. +Values update over a websocket. A connection banner tells you when the editor +cannot reach the engine. ## Adding a node -**Add node** on the dock, or ⌘P / Ctrl-P for the command palette, which also -jumps between flows and offers your shared nodes. Pick a type and it appears on -the canvas with its panel open. +Choose **Add node** in the bottom dock, or use ⌘P / Ctrl-P. The command palette +also jumps between flows and offers reusable nodes. Select a type to add it +and open its settings. ## The node panel -Floats on the right; a full-screen sheet on a phone. +**Consumes** and **Provides** declare the node's ports. Each has a port name, +payload type, bound message and delivery interval. Outputs also have a +streaming toggle. -**Consumes** and **Provides** are the ports. Each port has two lines: its name -and payload type (a `list` also says what its items are), then the message it -is bound to, how often at most it is delivered, and, on an output, a -*streaming* toggle for ports that publish repeatedly during one execution. The -message field suggests messages already in the flow, which is usually how you -wire something: type the name that already exists. +Message fields suggest existing names. Matching an output's message is how +you connect an input. -A port is named after its message until you name it yourself. Its name field -then stays empty and shows the name it follows. A name typed there stays when -the message changes, which is how a port is bound to a message called -something else, as a [`wren`](../reference/node-types.md#wren) target is; -clear it to follow the message again. The name is what the node's code sees, -so it is letters, digits and `_`, not starting with a digit, and no two inputs -or two outputs of a node share one. The field keeps the name it had otherwise. +A blank port-name field follows the message's last segment. Set a port name +to keep the argument name fixed while changing its message binding. Names use +letters, digits and `_`, cannot start with a digit, and must be unique among +the node's inputs or outputs. -Renaming an output renames the message everywhere it is read. Inputs are not -renamed, because an input is more often re-pointed at a different message than -renamed. +Renaming an output's message updates its consumers in the flow. Changing an +input binding points that input at a different message. -**Settings** is generated from the node type's parameter schema, so an MQTT -node shows broker, topic and QoS, and an ntfy node shows server, topic and -priority. A field marked as a credential renders as a picker over your stored -[secrets](operations.md) instead of a text box. A setting holding JSON, such as -a Switch node's rules, shows one line of it; click it to edit the whole value -in a larger editor, which saves only once the text parses. Escape or Cancel -leaves the setting as it was. An inject's payload and a flow input's starting -value of type `json`, `record` or `list` open the same editor, which also holds -the shape (a `record` is an object, a `list` an array); emptying it leaves no -value. +### Settings -A Function node gets a second settings section where you add your own: a -setting is a constant of this node's code and arrives as an argument beside its +Built-in nodes show forms for their configuration. Credential fields offer +stored [secrets](operations.md#secrets). JSON settings open a larger editor +and save only when the value parses. + +Function nodes also let you add settings passed as arguments alongside the ports. See [Where a node's values come from](../concepts/values.md). -**Timeout** (Function nodes) is how many seconds the code may be *silent* -before it is stopped: it measures silence, not duration, so a node that yields -every few seconds can run for hours under a timeout of 300. Left empty there is -no limit, which is the default. Set one where a node going quiet means it is -stuck rather than working. A `0` says so explicitly. +**Timeout** on a Function node limits silence, not total duration. A node +yielding regularly can run longer than its timeout. Empty or `0` means no +limit, the default. -**Code** is the editor. It saves as you type and applies on ⌘S; the node -reloads without the flow stopping. A node file defines `process(...)`; if it -defines exactly one public function under another name, that one is used -instead. +**Resources** selects a shared worker pool, a named flavor or custom CPU, +memory and GPU requirements. Placement by `device` and `device_policy` is +configured through the API or flow JSON; see [Remote workers](../code/workers.md). -Code runs as the engine, so changing it is a superuser's. Anyone else sees the -editor read-only, a Command node's settings disabled, and no Function, Command -or reusable nodes to add. +### Code and reuse -**Shared** turns a node's code into a library entry other flows can reuse. -Editing a shared node edits the copy every flow using it runs, which is the -point and also the caution. +Define `process(...)` in **Code**. If the file instead contains exactly one +public function, Fluksio uses that function. -!!! note "Not in the panel yet" +Code saves as you type. ⌘S / Ctrl-S in the code editor saves immediately; +**Publish** applies ordinary node code to the running flow. Generated code +from `fluksio sync` is read-only: edit its source repository and sync again. - `device` and `device_policy`, which machine a node's code runs on, are set - through the API rather than the panel, with `PUT /flows/{name}`. See - [Remote workers](../code/workers.md). +Only superusers can add or edit code-running nodes. Other users can read code +and configure permitted built-in nodes. + +**Reuse** shares node code as a library entry. Saving shared code takes effect +immediately in every flow using it, so check its users before changing it. ## The flow panel -Click the canvas title to open the flow's own settings. +Open **Flow settings** from the pencil button in the dock. -- **Running** — the switch that starts and stops the flow. Stopped means none - of its subscriptions, schedules or webhooks exist. -- **Mode** — `live` or `batch`. Live reacts to what arrives; batch only runs - when a run asks it to. See [Runs](../concepts/runs.md). -- **Inputs** — messages the flow takes from outside, with the value each starts - from. For a batch flow these are also the run's parameters. -- **Outputs** (batch only) — which messages a run reports as its result. -- **Contents** — a reminder that this flow's name namespaces every message - inside it. +| Setting | Purpose | +|---|---| +| **Running** | starts or stops the published live flow's subscriptions, schedules and webhooks | +| **Mode** | chooses continuous `live` operation or requested `batch` runs | +| **Inputs** | declares external messages and starting values; batch inputs become run parameters | +| **Result** | selects batch output messages | +| **Contents** | shows the node count and message namespace | + +For live inputs, **Publish this value now** sends the starting value immediately. +Changing a stored starting value alone does not replace an existing message. ## Inspecting an edge -Click a wire. You get the last payload that went down it, when, and a sparkline -of its recent numeric history. You can also republish that value from the node -that produced it, which re-runs everything downstream. It is the fastest way to -test a change without waiting for the real sensor. +Click an edge to see its latest payload, timestamp and recent numeric history. +Republish its value from the producer to run downstream nodes without waiting +for the next sensor reading. ## The dock -Along the bottom, floating over the canvas. - -| Control | What it does | +| Control | Action | |---|---| -| **Add node** | opens the palette | -| **Fit** | returns to the view the flow opened with | -| **Run** | injects a value by hand; on a batch flow, opens the parameters dialog | -| **Pause / Resume** | holds messages instead of running them | -| **Step** | releases exactly one held message | -| **Publish** | puts the draft live | -| **Discard** | throws the draft away | -| **Issues** | what is stopping this flow from running, per node | -| **Logs** | what the nodes printed, filterable | +| **Add node** | opens the command palette | +| **Fit to screen** | fits the graph; desktop only | +| **Run** | executes the current live flow once; for batch flows, asks for run parameters | +| **Pause / Resume** | holds or releases pending work while values still arrive | +| **Step** | releases one held item while paused | +| **Flow settings** | opens the flow panel | +| **Publish** | puts the saved draft live | +| **Discard** | removes unpublished changes | +| **Issues** | lists problems by node | +| **Logs** | shows filterable node output | -Pause and step together are how you test a flow before it moves anything -physical: pause it, inject a value, step it through, watch the values appear on -the edges, then publish. +**Run** uses the current draft when one exists without publishing it. It +executes real node code, including device and network actions. To try a single +value in a live flow, publish it from a flow input or edge inspector. + +Pause and step are useful for inspecting held work. They do not make a draft +run a simulation. ## Shortcuts -| Chord | Action | +Use ⌘ on macOS and Ctrl elsewhere. + +| Shortcut | Action | |---|---| | ⌘P / Ctrl-P | command palette | -| ⌘S / Ctrl-S | publish the flow — or, with focus in the code editor, apply the code | -| ⌘Z / ⌘⇧Z | undo / redo (the flow; the code editor has its own) | -| ⌘C / ⌘V | copy and paste nodes, including between flows | +| ⌘S / Ctrl-S | publish the flow; inside the code editor, save code | +| ⌘Z / Ctrl-Z | undo a graph edit | +| ⌘⇧Z / Ctrl-Shift-Z | redo a graph edit | +| ⌘C / Ctrl-C | copy selected nodes | +| ⌘V / Ctrl-V | paste nodes, including into another flow | -## Drafts and publishing +Text fields and the code editor keep their own undo and clipboard behavior. -Saving writes a draft; the engine keeps running what was published. Publishing -promotes it and reloads. +## Drafts, conflicts and issues -If someone else saved the same flow while you were working, publishing answers -with a conflict rather than overwriting them. The Flows list offers **Publish -all changes** when several flows are sitting on drafts. +Edits are saved as drafts. The engine keeps running the published version until +you publish; shared code follows the exception above. **Discard** returns to +the published version. Delete a never-published flow to remove it. -## Issues +If another editor saved the same flow, a stale save or publish returns a +conflict. Reload and review their changes before retrying. -The canvas validates as you edit and marks the node each issue belongs to: - -- a port that needs a message nothing in reach provides -- a message that exists but has never held a value -- a dependency cycle -- a node reading a message it also writes, with nothing to start it from -- code that did not load -- a webhook with no shared secret (advisory — it does not stop the flow) -- a node that loaded but reports itself down, such as a connector that cannot - reach its device - -A flow with any of these except the last two does not run, and the health -summary on Home counts it. The last one clears on its own once the node reports -itself well again. +The canvas reports unconnected inputs, missing starting values, cycles and +code-load errors. These prevent the affected flow from running. Unauthenticated +webhooks are advisory; runtime node-health problems clear when the node +recovers. See [Flow issues](../concepts/flows.md#when-a-flow-needs-attention). ## See also -- [Flows, nodes and messages](../concepts/flows.md) — the model behind the canvas -- [Node types](../reference/node-types.md) — everything the palette offers -- [Writing node code](../code/nodes.md) — what goes in the editor +- [Flows, nodes and messages](../concepts/flows.md) +- [Node types](../reference/node-types.md) +- [Writing node code](../code/nodes.md) diff --git a/docs/interface/index.md b/docs/interface/index.md index 5c49d9c..7005ebd 100644 --- a/docs/interface/index.md +++ b/docs/interface/index.md @@ -1,128 +1,87 @@ # The dashboard app -The web interface is a single-page app served at `app.${DOMAIN}`, or, for an -instance reached through a portal, at `${DOMAIN}/i/{instance-id}`. -Either way it is the same application, and it is a client of the same REST API -you can script against. +Open your instance's app address, or select it at [fluksio.com](https://fluksio.com) +after enrollment. For the local Docker setup, use +[app.localhost](http://app.localhost) and the admin login printed by +`make init`. See [Getting started](../getting-started/index.md) for setup. -Sign in with the account the instance was created with. On a fresh -instance that account was printed once, on the first start. +## Finding your way around -## The shell +The sidebar opens as a sheet on phones. -A floating sidebar on the left, the selected screen filling the rest. On a -phone the sidebar collapses to a sheet. - -| Entry | What lives there | +| Entry | Use it to | |---|---| -| **Home** | the brain graph, health, and everything that recently happened | -| **Flows** | the list of flows, and the canvas for each | -| **Dashboards** | the widget canvases, and the panels that display them | -| **Runs** | starting a run with values you choose, and the log of what ran | -| **Workers** | the machines that nodes can be placed on | -| **Secrets** | credentials your nodes reference without holding | -| **Modules** | the Python packages your node code may import | -| **Alerts** | where failures get sent | -| **Admin** | users (superusers only) | -| **Search** | anything in this instance, by name | -| **Settings** | your account, appearance, remote access, and the assistant | +| **Home** | see the brain graph, health and recent activity | +| **Flows** | create flows and open their editors | +| **Dashboards** | build widget dashboards and pair wall panels | +| **Runs** | start batch runs and review their results | +| **Workers** | manage machines that execute nodes | +| **Secrets** | store credentials; superusers only | +| **Modules** | see or install packages used by node code | +| **Alerts** | choose where failures are reported | +| **Admin** | manage users and OAuth clients; superusers only | +| **Search** | find items by name | +| **Settings** | change your account, appearance and remote access | + +**Assistant** appears when a provider is configured. **Documentation** opens +the docs in another tab. ### Search -**Search** at the foot of the sidebar, or ⌘K / Ctrl-K from anywhere, opens a -panel that finds things by name as you type: flows with the nodes inside them -and the inputs and outputs they declare, dashboards and the widgets on them, -panels, secrets, modules, workers and alert channels. Picking a node opens its -flow with that node in focus, an input or output opens its flow's settings, -where it is declared, and a widget opens its dashboard. +Use **Search**, or ⌘K / Ctrl-K, to find flows, nodes, flow inputs and outputs, +dashboards, widgets, panels, secrets, modules, workers and alert channels. +Selecting a result opens its editor or settings. -It searches this instance. Reached through a portal, other instances -are behind **All instances** at the top of the sidebar. +Search covers the current instance. When connected through a portal, +**All instances** returns to the instance list. ## Home -The one screen you leave open. Four things share it. - ### The brain graph -Every flow drawn as a neuron, wired to the flows it exchanges messages with. -This is the brand mark made live, and it is also the fastest read on the -instance: a neuron pulses when its flow is running work, and its ring turns -terracotta when the flow cannot run as written. A neuron with a problem keeps -its label showing so you can see which one it is without hovering. +The graph combines nodes from published flows. Nodes talking to the same +broker topic, URL or other external target share a neuron; computation nodes +appear individually. Messages connect them, and activity makes them pulse. -Each flow also has a switch beside it in the list, which starts and stops it. - -A [`wren` node](../reference/node-types.md#wren) anywhere adds a chat button -at the graph's top right, in the header on a phone, which opens the chat on -its [Control tab](../code/agents.md#control). +A problem keeps the affected neuron's label visible. A +[`wren` node](../reference/node-types.md#wren) also adds a chat button for its +[Control tab](../code/agents.md#control). ### Flows and dashboards -Two columns under the graph, exactly as tall as each other, most recently -worked on first. The flows column is the list with the switches; the dashboards -column is a mosaic, each tile a schematic of that dashboard's layout: blocks -where its widgets sit, shaded by what kind of widget each one is. Neither -column grows past about six rows: past that it scrolls in place rather than -pushing the health block down the page. - -The tiles are a footprint, not a live view. They show you which dashboard is -which at a glance; the readings are on the dashboard itself. +The flow list includes switches to start and stop flows. Dashboard tiles show +a small sketch of each layout; open one for its live readings. ### Health -Always answers, degraded or not. The tiles cover: - -- **Flows** — total, running, paused, quarantined, and how many cannot run - because their graph does not validate -- **Nodes** — how many failed to load -- **Runs running** — batch runs in flight right now, and how many are - waiting. Only on an instance that has run something -- **Queue** — depth, and how old the oldest pending item is -- **Loop lag** — whether the engine's event loop is keeping up - -`status: degraded` comes with a list of named problems, in words. "3 flow(s) -cannot run: house, pv, hallway" is more useful than a red dot, so that is what -it says. +Health shows flow status, node load errors, queue depth and event-loop lag. +After you have used batch runs, it also shows running and queued runs. +A degraded instance lists the problems that need attention. ### Activity -Charts of executions and failures over the selected range (1h / 6h / 24h / 7d), -with the recent runs, recent failures, dead-lettered work and the audit trail -underneath. +Choose 1h, 6h, 24h or 7d to review executions and failures, recent runs, +dead-lettered work and the audit trail. Hover a chart to filter the lists to +that moment; click to hold the selection while you read. -The charts are scrubbable: hover a moment and the lists below filter to it, -click to hold it while you read. That turns "something went wrong around two -o'clock" into the actual rows. +## Managing flows and dashboards -## Flows +The lists show titles, unpublished changes and status. Use **New flow** or +**New dashboard** to create one. The Flows list offers **Publish all changes** +when drafts are waiting. -The list shows each flow's title, node count, whether it has unpublished -changes, and whether it is enabled, paused or quarantined. The toolbar searches, -creates, and offers **Publish all changes** when several flows have drafts. - -Opening one takes you to [the flow editor](flow-editor.md). - -### Deleting several at once - -Press and hold a card, or ctrl-click it, to pick it, then tap the rest. While -anything is picked, **New flow** in the toolbar becomes a trash button, and it -asks once before deleting the lot. Unpicking the last one puts the list back; -so does Escape. The Dashboards screen works the same way. - -## Everything else - -- [The flow editor](flow-editor.md) — the canvas, the code editor, running and - testing -- [Dashboards and panels](dashboards.md) — widgets, bindings, and hanging a - screen on a wall -- [Secrets, modules and alerts](operations.md) — the three screens that keep an - instance running -- [Accounts and the portal](portal.md) — reaching an instance from outside - its network +To delete several items, hold a card or Ctrl-click it, then select the rest. +The create button becomes a trash button. Deletion asks for confirmation; +Escape clears the selection. ## Appearance -Light and dark follow your system by default; **Settings → Appearance** -overrides it. Both themes are first-class, and the wall-panel view is designed -to be legible in dark from across a room. +Light and dark follow your system by default. Change this under +**Settings → Appearance**. Dashboards also have their own theme settings. + +## Next stops + +- [The flow editor](flow-editor.md) +- [Dashboards and panels](dashboards.md) +- [Secrets, modules and alerts](operations.md) +- [Accounts and the portal](portal.md) diff --git a/docs/interface/operations.md b/docs/interface/operations.md index 9ca5dbb..d332425 100644 --- a/docs/interface/operations.md +++ b/docs/interface/operations.md @@ -1,139 +1,106 @@ # Secrets, modules and alerts -Three screens that have nothing to do with each other except that an -instance you actually depend on needs all of them. +Store credentials, manage node dependencies and choose where failures go. +These screens help keep an instance humming. ## Secrets -A node never holds a credential. A parameter written as +Open **Secrets** as a superuser, then add a name and value. The list shows +names only; saved values are not displayed again. + +Reference a secret from a node setting: ```json {"$secret": "influx-token"} ``` -is replaced with the stored value when the node is built, and the editor -renders any field a node type marks as a credential as a picker over what you -have stored. +Fluksio resolves it when building the node. Credential fields in the editor +offer a secret picker. Missing names produce an error on the affected node. -**Secrets** is where the values live. Add a name and a value; the value is -never shown again, and the list only ever shows names. +Secrets are encrypted with a key derived from `SECRET_KEY` and stored outside +the flow repository. Using references keeps credentials out of saved flows. -They are encrypted at rest with a key derived from the instance's -`SECRET_KEY`, and kept **outside** the flow repository. That matters because -flows are a git repository you may well push somewhere: what gets committed and -shared never contains a password. +!!! warning "Keep your encryption key" -A name that no longer resolves is a clear error on the node that wanted it -("No secret named 'x'; add it under Secrets") rather than a mysterious -authentication failure. - -!!! warning "Rotating `SECRET_KEY`" - - The secrets store is encrypted with a key derived from it. Change - `SECRET_KEY` and the store no longer decrypts. Re-enter the secrets, or - plan the rotation properly. + Changing `SECRET_KEY` makes existing secrets unreadable. Keep the old key + for re-encryption, or re-enter the secrets. Source installations include + `backend/scripts/rotate_secret_key.py` for rotating the secrets store. ## Modules -Node code runs in a virtual environment of its own, on the instance's data -volume, separate from the packages Fluksio itself runs on. A pin of yours can -never shadow one of ours, and vice versa. +**Modules** shows the Python environment used by node code, its installed +packages and Python version. There are two setups: -**Modules** is a pip manifest and an **Apply** button, which only a superuser -sees: a package runs its own code on install. It is installed with -`uv pip sync`, so a line you delete is uninstalled rather than left behind. -The manifest is versioned alongside your flows, so what a deployment installed -is recorded with what uses it. +- **Managed environment:** Docker uses a separate node environment. A + superuser edits **Requirements** and presses **Apply**. Packages are synced + with `uv pip sync`; deleting a requirement removes it. The manifest is + versioned with the flows, and successful installs refresh the workers. +- **Adopted environment:** a fresh pip installation inside your own virtual + environment can use that environment for nodes too. Modules is read-only. + Install dependencies there with pip or uv, then sync or restart Fluksio. -The page shows what is currently installed, the interpreter's Python version, -and whether the manifest and the environment agree. A failed resolve comes back -as `uv`'s own output, in the pane, because that is the only thing anyone can -act on. +Set `NODE_VENV=managed` to request a separate environment. Installation +failures show the package manager's output on the page. -An install takes effect immediately: the worker processes pick up the new -interpreter state without the engine restarting. - -!!! note "Big scientific stacks" - - Installing torch into this venv works and is often the simplest thing. The - alternative, when you already have an environment you would rather not - duplicate, is to attach it as a worker: `fluksio-worker --python - /path/to/venv/bin/python`. See [Remote workers](../code/workers.md). +For an existing scientific environment on another machine, attach a worker +with `fluksio-worker --python /path/to/venv/bin/python`. See +[Remote workers](../code/workers.md). ## Alerts -Everything that goes wrong already travels the engine's event bus. Alerts is -where you say who hears about it. +Create a channel, choose events in a rule, and use **Test** to check delivery. +Enable alerts for the rules to send. ### Channels -| Kind | Settings | +| Kind | Destination | |---|---| -| **ntfy** | server, topic, and a token for a protected topic | -| **SMTP** | an address to send to (the instance's mail settings do the rest) | -| **Webhook** | a URL to POST to | -| **Dashboard** | a message name a notification widget reads | +| **ntfy** | server and topic; optional token | +| **SMTP** | recipient address, using the instance's mail settings | +| **Webhook** | URL that receives a POST | +| **Dashboard** | declared message read by a Notification widget | +| **webpush** | browsers subscribed from the Alerts page | -Any channel setting may hold a `{"$secret": "name"}` reference instead of a -literal, so a webhook URL with a token in it does not sit in the config. +Channel settings may use `{"$secret": "name"}` references. Browser push also +needs notification permission and a secure browser context. ### Rules -A rule is a set of events and the channels they go to. **A rule with no events -ticked covers all of them**, including ones added in later versions, which is -usually what you want for the first rule you write. +A rule selects events and destination channels. Leaving all events unchecked +matches every supported alert event, including future additions. -| Event | Fires when | +| Event | Trigger | |---|---| -| `node_error` | a node failed | -| `node_health` | a connection dropped | -| `flow_quarantined` | a flow crashed often enough that the engine stopped restarting it | -| `task_crashed` | a background task crashed | -| `engine_degraded` | the engine is struggling | -| `cascade_dropped` | work was given up on | -| `queue_unavailable` | the work queue is unreachable | +| `node_error` | a node fails | +| `node_health` | a connection goes down | +| `flow_quarantined` | repeated crashes stop a flow's background work | +| `task_crashed` | a background task crashes | +| `engine_degraded` | engine health degrades | +| `cascade_dropped` | the engine drops work | +| `queue_unavailable` | the work queue becomes unreachable | -**Test** sends one message through a channel so you find out it is misconfigured -now rather than at three in the morning. +Repeated faults are deduplicated with a 15-minute default cooldown. +Flapping connections are suppressed, and the engine caps alert dispatches at +ten per hour. A quiet channel may be cooling down; **Test** helps distinguish +that from a broken configuration. -### What it mostly does is not send +## Admin: users and clients -This is the part worth knowing before you decide it is broken: +Superusers use **Admin** to add, edit and remove accounts, and manage OAuth +clients for [agents](../code/agents.md). -- the same fault repeating is **one** alert, with a cooldown (15 minutes by - default per rule) -- a connection flapping is one story, not one alert per transition -- however bad it gets, **ten alerts an hour** is the ceiling - -An engine that is genuinely on fire produces a handful of messages, not -thirty-six thousand. - -## Admin: users - -Superusers get an **Admin** screen: add, edit and remove accounts, and see who -is a superuser. - -Accounts that arrived through a portal are badged *Portal*. They have no -password (they sign in at the portal, not here) and they are never -superusers, so they read node code but cannot change it. Deleting one cuts their access immediately and independently of the -portal, so it works even if the portal cannot be reached. See -[Accounts and the portal](portal.md). - -The same screen manages OAuth clients, which is how [agents](../code/agents.md) -are registered. +Accounts admitted from a portal are marked *Portal*. They sign in through the +portal, have no local password and are not superusers. To block local access, +edit the account and clear **Is active?**. See [Accounts and the portal](portal.md). ## Settings -Your own account: name, email, password, and appearance (light, dark, or -follow the system). +Change your name, email, password and appearance under **Settings**. -**Remote access** is where an instance is paired with a portal, remote -users are admitted, and the link is cut again. That has [its own -page](portal.md). +**Remote access** manages enrollment and admitted users; see +[Accounts and the portal](portal.md). -**Assistant** turns on the chat that builds and edits flows: pick a provider, -paste a key, fetch the model list and choose one. The key is stored encrypted -on the instance and never shown again. One provider serves everyone who signs -in here, and until one is set nothing in the interface mentions the feature. -How the chat works, and what it may do, is on the [agents -page](../code/agents.md#the-assistant). +A superuser can configure **Assistant**: choose a provider, store its key, +fetch models and select one. The encrypted key is shared for assistant access +on this instance. The chat appears after configuration; see +[The assistant](../code/agents.md#the-assistant). diff --git a/docs/interface/portal.md b/docs/interface/portal.md index 251ebea..e407f2a 100644 --- a/docs/interface/portal.md +++ b/docs/interface/portal.md @@ -1,142 +1,86 @@ # Accounts and the portal -Fluksio is self-hosted by default. An instance runs offline, keeps its data -on its own disk, and never contacts anything unless you tell it to. +The optional portal lets you open an instance from [fluksio.com](https://fluksio.com), +including a pip installation with no local dashboard. The instance connects +outward over a WebSocket, so you do not need to open an inbound port. -The **portal** is optional, and it exists to solve two specific problems: +Flows still run on your instance. The portal serves the interface and carries +API requests, responses and live updates through the connection. -1. **Your machine has no inbound route.** A homelab behind CGNAT, a cluster - node with no open ports, a laptop. Opening one is work, and often not - allowed. -2. **You want a browser on it anyway.** A `pip install fluksio` instance - has no web server for the dashboard at all. +## Connect an instance -An enrolled instance dials *out* to the portal and holds one websocket -open. The portal serves the dashboard from its own side, and only the API calls -travel down the tunnel, so the interface loads at portal speed and your -machine stays unreachable from the internet. +1. Sign up or sign in at **fluksio.com**. +2. Open **Instances → Add instance** and copy the code. +3. On the machine running Fluksio, open another terminal in the same directory + as `fluksio serve` and run: -## Enrolling + ```sh + fluksio enroll + ``` -Two halves: whoever performs the second step decides what the instance's owner -gets. +The CLI uses `https://hub.fluksio.com` by default. If you started the engine +with `--data-dir` or `--global`, use the same option when enrolling. A running +engine picks up the connection automatically; otherwise start it with +`fluksio serve`. -**On the portal** (`hub.${DOMAIN}`, or [fluksio.com](https://fluksio.com) for -the hosted one): **Instances → Add instance**, give it a name, and copy -the code. +From a local dashboard, you can instead use **Settings → Remote access**: +leave the default portal URL, paste the code, and press **Connect**. +Enrollment requires a superuser account. -**On the instance**, either from the dashboard: - -> **Settings → Remote access**, enter the portal URL and the code, press -> **Connect**. - -or from the command line, which is the path for an instance with no web -interface of its own: +The portal owner's sessions use the local account that enrolled the instance. +Check **Acting as** under Remote access. With several local superusers, the +CLI can choose one explicitly: ```sh -fluksio enroll ABCD-1234 --portal https://hub.fluksio.com +fluksio enroll --as someone@example.com ``` -The code is single-use and expires in fifteen minutes. A fresh code on an -instance that is already paired replaces the pairing, so moving an instance to -another portal is one command rather than a disconnect and a reconnect. +A new valid code replaces an existing connection. An expired or invalid code +leaves the current connection intact. Once connected, open the instance from +the portal's **Instances** list. -A portal session then arrives as *that local account*, which the settings screen -states plainly. Use `--as someone@example.com` to enrol as a specific local -account when the instance has several superusers. +## Let someone else in -Once enrolled, the instance appears under **Instances** with its -status, when it was last seen and its version. **Open** takes you to its -dashboard at `${DOMAIN}/i/{instance-id}`. +Each person gets an ordinary local account, with no superuser rights. You can +admit them in either of these ways: -**Upgrade a self-hosted portal before its instances.** An instance dials the -tunnel with its credential in an `Authorization` header, where it stays out of -access logs, and a portal from before that reads it only from the URL. The -portal still takes the old `?token=` for one release, so older instances keep -attaching meanwhile. +- Share an invitation link from the portal's **Instances** list. +- Ask them for a code from **Instances → Join an instance**. Enter it on your + local instance under **Settings → Remote access → Add remote user**. -## What the portal can and cannot do +Their account appears under **Admin → Users** with a *Portal* badge. They +sign in through the portal rather than with a local password. Invitations are +time-limited; generate a fresh one if it expires. -The portal holds one credential for your instance and proxies requests down -the tunnel. What those requests may do is decided **on the instance**, by -the same checks a local session passes. +## Manage or end remote access -The trust anchor is a signing keypair on the portal. Every instance pins -its public half at enrolment and rejects anything else, which is what stops a -hijacked DNS entry or a mis-issued certificate from impersonating the portal. +**Settings → Remote access** shows the connection status and the latest +connection error. The instance needs outbound access to the portal, and any +proxy on that route must support WebSockets. -## Letting someone else in +- To remove a person's access, remove them from the portal's sharing dialog + or edit their account under **Admin → Users** and clear **Is active?**. +- To end all portal access from the instance, press **Disconnect**. This stops + the connection and invalidates portal credentials locally, without requiring + the portal to be online. +- To replace the pairing, use **Connect with a new code**. Remote users and + panels may need to pair again. -Anyone else on the portal reaches your instance only if you let them, and they -arrive as a local user of their own rather than as you. There are two ways, -and the difference is who starts. +Local sign-in and flows continue after disconnecting. Portal requests follow +the instance's normal account permissions, and the instance verifies portal +tokens against the public signing key pinned during enrollment. -### Share a link +## Custom portals -On the portal, the share icon on your instance's row gives a QR code and a URL. -Whoever opens it, signed in to Fluksio on that device, joins. It works once and -expires in fifteen minutes; **New link** replaces it, which is also how a link -you would rather not have sent is cancelled. +The CLI accepts another compatible portal with `--portal https://`; +the dashboard has the same URL field. HTTPS is required except for a portal on +loopback. -Nothing has to be entered on the instance. It learns of the new member when -they first arrive and asks the portal who they are, then gives them their local -account. - -### Take a code - -For a machine whose owner would rather admit people at the machine itself. - -1. **They**: **Instances → Join an instance**, and copy the code. It is - bound to their portal account and expires in fifteen minutes. -2. **You**, on the instance: **Settings → Remote access → Add remote - user**, and enter the code. - -Either way they now see the instance under **Instances**, marked *Shared by*, -with **Open** and nothing else. Renaming, re-keying and removing stay with you. - -The instance redeems the code against the portal using its own credential. -A portal session cannot do this, which is what stops somebody you let in from -letting others in. - -On the instance they appear under **Admin → Users**, badged *Portal*, never -a superuser and with no password. - -## Cutting it off - -| From | Action | Effect | -|---|---|---| -| The portal | share dialog → **Remove** | that member stops reaching it from the portal; their user on the machine stays until it is deleted there | -| The portal | **Leave**, on a row shared with you | gives up your own access to somebody else's instance | -| The portal | **New code** | rotates the credential and drops the current link | -| The portal | **Remove** | deletes the registration and cuts the connection | -| The instance | **Disconnect** | unilateral and immediate — the portal's tokens stop verifying here whatever the portal still has on file | -| The instance | delete a user under **Admin → Users** | that one person, immediately, independent of the portal | - -**New code** also cuts every credential the portal minted for this instance, -including [wall panels paired through it](dashboards.md#a-screen-somewhere-you-cannot-reach). - -The instance's own **Disconnect** is the one to reach for if you are ever -unsure: it does not need the portal's cooperation. - -## Running your own portal - -The portal is the `index` stack's `hub` service: accounts, the registry of -connected instances, and the websocket each one dials in on. Two things -about it are load-bearing: - -- **It runs a single process.** It keeps its attached instances in the - memory of the process holding their sockets, so a second worker would answer - for links it does not hold. Scaling out needs a routing layer first. -- **Back up the signing keypair with the database.** Replacing it forces every - instance to be enrolled again. - -Websocket upgrades must be enabled on the `hub` hostname in whatever proxy -fronts it. Without them every instance sits in a reconnect loop and the -portal shows them all offline. +This public repository contains the instance and its connector. The portal +service is not included, so the Docker stack does not start one. ## See also -- [The `fluksio` command](../code/cli.md) — `fluksio enroll`, and what it writes -- [Dashboards and panels](dashboards.md) — pairing a wall screen through the portal -- [Getting started: data science](../getting-started/data-science.md) — the - shortest path from a pip install to a browser +- [The fluksio command](../code/cli.md): data directories and enrollment options. +- [Dashboards and panels](dashboards.md): pair a wall screen through the portal. +- [Getting started](../getting-started/index.md): install and open your first instance. diff --git a/docs/llms.txt b/docs/llms.txt index 8206d65..1a28856 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -1,24 +1,25 @@ # Fluksio -> Fluksio is a node-based automation engine. A flow is a graph of small -> functions wired by message name, kept running by one resident engine. The -> same engine serves two shapes of work: live flows that never end (buildings, -> labs, homelabs) and batch runs that finish and leave a record (experiments, -> ML pipelines, CI-style jobs). Self-hosted by default; an optional portal -> exists only to reach an instance that has no inbound route. +> Fluksio is a node-based automation engine for live flows and batch runs. +> Nodes connect through typed, named messages. Start with pip for data science +> or Docker for facility automation; connect to a portal for optional remote +> access. Data stays on the instance. ## Getting started -- [Pick your starting point](/getting-started/): the two setup paths and how to choose -- [Data science](/getting-started/data-science/): pip install, wrap a training script, runs and sweeps -- [Facility automation](/getting-started/facility-automation/): Docker stack, MQTT to dashboard, InfluxDB, workers + +- [Pick your starting point](/getting-started/): choose data science or facility automation +- [Data science](/getting-started/data-science/): pip setup, a Python function, and a first run +- [Facility automation](/getting-started/facility-automation/): Docker setup, a live flow, dashboards, and devices ## Concepts + - [Flows, nodes and messages](/concepts/flows/): the graph comes from message names, not drawn wires - [Where a node's values come from](/concepts/values/): ports, settings, flow inputs - [Keeping state in a flow](/concepts/state/): a message a node both reads and writes - [Runs: pipelines that finish](/concepts/runs/): batch mode, metrics, artifacts, sweeps, durability ## The interface + - [The dashboard app](/interface/): the shell, Home, health and activity - [The flow editor](/interface/flow-editor/): canvas, node panel, dock, shortcuts, drafts - [Dashboards and panels](/interface/dashboards/): widgets, typed bindings, wall screens @@ -26,6 +27,7 @@ - [Accounts and the portal](/interface/portal/): enrolment, remote users, revoking ## Code and the CLI + - [The fluksio command](/code/cli/): serve, enroll, worker, and the data directory - [Writing node code](/code/nodes/): process(), generators, fluksio.emit, artifacts - [The HTTP API](/code/api/): auth, flows, messages, runs, observability, websocket @@ -34,6 +36,7 @@ - [Writing a connector](/code/connectors/): packaging a device integration as a node type ## Reference + - [Node types](/reference/node-types/): every built-in type and its settings - [Payload types](/reference/payload-types/): float, int, str, bool, json, record, list, series, artifact - [Configuration](/reference/configuration/): every environment variable diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 5ac5aa6..3e37778 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -1,238 +1,187 @@ # Configuration -Every setting comes from the environment, or from an env file. Which file -depends on how the instance was started: +Most installations can keep the defaults. Settings come from environment +variables and an env file: -| Started with | Reads | +| How you start Fluksio | Env file | |---|---| -| `fluksio serve` | `env` inside the data directory (`$FLUKSIO_ENV_FILE`) | -| the Docker stack | `.env` beside `docker/` | +| `fluksio serve` | `/env`, unless `FLUKSIO_ENV_FILE` names another file | +| Docker with the Makefile | `.env` in this repository's root | -Anything already exported wins over the file. +Exported variables override the file. `fluksio serve` flags override both. +For CLI installations, select storage with `--data-dir`, `FLUKSIO_HOME` or +`--global`; see [The fluksio command](../code/cli.md). ## Storage -| Variable | Default | Notes | +| Variable | Default | Purpose | |---|---|---| -| `DATA_DIR` | `flow-data` (`./.fluksio` via the CLI; `~/.fluksio` with `--global`) | everything below derives from this | -| `DATABASE_URL` | SQLite in `DATA_DIR` | any SQLAlchemy URL | -| `FLOWS_DIR` | `$DATA_DIR/flows` | the git repository holding flows | -| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials, kept outside the repo | -| `ALERTS_FILE` | `$DATA_DIR/alerts.json` | channels and rules | -| `PROVISIONERS_FILE` | `$DATA_DIR/provisioners.json` | clusters a machine can be started from; absent means none | +| `DATA_DIR` | `flow-data`; CLI uses `./.fluksio` or `~/.fluksio` with `--global` | base directory for instance data | +| `DATABASE_URL` | SQLite at `$DATA_DIR/fluksio.db` | SQLite URL; other database engines are unsupported | +| `FLOWS_DIR` | `$DATA_DIR/flows` | flow git repository | +| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials | +| `ALERTS_FILE` | `$DATA_DIR/alerts.json` | alert channels and rules | +| `WEBPUSH_FILE` | `$DATA_DIR/webpush.json` | push keys and browser subscriptions | +| `PROVISIONERS_FILE` | `$DATA_DIR/provisioners.json` | optional worker provisioning configuration | | `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings | -| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens | -| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any | -| `NODE_VENV` | `auto` | which interpreter node code runs on — see below | +| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | agent and worker signing key | +| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | optional portal enrollment | +| `AI_CONFIG_FILE` | `$DATA_DIR/ai.json` | optional assistant provider configuration | +| `CHATS_DIR` | `$DATA_DIR/chats` | assistant transcripts | +| `NODE_VENV` | `auto` | interpreter for node code | -Set `DATA_DIR` and the rest follow. Set one explicitly and it wins, which is -what the container images do to pin everything onto `/data`. +Changing `DATA_DIR` moves the derived paths unless you set them individually. +Docker stores instance data under `/data` on the `app-flow-data` volume. -`NODE_VENV` is the exception, being about an environment rather than a path: - -| Value | What node code runs on | +| `NODE_VENV` value | Behavior | |---|---| -| `auto` (default) | the venv Fluksio was installed into, when it was installed into one and there is no venv of its own already built. `pip install fluksio` beside your own packages is this case, and the packages are then already there — the Modules screen turns read-only, because that environment is not Fluksio's to install into. | -| `managed` | a venv the engine builds under `DATA_DIR` and owns, which the Modules screen installs into with `uv pip sync`. The container images set this: the venv in them holds the app and nothing of anybody else's. | -| a path | that interpreter, or that venv, whatever it is. | +| `auto` | keeps an existing managed venv; otherwise uses the installation's venv when available, or creates a managed one | +| `managed` | creates a venv under `DATA_DIR`; the Modules screen manages its packages | +| interpreter or venv path | uses the specified Python environment | -An instance that already has a managed venv keeps it on upgrade under -`auto`, because it may hold packages somebody installed on purpose. +Docker uses `managed`. When Fluksio adopts your own venv, the Modules screen +is read-only; install packages into that environment yourself. -!!! warning "The four files that must be on persistent storage" +Back up the whole data directory and the persistent `SECRET_KEY`. In Docker, +keep `.env` and `app-flow-data`; back up `app-redis-data` too when preserving +live state. Compose prefixes volume names with the project name `fluksio-app`. +Keep runtime files on persistent storage so rebuilds preserve credentials, +pairings and subscriptions. - `secrets.enc`, `alerts.json`, `panels.json` and `oauth-key.pem` are written - at runtime. In a container, anything not on a volume lands in the writable - layer and is lost on the next rebuild, un-pairing every screen and - revoking every agent. +## Live state -## State - -| Variable | Default | Notes | +| Variable | Default | Purpose | |---|---|---| -| `REDIS_HOST` | unset | without it, flow state lives in memory and does not survive a restart | -| `REDIS_PORT` | `6379` | | +| `REDIS_HOST` | unset | unset uses memory; a Redis host enables persistent flow state | +| `REDIS_PORT` | `6379` | Redis port | -Flow state is the last value of every message, node memory, and the run queue. -Redis here is persistence, not a cache. Run it with append-only persistence on -(`--appendonly yes`, which the compose stack sets). +Live state includes message values, node memory and queued work. In-memory +state is lost on restart. The Docker stack uses Redis with append-only +persistence on its own `app-redis-data` volume. Its default sync policy may +lose about a second of recent writes after an unclean shutdown. -That writes with Redis's default `appendfsync everysec`, so an unclean stop of -Redis or of its host loses up to a second of queued work. `appendfsync always` -closes the window and cuts engine throughput to about a third, so the default -stands. +## Accounts and access -## Identity and access - -| Variable | Default | Notes | +| Variable | Default | Purpose | |---|---|---| -| `SECRET_KEY` | generated | signs sessions and derives the secrets-store key | -| `ACCESS_TOKEN_EXPIRE_MINUTES` | `11520` (8 days) | | -| `FIRST_SUPERUSER` | — | absent means the CLI creates one on first run | -| `FIRST_SUPERUSER_PASSWORD` | — | absent means one is generated and printed once | -| `DOMAIN` | `localhost` | what the API and OAuth issuer are built from | -| `FRONTEND_HOST` | `http://localhost:5173` | used in mails, OAuth metadata and panel pairing links | -| `BACKEND_CORS_ORIGINS` | `[]` | comma-separated; `FRONTEND_HOST` is always allowed | -| `FORWARDED_ALLOW_IPS` | `127.0.0.1` (`127.0.0.1,172.16.0.0/12` in the compose stack) | proxies whose `X-Forwarded-For` is believed, as uvicorn reads it; the sign-in, sign-up and OAuth limits count the rightmost hop none of them added | +| `SECRET_KEY` | generated | signs sessions and encrypts the secrets store; the CLI persists it and `make init` writes it to `.env` | +| `ACCESS_TOKEN_EXPIRE_MINUTES` | `11520` (8 days) | session-token lifetime | +| `FIRST_SUPERUSER` | unset | initial administrator email; the CLI creates an account if needed | +| `FIRST_SUPERUSER_PASSWORD` | unset | initial password; the CLI generates and prints one if needed | +| `DOMAIN` | `localhost` | API hostname and OAuth issuer; Docker also uses it for routing | +| `FRONTEND_HOST` | `http://localhost:5173` | interface URL for links and metadata | +| `BACKEND_CORS_ORIGINS` | `[]` | allowed origins, as comma-separated URLs or a JSON list; `FRONTEND_HOST` is also allowed | +| `FORWARDED_ALLOW_IPS` | `127.0.0.1` | trusted proxies; Docker adds `172.16.0.0/12` | +| `SIGNUP_ENABLED` | `false` | allows account creation from the login page; otherwise use **Admin → Users** | -## Where the interface looks for the API +!!! warning "Keep your secret key" -The dashboard is a static bundle, so this one is a **build** argument of the -`frontend` image rather than a setting the running stack reads. + Changing `SECRET_KEY` signs everyone out and makes existing encrypted + secrets unreadable. Back it up with your data. -| Argument | Used by | Effect | -|---|---|---| -| `VITE_API_URL` | `frontend` at build time | the address the interface calls | +## Interface API address -The compose stack takes it from `.env`, falling back to `https://api.${DOMAIN}`. -`scripts/setup.sh` writes it there with the scheme `ENVIRONMENT` implies, so a -local build calls `http://` and does not fail a certificate check nothing is -there to satisfy. +`VITE_API_URL` is a frontend **build argument**, so changing it requires +rebuilding the frontend image. -Empty is the useful value: the interface then addresses the API relative to -whichever origin served the page, so one image answers on a hostname, on a -`http://:`, and through an ssh tunnel alike, and no origin has -to be added to `BACKEND_CORS_ORIGINS`, because there is only one. -`docker/compose.lan.yml` builds it that way and puts an `/api` proxy in front -of the backend to complete it; see -[getting started](../getting-started/facility-automation.md#on-your-own-network-by-address). +| Setup | Value | +|---|---| +| Production Docker | set `.env` to `https://api.` | +| `make dev-local` | overlay sets `http://api.localhost` by default | +| `make dev-lan` | overlay sets an empty value, using `/api` on the page's own origin | -Set it to an absolute URL only when the API genuinely lives somewhere else, and -remember it is fixed at build time: changing it means rebuilding that image. +Without an explicit value, the production Compose file falls back to +`https://api.${DOMAIN}`. Set `FRONTEND_HOST` and `BACKEND_CORS_ORIGINS` to +match your deployment as well; `make init` generates secrets, not deployment +URLs. -!!! danger "Rotating `SECRET_KEY`" - - The secrets store is encrypted with a key derived from it. Change it and - the store stops decrypting, and every session is signed out. Re-enter your - secrets, or plan the rotation properly. +The LAN overlay also configures nginx to proxy `/api` and WebSocket requests +to the backend. An empty value needs that proxy when the frontend and backend +are served separately. See [facility automation](../getting-started/facility-automation.md#run-it-long-term) +for deployment settings. ## Environment -| Variable | Default | Notes | +| Variable | Default | Purpose | |---|---|---| | `ENVIRONMENT` | `local` | `local`, `staging` or `production` | -| `PRIVATE_API_ENABLED` | `false` | unauthenticated test-only endpoints; needs `ENVIRONMENT=local` too | -| `API_DOCS_ENABLED` | `false` | serves `/docs`, `/redoc` and the OpenAPI document to any caller; otherwise only to one on this machine. Never in `production` | -| `SIGNUP_ENABLED` | `false` | lets anyone who reaches the login page create an account; otherwise a superuser adds them under **Admin** | -| `TZ` | `UTC` | the timezone every schedule is written in | +| `PRIVATE_API_ENABLED` | `false` | test endpoints; also requires `ENVIRONMENT=local` | +| `API_DOCS_ENABLED` | `false` | opens `/docs`, `/redoc` and `/api/v1/openapi.json` to network callers outside production | +| `TZ` | host timezone; Docker defaults to `UTC` | timezone used by cron schedules | -`TZ` is the container's own, not a setting the code reads: an `inject` or a -`delay` with a cron expression fires on local time. Left at `UTC`, "off at -02:00" means two in the morning UTC, which in most of the world is neither two -o'clock nor the same hour in summer as in winter. Set it to where the -instance is. +API documentation is available to loopback callers outside production. +`API_DOCS_ENABLED=true` opens it to other callers, as the Docker development +stack does. Production disables it entirely. Staging and production also +refuse `changethis` values for the session key or administrator password. -`production` closes `/docs`, `/redoc` and the OpenAPI document, because the -schema enumerates every endpoint the instance serves, including the paths -webhook nodes mounted at runtime. Elsewhere they answer a caller on the -engine's own machine, so `fluksio serve --host 0.0.0.0` does not hand them to -the network; `API_DOCS_ENABLED` opens them to everyone, as the dev stack does. It also turns a `changethis` secret from a -warning into a refusal to start. +Set `TZ` in `.env` for schedules that should follow local time, such as +`Europe/Berlin`. -## The engine +## Engine and resources -| Variable | Default | Notes | +| Variable | Default | Purpose | |---|---|---| -| `FLOW_MAX_WORKERS` | `4` | node-code subprocesses run in parallel | -| `FLOW_MAX_CASCADES` | `4` | cascades in flight at once; throughput is this over the mean cascade time, so raise it where nodes wait on a network rather than a CPU | -| `FLOW_MAX_RUNS` | `4` | batch runs driven at once. A different limit from the one above: a run drives a whole graph, and its nodes are bounded by `FLOW_MAX_WORKERS`. This is what a sweep queues behind | -| `FLOW_NODE_TIMEOUT` | `0` | seconds a node may be silent, unless it sets its own; 0 is no limit | -| `FLOW_CPUS` | `0` | cores nodes that declare `resources` may be given; 0 works it out as every core but two, which are what keeps the engine answering while the machine is busy | -| `FLOW_GPUS` | `0` | GPUs on this machine, each held by one node at a time. 0 counts NVIDIA's device nodes; say a number for anything they miss | -| `FLOW_RAM` | `0` | megabytes nodes that declare `ram` may be given; 0 works it out as the machine's memory, or the container's limit where that is lower, less a gigabyte for the engine | -| `FLOW_WORKER_MAX_RSS_MB` | `0` | a warm worker holding more resident memory than this when its node returns is retired rather than kept; 0 keeps every worker warm | -| `OBS_RETENTION_DAYS` | `30` | how long metrics, events and run records are kept | -| `ARTIFACT_GC_INTERVAL_S` | `3600` | how often artifact bytes nothing refers to are swept away; 0 never sweeps | -| `ARTIFACT_GC_GRACE_S` | `3600` | how long a freshly written artifact is spared, whatever refers to it | -| `MAX_ARTIFACT_BYTES` | `2147483648` | the largest body `PUT /artifacts` will take; 0 removes the limit | -| `ARTIFACT_VOLATILE_DIR` | worked out | where frames a flow only shows live are held; empty picks a directory under `/dev/shm` named for the data directory, and falls back to the temporary directory | -| `ARTIFACT_VOLATILE_BYTES` | `50331648` | how much that ring holds before the oldest frames fall out; 0 turns it off and volatile saves land in the store | +| `FLOW_MAX_WORKERS` | `4` | local node-code subprocesses | +| `FLOW_MAX_CASCADES` | `4` | concurrent live message cascades | +| `FLOW_MAX_RUNS` | `4` | concurrent batch runs | +| `FLOW_NODE_TIMEOUT` | `0` | maximum node silence in seconds, unless the node overrides it; `0` disables | +| `FLOW_CPUS` | `0` | CPU budget; `0` detects cores and reserves two for the engine | +| `FLOW_GPUS` | `0` | local GPU count; `0` detects NVIDIA device nodes | +| `FLOW_RAM` | `0` | memory budget in MB; `0` detects host/container memory and reserves 1 GiB | +| `FLOW_WORKER_MAX_RSS_MB` | `0` | retire a local worker above this memory use after a call; `0` disables | +| `OBS_RETENTION_DAYS` | `30` | retention for metrics, events and run records | -The three concurrency limits are also flags on `fluksio serve` -(`--max-workers`, `--max-cascades`, `--max-runs`), as are the card count and -memory, `--gpus` and `--ram`, and the worker ceiling, `--worker-max-rss`. The flags -outrank the file, and the engine says which numbers it started with in its -first lines. Each pool size must be at least 1 and the others at least 0: -a number below that is refused as a flag error naming it, rather than read as -the default. Leave one empty (or unset) to get the default. +The three concurrency limits must be at least `1`. They also have CLI flags: +`--max-workers`, `--max-cascades` and `--max-runs`. Resource flags are `--gpus`, +`--ram` and `--worker-max-rss`. -An artifact is referred to by a run that recorded it or by a message currently -holding it; anything else is what a camera published four hours ago, and the -sweep is what keeps a flow streaming media from filling the disk. It stands -aside entirely while a run is in flight, since a node may store a checkpoint -long before it returns the reference to it. +Nodes with declared resources are scheduled against the available budget. +Undeclared nodes use the shared pool and receive thread caps. Attached workers +report their own inventory; the engine's resource settings describe only its +own machine. See [Remote workers](../code/workers.md) and the **Workers** screen. -Frames saved with `volatile=True` skip all of that. They go to a ring in memory -instead of the volume, the oldest falling out once the newest need the room, -and the engine pushes them down the websocket to whichever screens are drawing -them — which is what a camera at ten frames a second needs and the store cannot -give it. A frame a run *records* is copied into the store on the way, so -returned media is kept and emitted media is not. Under Docker the ring lives in -the container's `/dev/shm`, whose default is 64 MB: raise `shm_size` alongside -`ARTIFACT_VOLATILE_BYTES`. +## Artifacts -A node that declares nothing is not accounted against `FLOW_CPUS`; it runs on -the shared pool and is given `FLOW_CPUS / FLOW_MAX_WORKERS` as a thread cap, so -several at once cannot each size themselves to the whole machine. Setting -`OMP_NUM_THREADS` (or any of its siblings) on the engine yourself overrides -that default. +| Variable | Default | Purpose | +|---|---|---| +| `MAX_ARTIFACT_BYTES` | `2147483648` (2 GiB) | upload limit; `0` removes it | +| `ARTIFACT_GC_INTERVAL_S` | `3600` | cleanup interval; `0` disables | +| `ARTIFACT_GC_GRACE_S` | `3600` | minimum age before cleanup | +| `ARTIFACT_VOLATILE_DIR` | auto-detected | live-media directory under `/dev/shm`, or the temporary directory | +| `ARTIFACT_VOLATILE_BYTES` | `50331648` (48 MiB) | live-media ring size; `0` stores volatile saves durably | -Workers are kept warm between calls, and a library like JAX keeps what it -allocated for as long as its process lives — so a worker that fitted a model -can sit on gigabytes after its run is over, on the shared pool as much as on a -declared one. `FLOW_WORKER_MAX_RSS_MB` retires such a worker when its node -returns, and the next call pays its imports again. That bounds idle memory -only; what a node holds while it runs is accounted, and only if it declared -`ram`. - -`FLOW_CPUS`, `FLOW_GPUS` and `FLOW_RAM` are this machine's figures. An attached worker reports its own when -it dials in, and a node goes to whichever machine can grant what it asked for, -so a GPU on a worker needs no `FLOW_GPUS` here. What every machine has free, and -which nodes are queued, is `GET /api/v1/workers/resources` and the Workers -screen. Named sizes live in the database and are `GET /api/v1/flavors`; -`PROVISIONERS_FILE` is where machines can be started from. See -[declaring resources](../getting-started/data-science.md#declaring-what-a-node-needs). +Cleanup removes durable artifacts no longer referenced by recorded runs or +current messages, and pauses while a run is active. Volatile artifacts use a +bounded ring; the oldest disappear as it fills. Recorded run outputs are +copied into durable storage. If you increase the ring size in Docker, increase +`shm_size` too. ## Agents -| Variable | Default | Notes | +| Variable | Default | Purpose | |---|---|---| -| `MCP_ENABLED` | `false` | opens the `/mcp` endpoint **and** OAuth client registration | -| `MCP_TOKEN_EXPIRE_MINUTES` | `60` | an agent's token is a bearer secret held by a program | -| `MCP_REFRESH_EXPIRE_DAYS` | `30` | | -| `OAUTH_CODE_EXPIRE_SECONDS` | `60` | | +| `MCP_ENABLED` | `false` | enables `/mcp` and OAuth client registration; `.env.example` sets it to `true` | +| `MCP_TOKEN_EXPIRE_MINUTES` | `60` | access-token lifetime | +| `MCP_REFRESH_EXPIRE_DAYS` | `30` | refresh-token lifetime | +| `OAUTH_CODE_EXPIRE_SECONDS` | `60` | authorization-code lifetime | See [Agents over MCP](../code/agents.md). -## Mail +## Mail and monitoring -Needed for password-reset mails. Without `SMTP_HOST` and `EMAILS_FROM_EMAIL`, -mail is simply off. +Password-reset mail requires `SMTP_HOST` and `EMAILS_FROM_EMAIL`. | Variable | Default | |---|---| -| `SMTP_HOST` | — | +| `SMTP_HOST` | unset | | `SMTP_PORT` | `587` | -| `SMTP_USER` / `SMTP_PASSWORD` | — | +| `SMTP_USER` / `SMTP_PASSWORD` | unset | | `SMTP_TLS` / `SMTP_SSL` | `true` / `false` | -| `EMAILS_FROM_EMAIL` | — | -| `EMAILS_FROM_NAME` | `Fluksio` | +| `EMAILS_FROM_EMAIL` | unset | +| `EMAILS_FROM_NAME` | `PROJECT_NAME`, normally `Fluksio` | | `EMAIL_RESET_TOKEN_EXPIRE_HOURS` | `48` | +| `SENTRY_DSN` | unset; enables error reporting when configured | -## Monitoring - -| Variable | Default | Notes | -|---|---|---| -| `SENTRY_DSN` | — | error reporting, if you want it | - -## Health check - -`GET /api/v1/utils/health/` is a *deep* check: it fails when the event loop is -wedged or the state backend is gone, not just when the process is up. That is -what the container healthcheck probes, and what an autoheal sidecar restarts -on. - -## See also - -- [The `fluksio` command](../code/cli.md) — what the data directory holds -- [Getting started: facility automation](../getting-started/facility-automation.md) +`GET /api/v1/utils/health/` checks the event loop and state backend. Docker +uses it as a health check; `make up` also starts the autoheal service to restart +an unhealthy backend. diff --git a/docs/reference/connector-contract.md b/docs/reference/connector-contract.md index 83a46f1..95aff46 100644 --- a/docs/reference/connector-contract.md +++ b/docs/reference/connector-contract.md @@ -1,180 +1,130 @@ # The connector contract -A connector is the device-facing node class. It talks to something outside the -engine (a device, a service, a protocol) and publishes what it finds as -ordinary typed messages, so the rest of a flow cannot tell the difference -between a reading from a heat pump and one from a `print` statement. - -This document is normative: everything here is what the engine relies on and -what a connector may rely on in return. The authoring walkthrough is in -[Writing a connector](../code/connectors.md). +Connectors integrate devices and services through typed messages. +This reference covers the contract the engine supports; start with +[Writing a connector](../code/connectors.md) for a working package. ## Contract version ```python from fluksio.flow.connector import CONTRACT_VERSION, ConnectorNode + class MyConnector(ConnectorNode): contract = CONTRACT_VERSION + title = "My connector" + description = "Reads a device." ``` -`contract` must be declared **on the connector class itself**. The loader reads -it from the class `__dict__` rather than through inheritance, so a connector -written against an older base cannot pass itself off as current. A mismatch is -logged and the connector is skipped; nothing else in the engine changes. +Declare `contract` on the class itself: inheritance does not count. +The current contract version is `1`. A mismatch is logged and the connector +is skipped. Breaking changes bump the version; compatible additions do not. -The version is bumped only when a change would break connectors written against -the old surface. Additions that leave existing connectors working do not bump -it. +`title` defaults to the entry point name; `description` may be empty. ## Packaging and discovery -A connector is an ordinary Python package. It advertises its node class through -an entry point in the `fluksio.node_types` group: +Register the class in your package's `pyproject.toml`: ```toml -[project] -name = "fluksio-connector-skeleton" -version = "0.1.0" - [project.entry-points."fluksio.node_types"] -random_sensor = "fluksio_connector_skeleton:RandomSensor" +my_device = "fluksio_connector_mydevice:MyConnector" ``` -- The **entry point name** is the node type as it appears on the canvas and in - a stored flow. It must not collide with a built-in type (`python`, `mqtt`, - `http`, `influxdb`, `delay`, `mlp`); a collision is logged and skipped. -- The **package name and version** are the manifest. There is no second - metadata file to keep in step, and the editor shows the provenance - (`fluksio-connector-skeleton 0.1.0`) on the node type. +The entry point name is the stored node type. Names already registered, +including built-in types, are skipped. The editor shows the package name +and version from its installed metadata. -Discovery happens once, when the engine starts. A connector's code is imported, -and Python does not re-import a changed module, so installing or upgrading one -means restarting the engine. - -## Declaring what a connector is - -```python -class RandomSensor(ConnectorNode): - contract = CONTRACT_VERSION - title = "Random sensor" # shown in the add-node palette - description = "Emits a random reading." -``` - -`title` falls back to the entry point name; `description` may be empty. +Discovery runs at engine startup. Install connectors in the engine's +environment and restart after installation or code changes. ## Parameters -Settings are a pydantic model, and the editor builds a form from its JSON -schema. Subclass the base so `poll_interval` stays available: +Subclass `ConnectorNode.Params` to keep `poll_interval` and add your settings: ```python +from pydantic import Field + + class Params(ConnectorNode.Params): host: str = "localhost" - api_key: str | None = Field(default=None, json_schema_extra={"x-secret": True}) + api_key: str | None = Field( + default=None, json_schema_extra={"x-secret": True} + ) ``` -- Booleans render as switches, numbers and strings as inputs. Objects and - arrays are not rendered, so keep settings flat. -- A field carrying **`x-secret`** renders as a picker over the stored secrets - and writes a reference, `{"$secret": "name"}`, rather than the value. The - engine resolves it when the node is built, so a credential never lands in - `flow.json`. Always mark credentials this way. -- The parsed model is available as `self.config`. +The node panel builds a form from this model's JSON schema: switches for +booleans, inputs for numbers and strings, and JSON fields for objects and +arrays. Parsed settings are available as `self.config`. + +Mark credentials with `x-secret`. The editor stores a reference such as +`{"$secret": "device-key"}`, and the engine resolves it while building the node. ## Ports and messages -Ports are declared by whoever places the node, not by the connector, and they -are `MessageSpec`s like any other node's: a message name, a `DType`, and an -optional `interval`. +The flow author declares ports as `MessageSpec` values. Each has a message +name, a `DType`, and optional delivery settings. -- `poll()` returns a dict **keyed by output port**, and the base class maps it - onto message names; `write()` receives a dict **keyed by input port** the - same way. -- Every value is checked against the port's declared `DType` on the way out. A - wrong type is an error, not a hint. -- `interval` on a port is enforced by the engine, not by the connector: an - output port publishes at most every *n* seconds, an input port wakes its node - at most that often. A connector should poll at the rate the device is - comfortable with and leave delivery rates to whoever wires it up. -- A key no port declares is an error, and it fails the whole reading rather - than the one value. A mistyped metric name is how a training curve goes - missing. +- `poll()` returns values keyed by **output port**. +- `write()` receives values keyed by **input port**. +- The base class maps local port names to message names and validates types. +- Unknown output keys fail the reading. Return only declared ports. +- Port `interval` limits are enforced by the engine; the connector controls + how often it reads the device. -### When the device decides what the ports are - -A connector for a device whose readings vary by model (which components a -relay has, which entities were flashed onto a board) cannot know its ports in -advance, and returning everything the device reports would fail on the first -value nobody bound. Narrow the reading to the ports that were declared: +For devices with variable readings, filter to the configured outputs: ```python declared = {spec.port for spec in self.output_ports if spec.name} return {port: value for port, value in reading.items() if port in declared} ``` -`spec.port` is the local, unqualified name and stays that way for the node's -whole life, so the set can be taken fresh each time. Say something in the log -when a declared port is not one the device has, once rather than once per poll: from -the canvas a renamed entity and a typo look the same, and both leave a port -silent forever. +Log missing device fields once so a typo does not leave a port silently empty. ## Polling -Set `poll_interval` and implement `poll()`; the base class runs the loop: - ```python -async def poll(self) -> dict[str, Any] | None: +async def poll(self): return {"reading": await self._read_device()} ``` -- Return `None` when there is nothing new. -- **Only changed values are published.** A device polled every few seconds - usually says the same thing, and every publication wakes everything - downstream, so the loop compares against what it last published, meaning what it - actually published, so a publication that failed is retried next tick rather - than counting as said. -- Raising is not fatal: it is reported as a health problem and retried on the - next tick. -- The loop calls `inject`, which runs the graph, on a worker thread. `poll()` - itself runs on the event loop and must not block it. +Set `poll_interval` in seconds; `0` disables polling. Return `None` when +there is nothing new. The base loop publishes only changed values and retries +a failed reading or publication on the next tick. It reports connection health +around each poll. -A connector that is pushed to rather than polled leaves `poll_interval` at 0, -overrides `start`/`stop` to open and close its own subscription, and calls -`self.inject({...})` when something arrives. +`poll()` runs on the event loop. Use `asyncio.to_thread` for blocking reads. +The base loop already calls `inject` on a worker thread. + +For pushed readings, leave `poll_interval=0`, manage the subscription in +`start()` and `stop()`, and call `self.inject({...})` when values arrive. +Offload `inject` too when calling it from the event loop. ## Writing -A connector that commands something rather than only reading it implements -`write()`. It is called when the node's inputs are satisfied, with the values -keyed by input port: - ```python -def write(self, **ports: Any) -> dict[str, Any] | None: +def write(self, **ports): self._device.set(ports["level"]) return None ``` -- Settings are already parsed on `self.config`; `write()` is handed ports only. -- Return `None` unless the device answers something worth publishing, in which - case return it keyed by output port, exactly as `poll()` does. -- It runs on a worker thread rather than the event loop, but the thread is the - scheduler's, so a long block holds up the cascade. Queue the work if the - device is slow. -- A connector that only reads leaves it alone. -- `poll()` and `write()` are independent: a connector may have both, and a - node with inputs and no `poll_interval` never polls. +Settings stay on `self.config`; arguments contain input ports only. +Return `None`, or a dictionary keyed by output port if the device responds +with useful values. -A write is a command, not a value: set `idempotent = False` on the class so a -redelivery after a crash does not undo a newer command that already landed. +Writes run on a scheduler thread. Return promptly or queue slow work. +Polling and writing are independent; a connector may support both. + +Set `idempotent = False` for commands that must not be redelivered after a +crash. Leave `write()` unchanged on a read-only connector. ## Devices whose readings are bytes -A camera frame or a recorded clip is far too big to be a message, so a -connector publishes a reference to it instead: - ```python -async def poll(self) -> dict[str, Any] | None: +import asyncio + + +async def poll(self): jpeg = await asyncio.to_thread(self._grab) return { "frame": self.save_artifact( @@ -183,135 +133,110 @@ async def poll(self) -> dict[str, Any] | None: } ``` -`save_artifact` stores the bytes and returns what an `image`, `audio` or -`video` port carries: the media type has to match the port's type. It only -works once the node has started, since the store is the engine's and is handed -over then. +`save_artifact(data, name="", media_type="application/octet-stream", +volatile=False)` returns a reference for an artifact or media port. +Match the media type to an `image`, `audio`, or `video` port. -`volatile=True` is what a camera publishes with. The frame goes to a ring in -memory rather than the data volume and is pushed down the websocket to -whichever screens are drawing it, so a wall panel sees ten frames a second and -the SD card under it is never written to. Frames last until newer ones need the -room; leave the flag off for a reading somebody asked to keep. +The artifact store is available after `start()`. Use `volatile=True` +for disposable live frames: they go to a bounded memory ring and expire as +new frames arrive. Use durable storage for recordings. -Each reading is a new artifact, which the poll loop publishes because its -digest differs from the last. Set `poll_interval` to what somebody actually -wants to look at. Through a portal the bytes are fetched rather than pushed, so -a remote panel wants a frame every second or two; higher rates than that are -for the local network, and full-rate video still belongs on the camera's own -stream. +Local screens receive pushed live frames; portal screens fetch the bytes. +Choose a modest poll rate for remote viewing. ## Keeping state across restarts -A connection, a session or a device handle lives in the connector's own -attributes and goes with the process. The few things that must not, such as -the certificate a device presented on first contact or a token it paired with, -go in `self.store`: +Keep connections and device handles in node attributes. Store short strings +such as pairing tokens or pinned certificates in `self.store`: ```python -pem = await asyncio.to_thread(self.store.get, f"cert:{self.config.host}") -await asyncio.to_thread(self.store.set, f"cert:{self.config.host}", pem) -await asyncio.to_thread(self.store.delete, f"cert:{self.config.host}") +value = await asyncio.to_thread(self.store.get, "token:device") +await asyncio.to_thread(self.store.set, "token:device", "value") +await asyncio.to_thread(self.store.delete, "token:device") ``` -- Keys and values are strings; `get` returns `None` for a key that is not - there, and `delete` returns whether there was one. -- The store belongs to the connector's **top-level package**, not to a node: - every node of the package sees the same keys, so two nodes on one bridge - share what was stored for it. Put the device in the key. -- It lives in the engine's database, so it survives a restart and needs - nothing `fluksio serve` does not have. It is available before `start`. -- Limits: a key is 1 to 200 characters without `/` or whitespace, a value at - most 64 kB, and a package at most 256 keys. Past any of them, `set` raises - `ValueError`. Bytes worth keeping belong in `save_artifact`. -- Every call is a database round trip, so on the event loop it goes through - `asyncio.to_thread`. +- Keys and values are strings. `get` returns `None` for a missing key; + `delete` returns whether it existed. +- All nodes from the connector's top-level package share the namespace. + Include a device identifier in each key. +- Storage is in the engine's database and is available before `start()`. +- Keys allow 1–200 characters, without `/` or whitespace. Values allow + 64 KiB and each package allows 256 keys; exceeding a limit raises `ValueError`. +- Calls block on the database: use `asyncio.to_thread` on the event loop. -A superuser can see the keys (never the values) and clear them, all of them or -one at a time, under *Stored state* in a connector node's settings, or through -the API, which names the connector by any node type it installs: +Superusers can list keys and clear them under **Stored state** in the node +panel. Values are never returned. The equivalent API paths are: -```sh -curl -s $FLUKSIO/connectors/hue/state -H "Authorization: Bearer $TOKEN" -curl -X DELETE $FLUKSIO/connectors/hue/state/cert:10.0.0.2 -H "Authorization: Bearer $TOKEN" -curl -X DELETE $FLUKSIO/connectors/hue/state -H "Authorization: Bearer $TOKEN" -``` +| Method | Path | +|---|---| +| `GET` | `/api/v1/connectors/{node_type}/state` | +| `DELETE` | `/api/v1/connectors/{node_type}/state/{key}` | +| `DELETE` | `/api/v1/connectors/{node_type}/state` | -So a connector has to cope with a key disappearing: clearing is how somebody -tells it to start over. +Handle missing keys: clearing one asks the connector to pair or learn again. ### Pinning a device's certificate -A device on the local network presents a certificate no system trusts and that -names a serial or a MAC rather than its address, so `verify=True` fails and -`verify=False` trusts anyone. `fluksio.flow.connector_tls` trusts the -certificate the device presented on first contact instead, kept in `self.store` -under `cert:
`: +`fluksio.flow.connector_tls` supports trusting a local device's certificate +on first contact: ```python -from fluksio.flow.connector_tls import context_for, pinned_certificate, raise_if_changed +import httpx + +from fluksio.flow.connector_tls import ( + context_for, + pinned_certificate, + raise_if_changed, +) address = f"{self.config.host}:{self.config.port}" -pem = pinned_certificate(self.store, address, self.config.timeout) # blocking +pem = pinned_certificate(self.store, address, self.config.timeout) try: with httpx.Client(verify=context_for(pem)) as client: - ... + response = client.get(f"https://{address}/status") + response.raise_for_status() except httpx.ConnectError as exc: - raise_if_changed(exc, pem, address, self.config.timeout, "my_node_type") + raise_if_changed(exc, pem, address, self.config.timeout, "my_device") raise ``` -- First contact sends a TLS hello and nothing else; a device that does not - complete one raises `OSError` and nothing is pinned. -- Any other certificate is refused in the handshake, before a key or a command - is sent. `raise_if_changed` turns that refusal into `CertificateChanged`, - which names the new SHA-256 and the key to clear, under the node type given. -- The pinned certificate's dates are not checked: a pin trusts that exact - certificate, so one that expired, or an engine clock that is off, does not - lock the device out. -- Clearing the key accepts whatever the device presents next. +Run this blocking work off the event loop. First contact pins the certificate +under `cert:
`; a changed certificate is refused before sending a +credential or command. `raise_if_changed` reports `CertificateChanged` +with the fingerprint and key to clear. -## Lifecycle +Pins trust the exact certificate, including after its expiry. Clearing the +key accepts the certificate presented on the next connection. + +## Lifecycle and health ```python -async def start(self, app=None) -> None: ... -async def stop(self, app=None) -> None: ... +async def start(self, app=None): + await super().start(app) + + +async def stop(self, app=None): + await super().stop(app) ``` -- `start` is called when the connector's flow starts, and after every rebuild. -- `stop` is called before a rebuild and when the flow is stopped. **It must be - idempotent**: it is called whether or not `start` succeeded. -- Overriding either means calling `super()` if the polling loop is also wanted. -- The `app` argument is the FastAPI application, for the rare connector that - needs to mount a route. Most ignore it. +`start` runs when a flow starts or rebuilds. `stop` runs before rebuilding +or stopping, even after a failed start; make it idempotent. Call `super()` +when overriding these hooks if you want the base polling loop. -A stopped flow gets no `start` at all: that is what stopping it means. +The optional `app` is the FastAPI application, for connectors that mount +routes. Most connectors can ignore it. -## Health +Report connection changes if you manage your own subscription: ```python self.report_health("ok") -self.report_health("degraded", "3 of 5 registers timed out") -self.report_health("down", str(exc)) +self.report_health("degraded", "Some readings timed out") +self.report_health("down", "Connection lost") ``` -Three values, `ok`, `degraded` and `down`, plus an optional detail string. -Reporting the same status twice is free, since only changes are published. A node -reporting `down` is named among its flow's issues and counted on the health -summary on Home; `degraded` means still working, and is not. The polling loop -already reports around `poll()`; a connector managing its own connection should -report when it connects and when it loses the connection. +Repeated statuses do not emit another event. `down` appears among flow issues +and on Home; `degraded` means the connection still works. Loading failures +are reported separately. -Health is about the connection, not about loading: a connector that fails to -load is reported separately and never runs. - -## What a connector must not do - -- **Hold state that belongs in the flow.** Logic nodes are stateless and run in - parallel; a connector may hold a connection, a session, or a device handle, - and keep what has to outlive a restart in `self.store`, but a value another - node needs belongs in a message. -- **Block the event loop.** `poll`, `start` and `stop` are async and run on the - loop. Use `asyncio.to_thread` for blocking I/O. -- **Read credentials from the environment.** They come through parameters, so - the person deploying a flow can change them without touching the host. +Keep shared flow values in messages, credentials in secret parameters, and +blocking I/O off the event loop. diff --git a/docs/reference/node-types.md b/docs/reference/node-types.md index 06afaa8..6d70d50 100644 --- a/docs/reference/node-types.md +++ b/docs/reference/node-types.md @@ -1,17 +1,18 @@ # Node types -Every type the canvas can place. Each is configured by filling in a form the -editor generates from its parameter schema, so they all behave the same way. - -Anything here can also be written as a **Function** node. These types cover the -shapes that come up most often, as a form to fill in rather than repeated code. +Built-in nodes cover common integrations, timing, and logic. Configure their +settings in the node panel; use a **Function** node for your own Python. `GET /flows/node-types` returns this list with each type's full parameter schema, including any [connectors](#connectors) installed on your instance. +MQTT and InfluxDB need `pip install 'fluksio[server]'` on a Python installation. +The Docker image includes these dependencies. Run the broker and database +separately, or use the development stack's test services. + ## Function -**`python`** — your own code, run on every incoming message. +**`python`** — run your own code when its triggering inputs arrive. The one type with a source file. Its arguments are its input ports and its own settings; its return value is a dict keyed by output ports. See @@ -46,9 +47,11 @@ A topic may be a filter: `+` matches one level, `#` the rest. Everything a filter matches lands on the same port, so use one port per topic where the difference matters. -`json_key` is for a device that wraps its reading: Victron publishes -`{"value": 47}` on every path. Without it, a payload object is unwrapped only -when it happens to carry the port's own name as a key. +Set an explicit topic: the default `*` is a literal topic, not MQTT's +match-all filter. + +For `{"value": 47}`, set `json_key` to `value`. Without it, an object is +unwrapped only if it contains the port's name as a key. ### HTTP @@ -81,6 +84,7 @@ Outputs only makes it a **webhook**: the engine mounts a route at | `url`, `token`, `org`, `bucket` | connection; `token` takes a secret reference | | `write_precision` | `ns`, `us`, `ms` (default) or `s` | | `query_range` | default window for queries, e.g. `-1h` | +| `timeout` | operation timeout in seconds; default `10.0` | | `writes` | per-input: `measurement`, `field`, `tags` | | `queries` | per-output: `measurement`, `field`, `tags`, `range`, `aggregation` | @@ -96,12 +100,10 @@ Outputs only makes it a **webhook**: the engine mounts a route at } ``` -**Query passthrough.** An incoming message holding a `flux` key is run as -written, and the rows come back on the first output port as -`{"rows": [{ts, value, field, measurement, tags}], ...}`. - -A database node holds the credentials and the connection, nothing else. -Building a query and shaping its rows are Function nodes on either side. +**Query passthrough.** An incoming object with a `flux` key supplies the query. +The first output receives an object with `rows`, each containing `ts`, `value`, +`field`, `measurement`, and `tags`. Use Function nodes to build queries and +shape results for charts. ### Notification @@ -115,10 +117,10 @@ Building a query and shaping its rows are Function nodes on either side. | `priority` | `default` | `min`, `low`, `default`, `high`, `urgent` | | `tags` | — | comma-separated ntfy tags | | `token` | — | for a protected topic; takes a secret reference | +| `timeout` | `10.0` | request timeout in seconds | -This is a *flow* deciding something is worth saying. The engine reporting that -it broke is [Alerts](../interface/operations.md#alerts), which is a different -thing configured elsewhere. +For engine failures rather than flow messages, configure +[Alerts](../interface/operations.md#alerts). ## Timing @@ -135,9 +137,7 @@ thing configured elsewhere. | `at_start` | `false` | emit once when the flow starts | | `start_delay` | `1.0` | how long to wait before that first emission | -The scheduler: a `cron` expression here is what makes a flow run by the clock. -It is also the most-placed node in a real instance, mostly as a button someone -presses. +Use `cron` for scheduled emissions, or trigger the node by hand for testing. ### Delay & schedule @@ -154,17 +154,12 @@ Order of operations: rate check → delay → forward. With a `cron` and no inpu it emits the current time on each tick; with inputs it emits the last value it received. -The engine sleeps until a delay is due rather than polling for it, so a `delay` -fires within a few milliseconds of its deadline on an idle engine, and a due -timer is taken off the queue before work that is merely waiting. It is not a -real-time guarantee: the wait ends on a normal thread, and a node that then -needs a busy cascade slot waits for one. +Timers wait until their deadline; busy workers can delay execution. !!! note "Not in a batch flow" - A rate limit holds a value back for a timer to release, and a run has no - timer, since the value would be dropped rather than delayed. Submitting a batch - flow with a rate-limited port is refused instead. + Batch flows reject rate-limited non-streaming ports. Runs have no live + timer to release a held value. ### Trigger @@ -181,14 +176,11 @@ starts from a value arriving. For a cron tick, see [`inject`](#inject). | `passthrough` | `false` | send the incoming value instead of `first` | | `wait_port` | — | an input carrying the wait, when it differs per message | -The shape this exists for: *the door opened, so turn the light on, and off -again in two minutes unless it opens again.* +For example: turn a light on when the door opens, then off after two quiet +minutes. -`wait_port` covers the case where how long to wait is itself a value: a -rollershutter takes 26 seconds up and 28 down, so the node that decides the -direction says how long to run for as well. A wait of zero or less sends -nothing afterwards, and still cancels whatever the last message scheduled, -which is how a *stop* is commanded exactly once. +Use `wait_port` when the delay comes from an input. A wait of zero or less +cancels the pending second value and schedules nothing new. ## Logic @@ -243,8 +235,8 @@ The cheapest fix for a sensor that reports every second and changes every hour. ### Command -**`exec`** — run a command in the engine's container and read its output. -Adding one or changing its settings is a superuser's, since the command is code. +**`exec`** — run a command where the engine runs and read its output. +Adding one or changing its settings requires a superuser. | Setting | Default | Notes | |---|---|---| @@ -255,11 +247,8 @@ Adding one or changing its settings is a superuser's, since the command is code. Outputs the command's stdout, stderr and exit code. -!!! warning "Inside the container, not on the host" - - A flow ported from something that read the host's journal, or poked a host - script, needs either a mount or a small listener on the host side. This - node cannot see the host. +In Docker, the command runs inside the backend container. Mount host files +explicitly if the command needs them. ### File @@ -272,8 +261,7 @@ Outputs the command's stdout, stderr and exit code. | `format` | `text` | `text` or `json` | | `newline` | `true` | end each written record with a newline | -Confined to a directory the engine owns. A flow that could name any path would -be a way to read the secrets store or overwrite a node's source. +Paths stay inside the engine's files directory. ## Numeric @@ -297,17 +285,15 @@ use a [batch flow](../concepts/runs.md) and a Function node. **`wren`** — turn a typed command into values for the devices wired to it. -[Wren](https://github.com/stroblme/wren) is a small language model that runs -on the engine's CPU: about 40 MB, a fifth of a second per command. It reads the -house as text, the areas, the devices in each and their states, and answers a -command with calls on those devices. It is a separate package, installed into -the engine with `pip install wren`; without it the node says so on the canvas. +[Wren](https://github.com/stroblme/wren) is an optional local language model +for device commands. Install it in the engine's environment with +`pip install wren`; the node reports a missing dependency if it is absent. | Input | Type | Notes | |---|---|---| | `prompt` | `str` | the command, and the only input that runs the node | | `presence` | `str` | the area the user is in; anything else reads as away | -| `outdoor` | `float` | °C; 20 without it | +| `outdoor` | `float` or `int` | °C; 20 without it, clamped to −5…30 | | `dark` | `bool` | without it, dark before 07:00 and from 18:00 | The outputs are `actions`, a list of records, and any number of **targets**. A @@ -346,43 +332,31 @@ reads. | `apply` | `false` | write to the targets; off, the node only reports | | `min_confidence` | `0.93` | below this nothing is written | -Each record in `actions` is `{message, value, applied, delay_s, call, -confidence}`, one per call and target. A call nothing is wired for, such as -`sensors.temperature` or a light in a room without one, gets a record with an -empty `message`. The confidence is the lowest probability the model gave any -part of its answer: at the default threshold, nine in ten right answers pass, -and 38 of the 40 wrong ones in the model's own evaluation do not. +Each record in `actions` contains `message`, `value`, `applied`, `delay_s`, +`call`, and `confidence`. Unwired calls have an empty `message` and are not +applied. Keep `apply=false` while checking how the model interprets commands. A call with a delay ("in 10m") is held by the engine's work queue, like the Trigger node's second value: it survives a restart where the queue is in Redis, and not otherwise. -Every house the model was trained on had a living room with a light, media -and heating, and a light in every other area. The node fills in what is not -wired, so the model reads a house it knows; a call on one of those is -reported, never applied. +The node adds default devices to the model's view for missing expected devices. +They are reported but cannot receive writes unless wired. !!! note "What the current model cannot do" - No levels: a light is on or off, never dimmed, coloured or half open. - Delays are whole minutes. Only one device per kind and area. Only the ten - areas above: one it was not trained on is answered confidently and wrongly, - which is why such a port is refused rather than tried. Commands are English, - phrased roughly like "turn on the lights in the kitchen". + Commands are English, such as "turn on the lights in the kitchen". + The model uses the areas above, one device per kind and area, on/off + actions, and whole-minute delays. It does not dim lights or set colors. The chat's [Control tab](../code/agents.md#control) sends commands by hand. ## Connectors -Anything else in the palette came from an installed **connector** package: a -node type written against a published contract and discovered through the -`fluksio.node_types` entry point group. The editor shows which package supplied -it. - -A connector declares its contract version, and one written for a version this -engine does not speak is ignored rather than half-loaded. Installing or -upgrading one takes effect on the next engine restart, because Python does not -re-import a changed module and a rescan would promise more than it delivers. +Installed connector packages add types through the `fluksio.node_types` +entry point group. The editor shows the source package. Incompatible contract +versions are skipped; installing or upgrading a connector requires an engine +restart. The contract itself is in [The connector contract](connector-contract.md), and [Writing a connector](../code/connectors.md) walks through building one. diff --git a/docs/reference/payload-types.md b/docs/reference/payload-types.md index 99f23f6..0359822 100644 --- a/docs/reference/payload-types.md +++ b/docs/reference/payload-types.md @@ -1,207 +1,170 @@ # Payload types -Every port declares a `dtype`, and every value that passes through it is -checked against that declaration. +Every port declares a `dtype`. Fluksio checks values against it and offers +compatible messages when you bind dashboard widgets. -This is not decoration. It is what lets the dashboard editor offer you only the -messages a gauge can actually draw, what lets the canvas refuse a binding before -anything runs, and what lets a downstream node know the shape of what it is -getting before the flow starts. +Messages carry JSON. Binary data lives in the artifact store; messages carry +references to it. -Everything on the wire is JSON. That is what lets the same value pass through -the state backend, the work queue and the worker protocol unchanged. - -## The scalars +## Scalars | `dtype` | Accepts | |---|---| -| `float` | any number — `int` or `float`, but not `bool` | -| `int` | a whole number, not `bool` | -| `str` | a string | -| `bool` | exactly `true` or `false` | +| `float` | integer or floating-point number, excluding booleans | +| `int` | integer, excluding booleans | +| `str` | string | +| `bool` | `true` or `false` | -`bool` is an `int` subclass in Python but is not a number here, so a switch -cannot bind to a temperature. +`NaN` and infinity are rejected, including in structured payloads. For an +unavailable numeric result, omit that output or return `None` to publish +nothing. A numeric port does not accept `None` as its value. -`NaN` and infinity are refused, wherever they sit, including inside a `json`, -`record`, `series` or `list`. JSON cannot spell either, so one that travelled -would come back as a response nobody can parse and a row the database rejects, -a long way from the node that made it. An empty subset or a division with no -denominator is what usually produces one; publish `None` instead. - -## The structured ones - -These are *declared shapes* rather than "some JSON", which is what makes a -widget binding checkable. +## Structured payloads ### `record` -Flat named scalars. +An object with string keys and scalar or null values. Nested objects and +lists are not allowed. ```json {"title": "Boiler", "body": "Pressure low", "severity": "warning"} ``` -A record cannot contain a record, so its shape is readable from the declaration -alone. - -Read by the **Notification** widget. It is also what an alert channel of kind -*dashboard* writes. +Notification and Player widgets read records. Dashboard alert channels also +publish them. ### `list` -Ordered items of one declared shape. The port also declares `item`: +An array with a declared item type: -| `item` | Meaning | +| `item` | Accepts | |---|---| -| unset | `record` — what the agenda and forecast widgets read | -| `float`, `int`, `str`, `bool` | a list of scalars | -| `json` | anything | +| unset or `record` | flat records; used by Agenda and Forecast | +| `float`, `int`, `str`, `bool` | scalars of that type | +| `json` | arbitrary JSON values | -A list of lists, or a list of series, is refused. One declared level is the -point. +`list` and `series` cannot be declared as item types. Use `item: "json"` if +items need arbitrary nested structures. ### `series` -Labelled lines of `(timestamp, value)` pairs, which is what a chart draws. +Labelled lines of numeric `[timestamp, value]` pairs: ```json { "lines": [ {"label": "living", "points": [[1717000000, 21.4], [1717000060, 21.5]]} ], - "range": "-24h" + "range_s": 86400, + "interval_s": 60 } ``` -Keys beside `lines` are carried through untouched, which is how a querying -chart puts the window and resolution it asked for on the request and reads them -back off the answer. That is what stops an answer to a *different* question -from overwriting the picture. +Additional top-level keys are preserved. A Query chart expects the +`range_s` and `interval_s` from its request in the response; this prevents +an answer for another window from replacing its chart. -`GET /runs/series/compare` answers in this shape, which is why comparing three -training curves is a widget binding rather than a screen of its own. +The run-series comparison endpoint also returns this shape. ### `artifact` -A reference to stored bytes. +A reference to bytes in the content-addressed artifact store: ```json {"digest": "sha256:…", "size": 4194304, "media_type": "application/octet-stream", "name": "weights.pt"} ``` -Binary payloads (tensors, checkpoints, images) never travel as a message. The -bytes go to a content-addressed store and the message carries this. A -thirty-megabyte checkpoint never sits in the state backend, and the reference -stays valid wherever the store is reachable from, including on another machine. +`digest` and `size` identify the content. `media_type` and `name` describe it. +The reference works wherever the store is reachable. -Node code produces one with `fluksio.save_artifact` and opens one with -`fluksio.load_artifact`. See [Writing node code](../code/nodes.md#bytes-artifacts). +Create one with `fluksio.save_artifact` and read its bytes with +`fluksio.load_artifact`. See [Artifacts](../code/nodes.md#bytes-artifacts). ### `image`, `audio`, `video` -The same reference, narrowed to a kind of media by its `media_type`. +Artifact references restricted to the matching `media_type` family: ```json -{"digest": "sha256:…", "size": 61344, "media_type": "image/jpeg", "name": "frame.jpg", - "meta": {"width": 1280, "height": 720, "seq": 41}} +{ + "digest": "sha256:…", + "size": 61344, + "media_type": "image/jpeg", + "name": "frame.jpg", + "meta": {"width": 1280, "height": 720, "seq": 41} +} ``` -An `audio` port takes `audio/*` and refuses anything else, so a speech -recogniser declares what it eats rather than taking any bytes at all and -finding out. An `artifact` port still accepts all three: media narrows -artifact, not the other way round. +An `audio` port accepts `audio/*`; `image` and `video` work the same way. +An `artifact` port accepts all three. Optional `meta` fields are for the +consumer; Fluksio does not interpret them. -`meta` is optional and nothing here reads it: sample rates, dimensions and -sequence numbers are for whoever consumes the media. +A media stream is a [streaming port](../concepts/flows.md#streaming-ports) +publishing one reference per frame or chunk. Saving with +[`volatile=True`](../code/nodes.md#media) keeps bytes in memory and lets local +websocket clients receive them with the value. Stored media and portal views +fetch the bytes separately. -Bytes still never travel as a message. A camera publishes one reference per -frame and a microphone one per chunk, which makes a media stream an ordinary -[streaming port](../concepts/flows.md#streaming-ports). What that costs -depends on how the bytes reach the screen: - -| Rate | Where it works | -|---|---| -| A clip a second (speech) | anywhere, including through the portal | -| Ten frames a second (a camera worth watching) | on the local network, with `volatile=True` | -| A frame every second or two (a glance at a door) | anywhere, including through the portal | -| Full-rate video, 30 fps and up | not here — see below | - -The difference is one flag. A frame saved with -[`volatile=True`](../code/nodes.md#media) is held in memory and pushed down the -websocket in front of the value naming it, so a screen draws it without asking -for anything. A stored one is fetched instead: a round trip per frame, which is -a glance rather than a view. Through a portal every frame is fetched, so a -remote panel is in the second row whatever the flag says. - -Above that, video is not a message-plane problem — every frame would still be -an event. Point a media widget's **stream URL** at whatever the camera already -serves and the browser plays it from source; the messages then carry the -occasional still, and the flow reacts to those. +Use occasional snapshots through a portal. For continuous video, point a +Media widget's **Live stream** at a browser-compatible source URL reachable +by the screen. ### `json` -Anything JSON-serializable. The escape hatch, and the right answer when a -payload genuinely has no fixed shape. +Any JSON-serializable value. Use it for payloads without a fixed shape. +Prefer a specific type when one fits; it gives widgets and other authors a +clearer contract. -Reach for it last. A `json` port tells the canvas, the widget picker and the -next author nothing. +## Referencing a run's output -## Naming a run's output +A run parameter can name an existing value instead of repeating it: -Any *run parameter* is also accepted as text, since nobody wants to paste an -object into a shell. `@run:.` names what an earlier run produced, -whatever its type, an `artifact` reference or a `json` config alike. A bare -`sha256:…` digest names content in the artifact store. Both resolve before the -run starts, so the CLI, the run dialog and a python caller all mean the same -thing by the same string. +- `@run:.` resolves an earlier run's output. +- `sha256:…` resolves an artifact digest. -Both spellings are reserved on every input, `str` included: an input that has to -carry one of them literally is asking for a value this engine reads as a name. +These strings resolve before the run starts, regardless of the input's type. +Both prefixes are reserved even for `str` inputs; they cannot be passed as +literal text in run parameters. -## What a widget will bind to +## Widget bindings | Widget | Accepts | |---|---| -| Gauge, Chart, Slider, Bar | `float`, `int` | +| Gauge, live Chart, Slider, Bar | `float`, `int` | +| Query Chart | `series` response, `record` request | | Switch | `bool` | | Agenda, Forecast | `list` | -| Notification | `record` | -| Value | anything | -| Icon | weather strings, booleans and numbers alike | +| Notification, Player | `record` | +| Colour | `list` for HSV/RGB, `str` for hex | | Media | `image`, `audio`, `video`, `artifact` | -| Clock, Text | nothing — they bind to no message | +| Value, Icon, Button, Buttons, Input, Selector | no fixed type restriction | +| Clock, Text, Website | no message binding | -Enforced on the server as well as in the editor. +Player transport commands publish to a separate `str` message. +Binding checks apply in the editor and on the server. ## Icon and colour names -A forecast is a `list` of records, one per column, each shaped -`{label, icon, value}` with an optional `color`. The flow decides what a -column is called and which glyph it gets; the widget only draws it. +Forecast reads a list of records with `label`, `icon` and `value`, plus +optional `color`: ```json [{"label": "Today", "icon": "cloud-rain", "color": "primary", "value": "14°"}] ``` -`icon` and `color` are names from a fixed vocabulary, the same one the icon -widget's picker offers: +The Icon widget uses the same vocabulary: | | Names | |---|---| -| Icons | `sun`, `moon`, `cloudy`, `cloud-rain`, `cloud-drizzle`, `cloud-snow`, `cloud-lightning`, `cloud-fog`, `wind`, `umbrella`, `snowflake`, `droplets`, `thermometer`, `thermometer-sun`, `flame`, `fan`, `sun-snow`, `door-open`, `house`, `bed`, `lightbulb`, `plug`, `music`, `volume-2`, `volume-x`, `zap`, `battery-charging`, `arrow-up`, `arrow-down`, `triangle-alert`, `check`, `circle-check` | +| Icons | `sun`, `moon`, `cloudy`, `cloud-rain`, `cloud-drizzle`, `cloud-snow`, `cloud-lightning`, `cloud-fog`, `wind`, `umbrella`, `snowflake`, `droplets`, `thermometer`, `thermometer-sun`, `flame`, `fan`, `sun-snow`, `door-open`, `house`, `bed`, `lightbulb`, `plug`, `cctv`, `music`, `volume-2`, `volume-x`, `zap`, `battery-charging`, `arrow-up`, `arrow-down`, `triangle-alert`, `check`, `circle-check` | | Colours | `default`, `muted`, `primary`, `success`, `danger` | -An empty icon draws nothing. An unknown one draws a placeholder glyph and an -unknown colour draws as `default`; either warns once in the browser console, -listing the names it expected. +An empty icon draws nothing. Unknown icons use a placeholder; unknown colours +use `default`. Each warns once in the browser console. ## Type failures -A value that does not match its port's declaration raises on the node that -published it, naming the port and what arrived. It does not get published, and -it does not reach anything downstream. A wrong value stopping at its source is -easier to diagnose than one propagating. +A mismatched value raises an error on the node that produced it, naming the +port and received value. It is not published downstream. ## See also