docs
Docs / docs (push) Successful in 26s
Playwright Tests / test-playwright (1, 2) (push) Failing after 4m7s
Playwright Tests / test-playwright (2, 2) (push) Failing after 2m30s
pre-commit / pre-commit (push) Failing after 2m42s
Test Backend / test-backend (push) Failing after 3m3s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m51s
Docs / docs (push) Successful in 26s
Playwright Tests / test-playwright (1, 2) (push) Failing after 4m7s
Playwright Tests / test-playwright (2, 2) (push) Failing after 2m30s
pre-commit / pre-commit (push) Failing after 2m42s
Test Backend / test-backend (push) Failing after 3m3s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m51s
This commit is contained in:
+45
-123
@@ -1,145 +1,67 @@
|
||||
# Agents over MCP
|
||||
|
||||
The engine exposes the flow API to agents over the
|
||||
[Model Context Protocol](https://modelcontextprotocol.io). Point Claude Code,
|
||||
Claude Desktop, Codex or anything else that speaks MCP at it, and it can read,
|
||||
build, publish and run flows.
|
||||
Fluksio exposes flow tools over MCP. An agent can inspect, edit, publish,
|
||||
and run flows with the permissions of the user who authorizes it.
|
||||
|
||||
## Switching it on
|
||||
## Connect an agent
|
||||
|
||||
```sh
|
||||
MCP_ENABLED=true
|
||||
```
|
||||
Set `MCP_ENABLED=true` in the engine's environment and restart it. The Python
|
||||
package defaults to off; the Docker template enables it.
|
||||
|
||||
It is off by default: switching it on opens client registration to whoever can
|
||||
reach the host.
|
||||
Configure your MCP client with an HTTP endpoint:
|
||||
|
||||
The endpoint is `https://api.${DOMAIN}/mcp`, or `http://api.localhost/mcp` on a
|
||||
local stack.
|
||||
- Local Docker stack: `http://api.localhost/mcp`
|
||||
- Deployment: `https://api.<your-domain>/mcp`
|
||||
|
||||
## Connecting a client
|
||||
The client registers, opens browser authorization, and receives an OAuth
|
||||
token after you approve access. The issuer and browser redirects use
|
||||
`DOMAIN`, `ENVIRONMENT`, and `FRONTEND_HOST`, so configure those for the
|
||||
addresses your client and browser can reach.
|
||||
|
||||
```sh
|
||||
claude mcp add --transport http fluksio https://api.${DOMAIN}/mcp
|
||||
```
|
||||
Manage registered clients under **Admin → Agents**. Revoking a client
|
||||
invalidates its access. MCP uses a separate signing key at
|
||||
`OAUTH_PRIVATE_KEY_FILE`; rotating that key invalidates all MCP tokens.
|
||||
|
||||
The OAuth flow does the rest: the client registers itself, you approve it in
|
||||
the browser, and it exchanges the grant for a token. Nothing to paste.
|
||||
MCP is not forwarded by the [portal tunnel](../interface/portal.md).
|
||||
Connect directly to the instance's API host.
|
||||
|
||||
## What an agent may do
|
||||
## Available tools
|
||||
|
||||
An agent acts as **the person who approved it**, and only through the same REST
|
||||
API the dashboard uses. It cannot do anything you could not do in the browser,
|
||||
and it cannot do anything *you* cannot do. An agent approved by a non-superuser
|
||||
is not one either.
|
||||
|
||||
Its tokens are signed with a keypair of their own, kept beside the flow store.
|
||||
Deleting that file revokes every agent's access without logging anyone out of
|
||||
the dashboard.
|
||||
|
||||
Registered clients are managed under **Admin**.
|
||||
|
||||
## The tools
|
||||
|
||||
Read-only:
|
||||
|
||||
| Tool | Returns |
|
||||
| Task | Tools |
|
||||
|---|---|
|
||||
| `list_flows` | every flow with its node count and state |
|
||||
| `get_flow` | one flow's definition, node status and issues |
|
||||
| `get_node_source` | a node's Python |
|
||||
| `list_node_types` | every type and its parameter schema |
|
||||
| `get_flow_state` | the current value of each of a flow's messages |
|
||||
| `get_message_history` | one message's recent numeric values |
|
||||
| `get_graph` | every flow as one graph |
|
||||
| `list_shared_nodes` | the shared library and its usages |
|
||||
| `list_secrets` | secret **names**, never values |
|
||||
| `get_modules` | the packages node code can import |
|
||||
| `get_health` | flows, nodes, queue and loop lag |
|
||||
| `get_metrics` | executions, errors and timings per minute |
|
||||
| `list_failures` | what went wrong recently |
|
||||
| `list_runs` | recent cascades |
|
||||
| Inspect flows | `list_flows`, `get_flow`, `get_node_source`, `list_node_types`, `get_graph`, `list_shared_nodes` |
|
||||
| Inspect values | `get_flow_state`, `get_message_history` |
|
||||
| Inspect the instance | `list_secrets` (names only), `get_modules`, `get_health`, `get_metrics`, `list_failures`, `list_runs` (live cascades) |
|
||||
| Edit flows | `save_flow`, `save_node_source`, `validate_flow`, `publish_flow`, `discard_draft`, `delete_flow` |
|
||||
| Operate flows | `run_flow`, `trigger_node`, `start_flow`, `stop_flow`, `pause_flow`, `resume_flow`, `cancel_node` |
|
||||
| Install node packages | `apply_modules` |
|
||||
|
||||
Editing:
|
||||
Tools call the same API as the browser. Drafts, validation, version checks,
|
||||
and superuser restrictions apply. Secrets cannot be read back.
|
||||
|
||||
| Tool | Does |
|
||||
|---|---|
|
||||
| `save_flow` | write a draft |
|
||||
| `save_node_source` | write a node's code into the draft |
|
||||
| `validate_flow` | the issues, without saving |
|
||||
| `publish_flow` | put the draft live (takes the version it saw) |
|
||||
| `discard_draft` | throw the draft away |
|
||||
| `delete_flow` | delete it |
|
||||
| `apply_modules` | replace the package manifest |
|
||||
|
||||
Operating:
|
||||
|
||||
| Tool | Does |
|
||||
|---|---|
|
||||
| `run_flow` | run every node once from the values given |
|
||||
| `trigger_node` | feed one node and run what is downstream |
|
||||
| `start_flow` / `stop_flow` | activate or tear down |
|
||||
| `pause_flow` / `resume_flow` | hold messages, or let them through |
|
||||
| `cancel_node` | kill a node's code while it runs |
|
||||
|
||||
## Why it is shaped this way
|
||||
|
||||
The tools do not reach into the engine. Each one calls the same REST endpoint
|
||||
the dashboard calls, over an in-process transport.
|
||||
|
||||
That keeps exactly one description of what a flow is and how it may be changed:
|
||||
the validation, the draft/publish split, and the version check that stops two
|
||||
clients overwriting each other. An agent saving a flow through a private back
|
||||
door could write one the canvas cannot open.
|
||||
|
||||
The caller's token rides along on every hop, so the API sees the agent's own
|
||||
identity rather than a service account. The audit trail on Home names it.
|
||||
A useful workflow: ask the agent to inspect the flow, save a draft, and
|
||||
validate it. Review the draft before asking it to publish. The agent has
|
||||
a publish tool, so make your intent explicit.
|
||||
|
||||
## The assistant
|
||||
|
||||
The same tools drive the chat inside the dashboard. An operator sets a
|
||||
provider under **Settings → Assistant** (anything OpenAI-compatible: OpenAI,
|
||||
Anthropic, OpenRouter, a local Ollama), and an **Assistant** entry appears in
|
||||
the sidebar for everyone. The panel opens on any screen and keeps its place
|
||||
from screen to screen; on a flow, it knows which one you are looking at.
|
||||
A superuser configures an OpenAI-compatible provider under
|
||||
**Settings → Assistant**, with a base URL, API key, and model. This enables
|
||||
the dashboard's Assistant entry for signed-in users.
|
||||
|
||||
The model calls the tools above, in-process, as **you**: every call carries
|
||||
your own token, so the chat can do nothing you could not do yourself, and its
|
||||
edits show up in the editor as drafts the moment they are saved. It is told to
|
||||
publish only when asked, but it has the tool, so read what it did before
|
||||
saying so. The trace under each answer lists every call it made.
|
||||
|
||||
The conversation is one per person, kept on the instance beside the
|
||||
configuration, and the bin in the panel's header starts it over. `MCP_ENABLED`
|
||||
is not needed for it: the endpoint is for agents outside, the chat for the
|
||||
one inside.
|
||||
The assistant uses the same tools with your own permissions. Draft edits
|
||||
appear in the editor, and each answer includes a tool trace. Conversations
|
||||
are stored per user on the instance; the bin button clears yours.
|
||||
`MCP_ENABLED` is only needed for external clients, so the built-in assistant
|
||||
works without it.
|
||||
|
||||
## Control
|
||||
|
||||
Where a live flow has a [`wren` node](../reference/node-types.md#wren) with
|
||||
its `prompt` and `actions` wired, the chat gains a **Control** tab, and Home a
|
||||
chat button at the top right of the brain graph (in the header, on a phone)
|
||||
that opens it. No provider is needed for this one: the model is the node.
|
||||
A live [Wren node](../reference/node-types.md#wren) with both `prompt` and
|
||||
`actions` wired adds a **Control** tab to the chat. It uses the node's local
|
||||
model and needs no assistant provider.
|
||||
|
||||
A command is published to the node's `prompt` message as a value from the
|
||||
chat, and the answer is the next value of its `actions`: each call, the
|
||||
message it went to, whether it was applied, and how sure the model was. A
|
||||
failing node answers with its error, and one that says nothing for fifteen
|
||||
seconds is reported as such. With several nodes, a picker in the header
|
||||
chooses. The conversation is kept in the browser only, and a reload clears
|
||||
it.
|
||||
|
||||
## A note on working this way
|
||||
|
||||
The pattern that works is the same one that works with a person: let the agent
|
||||
read the flow and its issues, let it write a **draft**, and publish yourself
|
||||
once you have looked. `validate_flow` before `publish_flow` is cheap and tells
|
||||
you what the canvas would have told you.
|
||||
|
||||
## Limits
|
||||
|
||||
- MCP is not currently reachable through a [portal](../interface/portal.md)
|
||||
tunnel: the proxy forwards `/api/v1/` only. Connect an agent on the same
|
||||
network as the instance.
|
||||
- Secrets are never readable, by an agent or by anyone else. `list_secrets`
|
||||
returns names.
|
||||
Choose a node, type a command, and inspect the reported actions: target
|
||||
message, value, confidence, and whether each action was applied.
|
||||
The node's `apply` and `min_confidence` settings decide whether it writes
|
||||
to devices. Control conversations stay in the browser and clear on reload.
|
||||
|
||||
+168
-201
@@ -1,274 +1,241 @@
|
||||
# The HTTP API
|
||||
|
||||
Everything the browser does, the API does first. The dashboard is a generated
|
||||
client of this schema, not a privileged path into the engine, so anything you
|
||||
can click, you can script.
|
||||
The API covers the same operations as the browser. Use it for scripts,
|
||||
integrations, or your own client.
|
||||
|
||||
Base URL: `https://api.${DOMAIN}/api/v1`, or `http://127.0.0.1:8000/api/v1` for
|
||||
a `fluksio serve` instance.
|
||||
Set the base URL for your instance:
|
||||
|
||||
```sh
|
||||
export FLUKSIO=http://127.0.0.1:8000/api/v1
|
||||
```
|
||||
|
||||
Use the port printed by `fluksio serve`. For Docker, use
|
||||
`http://api.localhost/api/v1`, `http://<host-ip>:8080/api/v1` on a LAN stack,
|
||||
or `https://api.<your-domain>/api/v1` for a deployment.
|
||||
All paths below are relative to this base.
|
||||
|
||||
## Authenticating
|
||||
|
||||
```sh
|
||||
export TOKEN=$(curl -s -X POST $FLUKSIO/login/access-token \
|
||||
-d "username=admin@example.com&password=..." | jq -r .access_token)
|
||||
`fluksio serve` signs the local CLI in automatically. To reuse its token
|
||||
from a project instance, with `jq` installed:
|
||||
|
||||
curl -s $FLUKSIO/users/me -H "Authorization: Bearer $TOKEN"
|
||||
```sh
|
||||
export TOKEN=$(jq -r .token .fluksio/client.json)
|
||||
```
|
||||
|
||||
A bearer token, valid for eight days. `POST /login/test-token` checks one.
|
||||
Password recovery and reset are `POST /password-recovery/{email}` and
|
||||
Adjust the path for `--global` or `--data-dir`. For a remote instance,
|
||||
use its login endpoint with the administrator credentials:
|
||||
|
||||
```sh
|
||||
export TOKEN=$(curl -s -X POST "$FLUKSIO/login/access-token" \
|
||||
--data-urlencode "username=admin@example.com" \
|
||||
--data-urlencode "password=<your-password>" | jq -r .access_token)
|
||||
|
||||
curl "$FLUKSIO/users/me" -H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
|
||||
Session tokens last eight days by default. `POST /login/test-token` checks
|
||||
a token. Password recovery uses `POST /password-recovery/{email}` and
|
||||
`POST /reset-password/`.
|
||||
|
||||
Agents authenticate differently; see [Agents over MCP](agents.md).
|
||||
[Agents over MCP](agents.md) use separate OAuth tokens.
|
||||
|
||||
!!! tip "The interactive schema"
|
||||
!!! tip "Interactive API reference"
|
||||
|
||||
When `ENVIRONMENT` is not `production`, the full OpenAPI schema is at
|
||||
`/docs` (Swagger) and `/redoc`, and the raw document at
|
||||
`/api/v1/openapi.json`, for a browser on the engine's own machine (or any,
|
||||
with `API_DOCS_ENABLED=true`). That is the authoritative reference; this
|
||||
page is the tour.
|
||||
|
||||
It is closed in production: the schema enumerates every endpoint the
|
||||
instance serves, including paths webhook nodes mounted at runtime.
|
||||
Outside production, open `/docs` or `/redoc` on the API host.
|
||||
The schema is at `/api/v1/openapi.json`. Access is limited to loopback
|
||||
unless `API_DOCS_ENABLED=true`; the Docker development stack enables it.
|
||||
Production disables these endpoints.
|
||||
|
||||
## Flows
|
||||
|
||||
| Method | Path | What |
|
||||
| Method | Path | Action |
|
||||
|---|---|---|
|
||||
| `GET` | `/flows/` | every flow, with node counts, draft and running state |
|
||||
| `GET` | `/flows/{name}` | one flow — the draft if there is one — with node status and issues |
|
||||
| `PUT` | `/flows/{name}` | save a draft (`version` must match, or 409) |
|
||||
| `POST` | `/flows/{name}/publish` | put the draft live |
|
||||
| `POST` | `/flows/{name}/discard-draft` | throw the draft away |
|
||||
| `POST` | `/flows/{name}/rename` | rename it |
|
||||
| `DELETE` | `/flows/{name}` | delete it. Its runs stay — a run is the record of what ran, and only `DELETE /runs` clears one. Refused with 409 while a run of it is `running` or `queued` |
|
||||
| `GET` | `/flows/node-types` | every node type and its parameter schema |
|
||||
| `GET` | `/flows/graph` | every flow as one graph — what Home draws |
|
||||
| `GET` | `/flows/` | list flows and their state |
|
||||
| `GET` | `/flows/{name}` | read the draft when present, with status and issues |
|
||||
| `PUT` | `/flows/{name}` | save a draft; send the current `version` |
|
||||
| `POST` | `/flows/{name}/publish` | publish; send `{"version": <current-version>}` |
|
||||
| `POST` | `/flows/{name}/discard-draft` | discard unpublished edits |
|
||||
| `POST` | `/flows/{name}/rename` | rename |
|
||||
| `DELETE` | `/flows/{name}` | delete; refused while a run is queued or running |
|
||||
| `GET` | `/flows/node-types` | list types and parameter schemas |
|
||||
| `GET` | `/flows/graph` | read the graph shown on Home |
|
||||
|
||||
Reading a flow returns the *draft* when one exists, along with a preview of how
|
||||
it would run: node statuses and validation issues for the version you are
|
||||
editing, not the one running underneath it.
|
||||
Deleting a flow keeps its run history. Delete runs separately.
|
||||
|
||||
### Node source
|
||||
|
||||
```sh
|
||||
curl -s $FLUKSIO/flows/house/nodes/decide/source -H "Authorization: Bearer $TOKEN"
|
||||
curl -X PUT $FLUKSIO/flows/house/nodes/decide/source -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"code": "def process(x):\n return {\"y\": x * 2}\n"}'
|
||||
curl "$FLUKSIO/flows/house/nodes/decide/source" \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
|
||||
curl -X PUT "$FLUKSIO/flows/house/nodes/decide/source" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"code": "def process(x):\n return {\"y\": x * 2}\n"}'
|
||||
```
|
||||
|
||||
Writing a source is a superuser's, and so is any `PUT /flows/{name}` that
|
||||
adds a Function or Command node, retypes or repoints one, changes a Command
|
||||
node's settings, or changes any node's `resources.env`: each of them is code
|
||||
the engine runs. Anyone else gets a 403 and keeps the rest — wiring, titles,
|
||||
a Function node's settings and every other node.
|
||||
Writing code requires a superuser. So does adding, retyping, or repointing
|
||||
Function and Command nodes, changing Command settings, or changing
|
||||
`resources.env`. Other users can edit wiring, titles, and ordinary settings.
|
||||
|
||||
`POST .../share` promotes a node's source to the shared library;
|
||||
`POST .../unshare` gives it a private copy back. `GET /flows/library` lists the
|
||||
shared sources and which flows use each.
|
||||
Ordinary source edits stay in the draft until publishing. Shared library
|
||||
source changes affect its users immediately.
|
||||
`POST /flows/{name}/nodes/{id}/share` promotes a source to the library;
|
||||
`/unshare` makes it private again. `GET /flows/library` lists shared sources.
|
||||
|
||||
### Running and controlling
|
||||
|
||||
| Method | Path | What |
|
||||
| Method | Path | Action |
|
||||
|---|---|---|
|
||||
| `POST` | `/flows/{name}/start` · `/stop` | activate or tear down its subscriptions and schedules |
|
||||
| `POST` | `/flows/{name}/pause` · `/resume` | hold messages instead of running them |
|
||||
| `POST` | `/flows/{name}/step` | release exactly one held message |
|
||||
| `POST` | `/flows/{name}/validate` | the issues, without saving |
|
||||
| `POST` | `/flows/{name}/run` | run every node once, from the values you pass |
|
||||
| `POST` | `/flows/{name}/nodes/{id}/trigger` | feed values into one node |
|
||||
| `POST` | `/flows/{name}/nodes/{id}/cancel` | kill the worker running it right now |
|
||||
| `POST` | `/flows/{name}/nodes/{id}/acknowledge` | clear the failure the canvas is marking |
|
||||
| `POST` | `/flows/{name}/start`, `/flows/{name}/stop` | activate or stop a live flow |
|
||||
| `POST` | `/flows/{name}/pause`, `/flows/{name}/resume` | hold or release incoming work |
|
||||
| `POST` | `/flows/{name}/step` | release one held message |
|
||||
| `POST` | `/flows/{name}/validate` | validate the current draft |
|
||||
| `POST` | `/flows/{name}/run` | run the current draft once with `{"inputs": {...}}` |
|
||||
| `POST` | `/flows/{name}/nodes/{id}/trigger` | feed a node with `{"values": {...}}` |
|
||||
| `POST` | `/flows/{name}/nodes/{id}/cancel` | stop active node code |
|
||||
| `POST` | `/flows/{name}/nodes/{id}/acknowledge` | clear its failure marker |
|
||||
|
||||
`POST /flows/{name}/run` on a **batch** flow submits a run instead, because
|
||||
that is what running one means: the parameters, the series and the result are
|
||||
the point.
|
||||
A preview run can have real side effects. On a batch flow, `/run` submits a
|
||||
recorded run; use the run endpoints below for parameters and tracking.
|
||||
|
||||
## Messages
|
||||
|
||||
```sh
|
||||
curl -s $FLUKSIO/messages/ -H "Authorization: Bearer $TOKEN"
|
||||
curl -X POST $FLUKSIO/messages/house.setpoint -H "Authorization: Bearer $TOKEN" \
|
||||
curl "$FLUKSIO/messages/" -H "Authorization: Bearer $TOKEN"
|
||||
curl -X POST "$FLUKSIO/messages/house.setpoint" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"value": 22.5}'
|
||||
curl -s $FLUKSIO/messages/house.temperature/history -H "Authorization: Bearer $TOKEN"
|
||||
curl "$FLUKSIO/messages/house.temperature/history" \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
|
||||
`GET /messages/` is the catalogue: every message any published flow declares,
|
||||
with its last value. Publishing puts a value into the graph exactly as a
|
||||
dashboard control does, which means only a message some flow *declares* can be
|
||||
published to. Flows own the namespace; everything else is a client of it.
|
||||
The catalogue contains messages declared by published flows and their last
|
||||
values. You can publish only to declared messages, just as dashboard controls do.
|
||||
|
||||
## Runs
|
||||
|
||||
| Method | Path | What |
|
||||
| Method | Path | Action |
|
||||
|---|---|---|
|
||||
| `POST` | `/runs/flows/{name}` | queue one run — `{"params": {...}, "seed": 7, "draft": false, "no_cache": false}`. `"cause"` says where it came from — `api` (the default), `cli` or `sdk` |
|
||||
| `POST` | `/runs/flows/{name}/sweep` | queue many, sharing a `group_id` |
|
||||
| `POST` | `/runs/import` | record runs executed somewhere else, up to 200 per request. Answers `[{"external_id", "id", "created"}]` |
|
||||
| `GET` | `/runs` | the queryable history: `?flow=`, `?status=`, `?group=`, `?digest=`, `?since=`, `?before=`, `?limit=`, `?offset=` |
|
||||
| `DELETE` | `/runs?flow=` | every run of one flow, and everything hanging off them. `flow` is required. Refused with 409 while one of them is still going |
|
||||
| `GET` | `/runs/overview` | one row per flow that has runs, with how many are running or queued |
|
||||
| `GET` | `/runs/export/metrics?…&name=&stride=&format=` | every selected run's series as one long table: `run, name, step, ts, value` |
|
||||
| `GET` | `/runs/export/runs?…¶ms=&metrics=&format=` | one row per run: its inputs as columns, its final numbers, its status and provenance |
|
||||
| `GET` | `/runs/metrics/names?…` | every metric name the selected runs recorded, distinct; takes the export's own filters |
|
||||
| `GET` | `/runs/{id}` | one run in full: params, result, per-node record, artifacts |
|
||||
| `POST` | `/runs/{id}/cancel` | stop it |
|
||||
| `DELETE` | `/runs/{id}` | forget it, with its nodes, series and artifact rows. Cancel a live one first |
|
||||
| `POST` | `/runs/{id}/retry` | run the same thing again, as a new run naming this one |
|
||||
| `GET` | `/runs/{id}/metrics?name=&stride=` | one metric's series, in step order; every series of the run without `name` |
|
||||
| `GET` | `/runs/series/compare?ids=a,b,c&metric=&x=` | that metric across several runs. `x` is what to plot against: nothing or `step`, `time` (seconds since each run's own first reading), or another metric's name, joined on the step the two share |
|
||||
| `POST` | `/runs/flows/{name}` | queue a run with `{"params": {...}, "seed": 7}` |
|
||||
| `POST` | `/runs/flows/{name}/sweep` | queue runs with a shared group |
|
||||
| `POST` | `/runs/import` | import up to 200 completed runs |
|
||||
| `GET` | `/runs` | query history |
|
||||
| `GET` | `/runs/overview` | summarize runs per flow |
|
||||
| `GET` | `/runs/{id}` | read parameters, results, nodes, and artifacts |
|
||||
| `POST` | `/runs/{id}/cancel`, `/runs/{id}/retry` | cancel, or create a retry |
|
||||
| `DELETE` | `/runs/{id}` | delete a finished run and its records |
|
||||
| `DELETE` | `/runs?flow={name}` | delete a flow's runs; refused while any are active |
|
||||
| `GET` | `/runs/{id}/metrics?name=&stride=` | read metric series |
|
||||
| `GET` | `/runs/metrics/names` | list metric names across a selection |
|
||||
| `GET` | `/runs/series/compare?ids=a,b,c&metric=&x=` | compare curves by step, time, or another metric |
|
||||
| `GET` | `/runs/export/metrics`, `/runs/export/runs` | stream CSV or JSONL |
|
||||
|
||||
Submitting answers immediately with a `queued` run. Wrong parameters (an
|
||||
undeclared name, a value of the wrong type) come back as a 422 naming the
|
||||
problem, before anything executes.
|
||||
Submissions return a queued run immediately. Invalid or undeclared parameters
|
||||
return `422` before execution. Optional `draft` and `no_cache` flags select
|
||||
unpublished code or skip stage caching.
|
||||
|
||||
`/runs/import` takes `{"runs": [...]}`. Each entry carries `flow`,
|
||||
`external_id`, `params`, `seed`, `status` (`ok`, `error` or `cancelled`),
|
||||
`created_at`, `started_at`, `finished_at`, `result`, `metrics` (`name`,
|
||||
`step`, `ts`, `value`), `artifacts` (`name`, `filename`, `media_type`,
|
||||
`content_base64`), `labels` and `group_id`. The parameters are completed and
|
||||
checked as a submit's are, the run is stored with the cause `import` and an
|
||||
empty commit and code digest, and nothing is queued. The request is checked
|
||||
whole before anything is written, and an `external_id` already imported
|
||||
answers with that run and `"created": false`.
|
||||
History filters include `flow`, `status`, `group`, `digest`, `since`,
|
||||
`before`, `limit`, and `offset`. Rows arrive newest first; use the last
|
||||
row's `created_at` as `before` for stable pagination.
|
||||
|
||||
`?digest=` filters by the hash of the parameters, which is how you find "every
|
||||
run that used exactly this configuration".
|
||||
Exports also accept `ids` and `until`, with `format=csv` or `jsonl`.
|
||||
The metrics export is a long table; the runs export is one row per run.
|
||||
Use `params` and `metrics` to choose columns, including dotted paths into
|
||||
record outputs.
|
||||
|
||||
`?before=` is how a long history is paged: rows come newest first, so handing
|
||||
back the last row's `created_at` reads the next page whatever landed
|
||||
meanwhile, which `?offset=` cannot, since a run submitted between two pages
|
||||
shifts every row down one. `?since=` is inclusive and bounds the other end.
|
||||
|
||||
Both exports stream `csv` (the default) or `jsonl`, and take the selection the
|
||||
history takes plus `?ids=a,b,c`, `?since=` and `?until=`. `export/runs` puts
|
||||
the inputs that *vary* across the selection in `param.` columns (the sweep
|
||||
axis) unless `?params=` names them, and the run's numbers in `metric.`
|
||||
columns. Both are dotted paths into whatever a node returned, so
|
||||
`?metrics=final_metrics.train_loss` selects a field of a record and the
|
||||
defaults reach every number inside one. A key with a dot in its own name, such
|
||||
as `-1.0`, cannot be addressed that way and gets no column. The run id is on every row of both, which is what makes an exported
|
||||
file a join back to the run rather than a loose table.
|
||||
|
||||
`compare` answers in the same shape a chart widget draws, so three training
|
||||
curves side by side is a widget binding rather than a screen of its own.
|
||||
|
||||
See [Runs: pipelines that finish](../concepts/runs.md).
|
||||
Import takes `{"runs": [...]}`, records completed results without executing
|
||||
code, and deduplicates by `external_id`. See
|
||||
[Runs](../concepts/runs.md) for examples, artifacts, caching, and exports.
|
||||
|
||||
## Artifacts
|
||||
|
||||
```sh
|
||||
curl -X PUT $FLUKSIO/artifacts -H "Authorization: Bearer $TOKEN" \
|
||||
--data-binary @model.pt
|
||||
curl -s $FLUKSIO/artifacts/sha256:abc... -H "Authorization: Bearer $TOKEN" -o model.pt
|
||||
curl -X PUT "$FLUKSIO/artifacts?name=model.pt" \
|
||||
-H "Authorization: Bearer $TOKEN" --data-binary @model.pt
|
||||
curl "$FLUKSIO/artifacts/<digest>" \
|
||||
-H "Authorization: Bearer $TOKEN" -o model.pt
|
||||
```
|
||||
|
||||
Content-addressed, so uploading the same bytes twice stores them once. Node
|
||||
code normally reaches these through `fluksio.save_artifact` /
|
||||
`fluksio.load_artifact` rather than here.
|
||||
The upload returns a reference with the content digest. Identical bytes are
|
||||
stored once. Node code uses `save_artifact` and `load_artifact` instead.
|
||||
|
||||
## Dashboards and panels
|
||||
|
||||
| Method | Path | What |
|
||||
| Method | Path | Action |
|
||||
|---|---|---|
|
||||
| `GET` | `/dashboards/` | every dashboard, without its contents |
|
||||
| `GET` | `/dashboards/{name}?draft=true` | the published document, or the editor's copy |
|
||||
| `POST` | `/dashboards/{name}` | create one |
|
||||
| `POST` | `/dashboards/from-flow/{flow}` | build `{flow}_results` from a batch flow's declared ports — a chart per streamed metric, a stat per output. 409 if it already exists |
|
||||
| `GET` | `/dashboards/` | list dashboards |
|
||||
| `GET` | `/dashboards/{name}?draft=true` | read a draft; omit the query for published content |
|
||||
| `POST` | `/dashboards/{name}` | create |
|
||||
| `POST` | `/dashboards/from-flow/{flow}` | create a batch flow's results dashboard |
|
||||
| `PUT` | `/dashboards/{name}` | save a draft |
|
||||
| `POST` | `/dashboards/{name}/publish` · `/discard` · `/rename` | |
|
||||
| `GET` `PUT` | `/panels/` | which device shows which dashboards |
|
||||
| `POST` | `/panels/pair` | start a pairing |
|
||||
| `GET` | `/panels/pair/{code}` | what is holding a code |
|
||||
| `POST` | `/panels/{id}/unpair` | drop this panel's credential, keep the panel |
|
||||
| `POST` | `/dashboards/{name}/publish`, `/dashboards/{name}/discard`, `/dashboards/{name}/rename` | publish, discard, or rename |
|
||||
| `GET`, `PUT` | `/panels/` | read or update panel assignments |
|
||||
| `POST` | `/panels/pair` | start device pairing |
|
||||
| `GET` | `/panels/pair/{code}` | poll pairing with its `secret` |
|
||||
| `POST` | `/panels/{id}/pair` | approve the pairing |
|
||||
| `POST` | `/panels/{id}/unpair` | revoke the panel credential |
|
||||
|
||||
## Secrets, modules, alerts
|
||||
## Secrets, modules, and alerts
|
||||
|
||||
```sh
|
||||
curl -s $FLUKSIO/secrets/ -H "Authorization: Bearer $TOKEN" # names only
|
||||
curl -X PUT $FLUKSIO/secrets/influx-token -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"value": "..."}'
|
||||
|
||||
curl -s $FLUKSIO/modules/ -H "Authorization: Bearer $TOKEN"
|
||||
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"requirements": "numpy>=2\n"}'
|
||||
|
||||
curl -s $FLUKSIO/alerts/config -H "Authorization: Bearer $TOKEN"
|
||||
curl -X POST $FLUKSIO/alerts/test/my-phone -H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
|
||||
Secrets are write-only over the API: you can list names and set values, never
|
||||
read one back. Secrets and `POST /modules/apply` are a superuser's, and so is
|
||||
what a connector keeps across restarts: `GET` or `DELETE
|
||||
/connectors/{node_type}/state`, or `DELETE /connectors/{node_type}/state/{key}`
|
||||
for one key (see [the connector contract](../reference/connector-contract.md#keeping-state-across-restarts)).
|
||||
|
||||
## Observability
|
||||
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| `/observability/summary` | engine health — always 200, degraded or not — and the `version` it is running |
|
||||
| `/observability/timeseries` | executions and failures over a window |
|
||||
| `/observability/flows` | per-flow rollups with a 60-slice trend |
|
||||
| `/observability/runs` | recent cascades, with `?flow=`, `?since=`, `?until=` |
|
||||
| `/observability/events?kind=failure\|audit` | what went wrong, or who changed what; narrows by `?flow=`, `?run=`, `?since=`, `?until=` |
|
||||
| `/observability/dead-letter` | work the engine gave up on |
|
||||
|
||||
`GET /utils/health/` is the deep health check the container probe uses: it
|
||||
fails when the event loop is wedged or the state backend is gone, not just when
|
||||
the process is dead.
|
||||
|
||||
## Workers
|
||||
|
||||
| Method | Path | What |
|
||||
| Method | Path | Action |
|
||||
|---|---|---|
|
||||
| `GET` | `/workers` | what is attached, its labels and how busy it is |
|
||||
| `POST` | `/workers/tokens` | mint a worker credential with its name and labels (superuser; shown once) |
|
||||
| `GET` | `/workers/tokens` | the worker credentials minted, without the tokens (superuser) |
|
||||
| `POST` | `/workers/tokens/{jti}/revoke` | refuse one from now on and drop a worker attached with it (superuser) |
|
||||
| `GET` | `/workers/runtime` | the node runner's source, for a host without pip |
|
||||
| `WS` | `/workers/attach` | where a worker dials in, with `Authorization: Bearer <worker token>` |
|
||||
| `GET` | `/secrets/` | list names only |
|
||||
| `PUT` | `/secrets/{name}` | set `{"value": "..."}` |
|
||||
| `GET` | `/modules/` | read installed packages and the manifest |
|
||||
| `POST` | `/modules/apply` | replace the manifest with `{"requirements": "numpy>=2\n"}` |
|
||||
| `GET`, `PUT` | `/alerts/config` | read or update alert configuration |
|
||||
| `POST` | `/alerts/test/{channel}` | test a channel |
|
||||
|
||||
See [Remote workers](workers.md).
|
||||
Secrets and package changes require a superuser. Applying a manifest replaces
|
||||
the whole managed environment's package list. Adopted environments are read-only
|
||||
here; install their packages yourself.
|
||||
|
||||
Connector state keys can also be listed and cleared by a superuser; see
|
||||
[the connector contract](../reference/connector-contract.md#keeping-state-across-restarts).
|
||||
|
||||
## Observability and workers
|
||||
|
||||
| Method | Path | Action |
|
||||
|---|---|---|
|
||||
| `GET` | `/observability/summary` | health and engine version; always returns `200` |
|
||||
| `GET` | `/observability/timeseries`, `/observability/flows` | execution metrics or flow rollups |
|
||||
| `GET` | `/observability/runs`, `/observability/events`, `/observability/dead-letter` | cascades, events, or abandoned work |
|
||||
| `GET` | `/utils/health/` | deep health check; fails on loop or state-backend problems |
|
||||
| `GET` | `/workers`, `/workers/resources` | attached workers and resource availability |
|
||||
| `GET`, `POST` | `/workers/tokens` | list or mint worker credentials; superuser only |
|
||||
| `POST` | `/workers/tokens/{jti}/revoke` | revoke a worker |
|
||||
| `GET` | `/workers/runtime` | read node-runner source |
|
||||
| `WS` | `/workers/attach` | attach with a worker token |
|
||||
| `GET`, `POST` | `/flavors/` | list or create resource sizes |
|
||||
| `PATCH`, `DELETE` | `/flavors/{name}` | update or delete a size |
|
||||
|
||||
Changing flavors requires a superuser. See [Remote workers](workers.md).
|
||||
|
||||
## Live events
|
||||
|
||||
```text
|
||||
ws://127.0.0.1:8000/api/v1/flows/ws
|
||||
```
|
||||
Connect to `ws://127.0.0.1:8000/api/v1/flows/ws` with an
|
||||
`Authorization: Bearer <access-token>` header.
|
||||
|
||||
A client that can set headers sends `Authorization: Bearer <access token>` on
|
||||
the handshake. A browser cannot, so it first calls `POST /flows/ws-ticket` and
|
||||
opens `/flows/ws?ticket=<ticket>`: a ticket is good for one connect, within 30
|
||||
seconds, so the URL that access logs keep is worth nothing afterwards.
|
||||
`?token=` is still accepted for one more release. The socket sends a snapshot
|
||||
on connect and then every engine event: node executions, values published,
|
||||
health changes, run started and finished. This is what the canvas and the
|
||||
dashboards draw from.
|
||||
Browsers first call `POST /flows/ws-ticket`, then connect with
|
||||
`?ticket=<ticket>`. Tickets are single-use and expire after 30 seconds.
|
||||
The socket sends a snapshot followed by node, value, health, and run events.
|
||||
|
||||
## Errors
|
||||
## Errors and client generation
|
||||
|
||||
| Code | Means |
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| `400` | the request was malformed, or the node refused it with a message |
|
||||
| `401` / `403` | not signed in, or not allowed |
|
||||
| `404` | no such flow, dashboard, run or message |
|
||||
| `409` | someone else saved first — the body carries `current_version` |
|
||||
| `422` | a parameter, port or binding did not typecheck |
|
||||
| `503` | that subsystem is not available on this instance |
|
||||
| `400` | malformed request or refused operation |
|
||||
| `401` / `403` | missing authentication or permission |
|
||||
| `404` | resource not found |
|
||||
| `409` | version conflict, duplicate, or resource still in use |
|
||||
| `413` | artifact upload exceeds the configured limit |
|
||||
| `422` | invalid parameters, ports, or bindings |
|
||||
| `503` | subsystem unavailable |
|
||||
|
||||
A 409 on a save or a publish is not an error to retry blindly: it means the
|
||||
stored version moved past the one you were editing. Re-read, merge, save again.
|
||||
For a version conflict, read `current_version`, reload, merge your edits, and
|
||||
save again. Other `409` responses describe their own conflict.
|
||||
|
||||
## Generating a client
|
||||
|
||||
The frontend's TypeScript client is generated from the OpenAPI schema
|
||||
(`make generate-client`). Any OpenAPI generator will do the same for your
|
||||
language: point it at `/api/v1/openapi.json` on a non-production instance,
|
||||
from the machine it runs on.
|
||||
`make generate-client` regenerates the frontend SDK. To generate a client in
|
||||
another language, use the instance's `/api/v1/openapi.json` schema.
|
||||
|
||||
+221
-468
@@ -1,467 +1,261 @@
|
||||
# The `fluksio` command
|
||||
|
||||
Install on Linux or macOS with Python 3.12 or newer:
|
||||
|
||||
```sh
|
||||
pip install fluksio
|
||||
```
|
||||
|
||||
Installs the engine and the `fluksio` command. Python 3.12 or newer, Linux or
|
||||
macOS.
|
||||
|
||||
The MQTT and InfluxDB connectors, outbound mail and error reporting are
|
||||
`pip install 'fluksio[server]'`: a deployment talking to devices wants them,
|
||||
and a laptop waiting on them to install does not. Everything else, the engine
|
||||
and every python node included, is in the plain install; a node type whose
|
||||
library is missing says which extra to add when one is actually built.
|
||||
|
||||
There is a second, smaller distribution, `fluksio-worker`, for a machine that
|
||||
should only *run nodes* for an engine elsewhere. It has none of the engine in
|
||||
it. See [Remote workers](workers.md).
|
||||
|
||||
The command is two things at once: `serve`, `enroll` and `worker` *are* an
|
||||
instance, while `login`, `sync`, `run`, `runs`, `artifacts`, `sweep` and
|
||||
`status` talk to one that may be anywhere.
|
||||
|
||||
## Where an instance lives
|
||||
|
||||
`.fluksio` beside your code, found the way `.git` is: from the working
|
||||
directory, or any directory above it. Two repositories on one machine are
|
||||
therefore two engines, with their own flows, runs and token. `fluksio serve`
|
||||
makes one where there is none, and it ignores itself from within with a
|
||||
`.gitignore` of `*`, so a database and a credential cannot be committed by
|
||||
accident.
|
||||
|
||||
`--global` uses `~/.fluksio` instead, shared by every directory. `--data-dir`
|
||||
(or `FLUKSIO_HOME`) names any directory outright and wins over both.
|
||||
|
||||
## `fluksio serve`
|
||||
|
||||
Runs the engine.
|
||||
|
||||
```sh
|
||||
fluksio serve
|
||||
```
|
||||
|
||||
On the first start it creates an admin account and prints its password **once**.
|
||||
Nothing else has to be running: no database server, no message broker, no
|
||||
Docker.
|
||||
For the browser dashboard, sign up at [fluksio.com](https://fluksio.com), add
|
||||
an instance, then run `fluksio enroll <code>` in the same directory.
|
||||
|
||||
At a terminal this opens a dashboard with the engine running under it; see
|
||||
[below](#the-dashboard). `--plain` prints the log stream instead, which is
|
||||
also what happens with no terminal: in a container, under systemd, or in CI.
|
||||
Use `fluksio --help` or `fluksio COMMAND --help` for command options.
|
||||
`pip install 'fluksio[server]'` adds MQTT, InfluxDB, outbound mail and error
|
||||
reporting. A machine that only executes remote nodes can install
|
||||
[`fluksio-worker`](workers.md).
|
||||
|
||||
The default port moves out of the way when something already has it (8001,
|
||||
8002, and so on) and says which one it took; the URL written to
|
||||
`client.json` is the one it is actually on. A port you *asked* for is never
|
||||
moved off: `--port 9000` on a taken 9000 fails, because something else is
|
||||
there and you named it. It fails before `client.json` is written, so every
|
||||
client keeps talking to the engine that is actually up.
|
||||
## Where an instance lives
|
||||
|
||||
What it will *not* do is start a second engine for the same instance. Before
|
||||
anything else, `serve` looks for an engine already serving this directory —
|
||||
the pidfile beside the data says where, and a token this directory's key
|
||||
signed says whether what answers there is ours — and stops if it finds one,
|
||||
whatever port the second was asked for. One SQLite database wants one engine,
|
||||
and the second would repoint `client.json` at a port that dies with it.
|
||||
Another instance's Fluksio on the port is named, and the move happens as usual.
|
||||
By default, Fluksio finds the nearest `.fluksio/` directory by walking up from
|
||||
your working directory. `serve` creates one in the current directory if needed.
|
||||
Separate project directories can therefore hold separate instances.
|
||||
|
||||
| Option | Default | What it does |
|
||||
|---|---|---|
|
||||
| `--data-dir PATH` | `./.fluksio` (or `$FLUKSIO_HOME`) | where this instance keeps everything |
|
||||
| `--host HOST` | `127.0.0.1` | what to bind |
|
||||
| `--port PORT` | `8000`, or the next free one | what to listen on |
|
||||
| `--plain` | off at a terminal | the log stream rather than the dashboard |
|
||||
| `--log-level LEVEL` | `info` | uvicorn's log level |
|
||||
| `--admin-email ADDR` | `admin@example.com` | the account created on first run |
|
||||
| `--admin-password PW` | generated | set it instead of having one generated |
|
||||
| `--enroll CODE` | — | pair with a portal as part of coming up |
|
||||
| `--portal URL` | — | the portal `--enroll` redeems at |
|
||||
| `--max-runs N` | 4 | batch runs driven at once (`FLOW_MAX_RUNS`) |
|
||||
| `--max-cascades N` | 4 | cascades in flight at once (`FLOW_MAX_CASCADES`) |
|
||||
| `--max-workers N` | 4 | python worker processes (`FLOW_MAX_WORKERS`) |
|
||||
| `--gpus N` | counted | GPUs on this machine a node may be given (`FLOW_GPUS`) |
|
||||
| `--ram MB` | worked out | memory on this machine a node may be given (`FLOW_RAM`) |
|
||||
| `--worker-max-rss MB` | 0 (never) | retire a warm worker holding more when its node returns (`FLOW_WORKER_MAX_RSS_MB`) |
|
||||
`--global` selects `~/.fluksio`. `--data-dir PATH` or `FLUKSIO_HOME`
|
||||
selects a directory explicitly. The data directory contains a `.gitignore`
|
||||
that excludes its database, credentials and other contents from your project.
|
||||
|
||||
Cards are counted from NVIDIA's device nodes (`/dev/nvidia0`, `/dev/nvidia1`, …),
|
||||
which asks no vendor tool and so keeps the one dependency from becoming two.
|
||||
Anything they do not cover — another vendor, or a card this process cannot see
|
||||
— reports none until `--gpus` says otherwise, and a node asking for one is then
|
||||
clamped to zero and runs alongside every other. `--gpus 1` is what serialises
|
||||
them. Passing a number always wins over the count.
|
||||
## `fluksio serve`
|
||||
|
||||
`--enroll` with `--portal` is the one-command setup: it pairs before the engine
|
||||
starts, so the connection is dialled as part of coming up rather than needing a
|
||||
restart. It is skipped if the instance is already enrolled.
|
||||
Starts the API and flow engine. No external database, broker or Docker is
|
||||
needed for the default setup.
|
||||
|
||||
!!! warning "One process"
|
||||
|
||||
`fluksio serve` holds the flow engine. A second one is a *second engine*:
|
||||
duplicated subscriptions, duplicated cron ticks, two webhooks answering the
|
||||
same path. Run one, and distribute work with
|
||||
[workers](workers.md) instead.
|
||||
|
||||
!!! note "`$HOME` on a cluster"
|
||||
|
||||
A login node's home directory is often NFS, where SQLite's write-ahead log
|
||||
does not work, since the database would be locked or corrupt. `fluksio serve`
|
||||
warns when it notices; point `--data-dir` at local disk.
|
||||
|
||||
### What it prints
|
||||
|
||||
```text
|
||||
Created the admin account admin@example.com
|
||||
password: example-password
|
||||
Shown once. Change it from the dashboard.
|
||||
Fluksio 0.1.0 — data in /home/you/.fluksio
|
||||
API http://127.0.0.1:8000/api/v1
|
||||
No portal. Pair this instance with:
|
||||
fluksio enroll <code>
|
||||
```sh
|
||||
fluksio serve
|
||||
fluksio serve --plain --port 9000
|
||||
```
|
||||
|
||||
An enrolled instance says which portal it is on instead, and notes that the
|
||||
dashboard is served from there rather than here.
|
||||
The first start prints an admin password once. Local CLI commands receive a
|
||||
saved token automatically; `login` is for connecting to another engine.
|
||||
|
||||
The default port is 8000, advancing to the next free port if necessary.
|
||||
An explicit `--port` fails when occupied. The startup banner and saved client
|
||||
configuration show the selected URL.
|
||||
|
||||
Run one engine per instance. At a terminal, a second `serve` reattaches to
|
||||
that instance's running engine; plain mode refuses to start a duplicate.
|
||||
|
||||
| Option | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `--data-dir PATH` | Nearest `.fluksio`, or a new one here | Instance storage |
|
||||
| `--global` | Off | Use `~/.fluksio` |
|
||||
| `--host HOST` | `127.0.0.1` | Bind address |
|
||||
| `--port PORT` | `8000`, or next free | Listening port |
|
||||
| `--plain` | Off at a terminal | Print logs instead of opening the terminal dashboard |
|
||||
| `--log-level LEVEL` | `info` | Uvicorn log level |
|
||||
| `--admin-email ADDR` | `admin@example.com` | First admin account |
|
||||
| `--admin-password PW` | Generated | First admin password |
|
||||
| `--enroll CODE` | Unset | Enroll before starting, if not already paired |
|
||||
| `--portal URL` | `https://hub.fluksio.com` | Portal used by `--enroll` |
|
||||
| `--max-runs N` | `4` | Concurrent batch runs |
|
||||
| `--max-cascades N` | `4` | Concurrent live cascades |
|
||||
| `--max-workers N` | `4` | Local Python worker processes |
|
||||
| `--gpus N` | Auto-detected | Available local GPUs |
|
||||
| `--ram MB` | Auto-detected | Available local memory |
|
||||
| `--worker-max-rss MB` | `0` (disabled) | Retire a worker above this resident memory after its call |
|
||||
|
||||
Resource defaults can also come from the corresponding `FLOW_*` settings;
|
||||
see [Configuration](../reference/configuration.md). GPU detection uses
|
||||
NVIDIA device nodes. Set `--gpus` when your hardware is not detected.
|
||||
|
||||
Use local disk for the data directory on a cluster: SQLite's write-ahead log
|
||||
does not support NFS.
|
||||
|
||||
### The dashboard
|
||||
|
||||
At a terminal, `serve` opens three tabs: **Overview** is the health block and
|
||||
the flows, **Runs** is the history, and **Logs** is the engine's own output.
|
||||
At a terminal, `serve` opens **Overview**, **Runs** and **Logs**:
|
||||
|
||||
| Key | What it does |
|
||||
| Key | Action |
|
||||
|---|---|
|
||||
| `1` `2` `3` | the Overview, the Runs and the Logs |
|
||||
| `q` | close the dashboard. **The engine keeps running**, and the pid is printed |
|
||||
| `s` | stop the engine, or start it again |
|
||||
| `r` | restart it |
|
||||
| `e` | pair with a portal, without leaving the screen |
|
||||
| `1`, `2`, `3` | Switch tabs |
|
||||
| `q` | Close the dashboard; the engine keeps running |
|
||||
| `s` | Stop or start the engine |
|
||||
| `r` | Restart the engine |
|
||||
| `e` | Enroll with a portal |
|
||||
|
||||
On the Runs tab the toolbar carries what a run is for:
|
||||
On **Runs**, `space` selects runs, `enter` opens a chart or comparison,
|
||||
`c` cancels a run, and `a` lists or downloads artifacts. Compare curves by
|
||||
step, elapsed time or another metric.
|
||||
|
||||
| Key | What it does |
|
||||
|---|---|
|
||||
| `space` | tick the run under the cursor into a comparison |
|
||||
| `enter` | compare the ticked runs, or draw the one under the cursor |
|
||||
| `c` | cancel the run the cursor is on |
|
||||
| `a` | list what the run left behind, and fetch it |
|
||||
|
||||
The engine is a child process rather than a thread, which is what makes those
|
||||
possible, and what makes `q` a way out of the screen rather than a way to
|
||||
stop the engine. Running `fluksio serve` again reattaches to it.
|
||||
|
||||
Its output goes to `serve.log` in the data directory rather than down a pipe,
|
||||
which is what lets the screen be closed while the engine keeps running — a
|
||||
pipe with nobody reading it breaks the next line the engine writes, and a
|
||||
node's `print` is one of those. It also means the Logs tab has the output of
|
||||
an engine this screen only adopted, and the scrollback of the one before it.
|
||||
The engine cuts the file back to nothing once it passes 5 MB, so the bound
|
||||
holds whether or not a screen is open.
|
||||
|
||||
The screen takes its colours from the terminal rather than painting its own,
|
||||
so it sits inside a light profile as readily as a dark one.
|
||||
|
||||
An engine started elsewhere is adopted rather than duplicated, and can be
|
||||
stopped from here only when it is this instance's own: both the pidfile
|
||||
beside the data and a token this directory's key signed have to agree. Another
|
||||
instance's engine is named and left alone.
|
||||
|
||||
The screen subscribes to the engine's event bus over the same websocket a
|
||||
browser uses, so a run appears the moment it starts. It reconnects on its own,
|
||||
and an engine stopped from this screen is a normal state rather than an error.
|
||||
|
||||
#### Comparing runs
|
||||
|
||||
`enter` opens a comparison of the ticked runs: one metric's curve for each,
|
||||
over a table of what differs between them.
|
||||
|
||||
The curves are drawn in braille, five to a chart, in five distinct colours: a
|
||||
departure from the browser's chart palette, where the five series
|
||||
step one hue by lightness. Hue is what survives a terminal. Every curve is
|
||||
named in the legend either way.
|
||||
|
||||
Two pickers sit above the chart. The first is the metric, named as the run
|
||||
records it (`train.loss` rather than `loss`, since a name is qualified by the
|
||||
node that published it). The second is what it is plotted against: the step,
|
||||
`time (s)` measured from each run's own first reading so runs started hours
|
||||
apart lie on top of each other, or another metric of the same runs.
|
||||
|
||||
The table below keeps only what actually differs (the inputs whose values are
|
||||
not the same in every run, and the seed when it varies) beside each run's
|
||||
status, duration and final numbers. A parameter every run shared is not a
|
||||
column worth scanning.
|
||||
|
||||
A run that is still going has its curve re-read once a second, and the whole
|
||||
comparison refreshes when the engine says a run finished.
|
||||
Run `fluksio serve` again to reopen the dashboard. Its engine writes to
|
||||
`serve.log` in the data directory; the log is truncated after 5 MB.
|
||||
Without a terminal, or with `--plain`, the command prints logs directly.
|
||||
|
||||
## `fluksio enroll`
|
||||
|
||||
Pairs an existing instance with a portal.
|
||||
Connects this instance to the portal:
|
||||
|
||||
```sh
|
||||
fluksio enroll ABCD-1234
|
||||
```
|
||||
|
||||
| Option | What it does |
|
||||
Get a code from **Instances → Add instance** in the portal. Run the command
|
||||
from the instance's project directory, or select it explicitly.
|
||||
|
||||
| Option | Purpose |
|
||||
|---|---|
|
||||
| `--portal URL` | a portal of your own, instead of `https://hub.fluksio.com` |
|
||||
| `--as EMAIL` | the local account a portal session arrives as |
|
||||
| `--data-dir PATH` | which instance, if not the one this directory is in |
|
||||
| `--portal URL` | Use another portal instead of `https://hub.fluksio.com` |
|
||||
| `--as EMAIL` | Choose the local superuser represented by portal sessions |
|
||||
| `--data-dir PATH` | Select instance storage |
|
||||
| `--global` | Select `~/.fluksio` |
|
||||
|
||||
Get the code from the portal under **Instances → Add instance**. It is
|
||||
single-use and expires in fifteen minutes. `--as` matters when the instance
|
||||
has several superusers. Without it, enrolment refuses rather than guessing.
|
||||
With multiple local superusers, `--as` is required. A new code replaces an
|
||||
existing enrollment only after the portal accepts it. A running engine picks
|
||||
up the change and reconnects automatically.
|
||||
|
||||
An instance that is already paired is re-paired: a new claim code replaces the
|
||||
connection, at the same portal or another one, and a running engine drops the
|
||||
old tunnel within a few seconds. The code is redeemed before anything is
|
||||
written, so one the portal rejects leaves the connection as it was. Moving to
|
||||
a different portal clears the local accounts' mappings to the old one, since
|
||||
nothing in a mapping says which portal issued it — the people who had access
|
||||
are admitted again from the new portal.
|
||||
|
||||
Afterwards, `fluksio serve` dials the portal as it comes up, and keeps dialling:
|
||||
a portal that restarts, a wifi that changes, a laptop that suspends and wakes
|
||||
somewhere else all end the same connection, and the link is put back up without
|
||||
anybody noticing. A connection that stood up and then dropped is retried at
|
||||
once; one that never stood up waits a little longer each time, up to half a
|
||||
minute. See [Accounts and the portal](../interface/portal.md).
|
||||
The instance keeps an outbound connection to the portal and reconnects after
|
||||
network interruptions. See [Accounts and the portal](../interface/portal.md).
|
||||
|
||||
## `fluksio worker`
|
||||
|
||||
Runs nodes for an engine elsewhere. Everything after `worker` belongs to the
|
||||
agent's own parser: it is the same program `fluksio-worker` installs, so the
|
||||
two are interchangeable:
|
||||
Starts the same agent as the standalone `fluksio-worker` command:
|
||||
|
||||
```sh
|
||||
fluksio worker --url wss://api.${DOMAIN}/api/v1/workers/attach --labels gpu
|
||||
FLUKSIO_WORKER_TOKEN=your-token fluksio worker \
|
||||
--url wss://api.example.com/api/v1/workers/attach --labels gpu
|
||||
```
|
||||
|
||||
See [Remote workers](workers.md).
|
||||
Create the worker token on the engine first. See [Remote workers](workers.md)
|
||||
for token creation, interpreter selection and labels.
|
||||
|
||||
## Talking to an engine
|
||||
|
||||
The commands below are the client half: they run wherever you work, and
|
||||
address an engine over its API rather than being one, except under `--local`,
|
||||
which boots one inside the command instead.
|
||||
Client commands use the nearest `.fluksio/client.json`, or the global
|
||||
configuration when there is no project instance. Legacy client configuration
|
||||
is a fallback.
|
||||
`FLUKSIO_URL` and `FLUKSIO_TOKEN` override saved values; command options
|
||||
`--url` and `--token` override both.
|
||||
|
||||
### `fluksio login`
|
||||
|
||||
```sh
|
||||
fluksio login --url https://api.${DOMAIN}
|
||||
fluksio login --url https://api.example.com
|
||||
```
|
||||
|
||||
For an engine somewhere *else*. One you started yourself needs no login:
|
||||
`fluksio serve` writes the token as it comes up and says where it put it.
|
||||
|
||||
The token goes in this project's `.fluksio/client.json`, or with `--global` in
|
||||
`~/.fluksio/client.json`. Every command below reads it from there, nearest
|
||||
first walking up from the working directory, or from `FLUKSIO_URL` and
|
||||
`FLUKSIO_TOKEN`, or from its own `--url` and `--token`. A token an older
|
||||
version wrote to `~/.config/fluksio/client.json` is still read.
|
||||
Prompts for credentials and saves an API token for a remote engine.
|
||||
`--global` saves it in `~/.fluksio` instead of this project. Your locally
|
||||
started engine already saves a token.
|
||||
|
||||
### `fluksio sync`
|
||||
|
||||
```sh
|
||||
fluksio sync [PATH_OR_MODULE ...] # default: the current directory
|
||||
fluksio sync study
|
||||
fluksio sync examples/myresearch
|
||||
fluksio sync --dry-run study
|
||||
```
|
||||
|
||||
Imports what you name, collects the flows the decorators declared, and uploads
|
||||
each one with a generated import shim per node. A directory that is a package
|
||||
is walked; a dotted name is imported as it stands; nothing is loaded from a
|
||||
file path, because the shim has to import the same way.
|
||||
Imports modules, packages or directories and uploads the declared flows.
|
||||
With no target, it searches the current directory. Directory searches skip
|
||||
hidden folders, virtual environments, `__pycache__` and `node_modules`.
|
||||
File paths are not targets: use `study`, not `study.py`.
|
||||
|
||||
A plain directory is walked all the way down, so one folder per study
|
||||
(`fluksio sync dev` over `dev/s1_baseline/study.py`) needs no naming. Hidden
|
||||
directories, `__pycache__`, `node_modules` and virtualenvs are left alone.
|
||||
The generated node bodies import your functions. Keep the project available
|
||||
to each machine running them; sync does not copy your package to a remote
|
||||
worker. Uploading source requires a superuser token.
|
||||
|
||||
Each file is imported under the name its path spells beneath the directory
|
||||
being synced, so `dev/s1/study.py` and `dev/s2/study.py` are `s1.study` and
|
||||
`s2.study` and a `study.py` per study collides with nothing. No `__init__.py`
|
||||
is needed, since the directories in between are namespace packages, which leaves a
|
||||
bare `from study import ...` in a test beside it working. A file at the top of
|
||||
what is synced keeps its plain name.
|
||||
|
||||
| Flag | What it does |
|
||||
| Flag | Purpose |
|
||||
|---|---|
|
||||
| `--dry-run` | print the flow documents and shims, upload nothing |
|
||||
| `--no-publish` | leave the upload as a draft |
|
||||
| `--force` | overwrite a flow, or a node body, that was edited on the canvas |
|
||||
| `--dry-run` | Print flow documents and node bodies without uploading |
|
||||
| `--no-publish` | Save changes as drafts |
|
||||
| `--force` | Overwrite canvas edits that conflict with your local definitions |
|
||||
|
||||
Every sync retires the engine's workers, including one that had nothing to
|
||||
upload: a worker holds your package in memory, so an edit to it is invisible
|
||||
until the process goes.
|
||||
|
||||
It also records, per node, which of your modules that node's function imports
|
||||
its way to, and what they hash to. That is what the
|
||||
[stage cache](../concepts/runs.md#stage-caching) keys on, so editing a
|
||||
helper a node calls into is reported as that node changing (`train: updated
|
||||
(flow, fit)`) and re-runs it, while editing something the node never reaches
|
||||
is left alone.
|
||||
|
||||
Uploading node code takes a superuser's token; anyone else's sync is refused.
|
||||
|
||||
A sync that changed nothing says `unchanged`, which is the answer worth
|
||||
having. `published` and `draft` are said only when there was something to
|
||||
publish or a draft was genuinely left behind. An engine too old to store what
|
||||
a node's code reaches says so in a line naming both versions; until it is
|
||||
upgraded its cache is keyed on the whole repository, as it was before. See
|
||||
[Getting started: data science](../getting-started/data-science.md).
|
||||
Sync records source digests for [stage caching](../concepts/runs.md#stage-caching)
|
||||
and refreshes local workers, so edited imports are picked up. An unchanged
|
||||
flow reports `unchanged`. See the
|
||||
[data science guide](../getting-started/data-science.md) for a complete example.
|
||||
|
||||
### `fluksio run`
|
||||
|
||||
```sh
|
||||
fluksio run train --lr 0.05 --seed 7 [--wait]
|
||||
fluksio run train --lr 0.05 --epochs 20 --follow
|
||||
```
|
||||
|
||||
Syncs the flow, then submits a run, so the command after an edit is this one
|
||||
and nothing else, from the repository root as readily as from the study's own
|
||||
folder. What it imports is the module that declared the flow, as the last
|
||||
`sync` recorded it, under the same module name. Importing a module runs it, so
|
||||
the other scripts in the repository are left alone.
|
||||
Syncs local code, then submits a run. Flags such as `--lr` name the flow's
|
||||
declared inputs and are converted to their declared types. Unknown inputs are
|
||||
rejected. Use `--param name=a,b` with `sweep` for multiple values.
|
||||
|
||||
A flow the engine has no such record for (never synced, drawn on the canvas,
|
||||
or on an engine older than the record) syncs the working directory *and
|
||||
everything under it* instead, skipping whatever the repository's `.gitignore`
|
||||
covers. A study there that will not import is a warning rather than a stopped
|
||||
run. `--sync dev/s1_baseline` (repeatable) names what to sync outright, and
|
||||
`--no-sync` skips it entirely.
|
||||
After the first sync, `run` imports the module that declared the flow. If no
|
||||
declaration is recorded, it searches the working directory and skips paths
|
||||
covered by `.gitignore`. Import errors in that fallback search produce
|
||||
warnings. Use `--sync PATH` to choose a target explicitly.
|
||||
|
||||
Flags that are not its own are the flow's inputs, typed by what the flow
|
||||
declares them as, so a name the flow has not got is refused by name, and
|
||||
`--param lr=0.002` is told that one value is `--lr 0.002` and several is a
|
||||
[sweep](#fluksio-sweep). `--wait` blocks until the run
|
||||
finishes and exits non-zero if it failed. `--follow` waits as well, and prints
|
||||
the numbers the run reports as they arrive:
|
||||
| Flag | Purpose |
|
||||
|---|---|
|
||||
| `--wait` | Wait for completion; exit non-zero if the run fails |
|
||||
| `--follow` | Wait and print emitted metrics |
|
||||
| `--timeout SECONDS` | Stop waiting after this long; leave the run active |
|
||||
| `--seed N` | Record the seed and fill a declared input named `seed` |
|
||||
| `--defaults` | Use declared input defaults without prompting |
|
||||
| `--sync PATH` | Choose what to sync; repeat for multiple targets |
|
||||
| `--no-sync` | Run the version already on the engine |
|
||||
| `--no-cache` | Execute every node |
|
||||
| `--local` | Start an engine in this command and wait for the result |
|
||||
|
||||
```text
|
||||
train.loss[14] = 3.40295e-06
|
||||
```
|
||||
With no parameters at a terminal, the command prompts for each input.
|
||||
Press Enter to keep its default. Scripted commands do not prompt.
|
||||
Ctrl-C while waiting cancels the run and exits with status 130.
|
||||
|
||||
Ctrl-C while either is waiting cancels the run on the engine rather than only
|
||||
stopping the watching, and exits 130. `--timeout SECONDS` gives up *waiting*
|
||||
after that long and leaves the run going; it is nothing to do with a node's own
|
||||
timeout.
|
||||
A seed does not initialize your libraries' random number generators. Declare
|
||||
a `seed` input and pass it to your code if it controls randomness.
|
||||
|
||||
`--seed N` is the experiment's seed, and it does three things. It is recorded
|
||||
on the run, so what a result came from is answerable later. It goes into the
|
||||
digest that identifies a run's inputs, so two runs of one configuration with
|
||||
different seeds are different runs rather than a cache hit. And if the flow
|
||||
declares an input named `seed`, that is what fills it, so the number the run
|
||||
is labelled with is the number your code actually drew from, instead of merely
|
||||
looking like it. A flow that declares no such input still records it, and
|
||||
nothing reads it. Sweep over seeds with `--param seed=1,2,3`.
|
||||
|
||||
Any input takes what a previous run produced, named rather than typed out: a
|
||||
checkpoint, but equally a config object nobody wants to paste into a shell:
|
||||
Inputs can reference earlier results:
|
||||
|
||||
```sh
|
||||
fluksio run evaluate --dataset @run:1758042000123-9f2ab41c.dataset
|
||||
fluksio run evaluate --dataset sha256:6dd1f0…
|
||||
fluksio run train --meta @run:1758042000123-9f2ab41c.dataset_meta
|
||||
fluksio run evaluate --dataset @run:<run-id>.dataset
|
||||
fluksio run evaluate --dataset sha256:<digest>
|
||||
```
|
||||
|
||||
`@run:<id>.<output>` is whatever that run's output was, whole and with its own
|
||||
type; a bare digest is the content itself, resolved into a reference. Both
|
||||
spellings are reserved on every input, `str` included, so an input that has to
|
||||
carry one of them literally cannot. Passing the value as JSON still works and
|
||||
is what a script that already holds one does, the same thing
|
||||
`flow.submit(dataset=run.result["dataset"])` does from Python.
|
||||
`@run:<id>.<output>` resolves a run output; `sha256:<digest>` resolves
|
||||
stored bytes into an artifact reference.
|
||||
|
||||
Run a flow with no parameters at a terminal and it asks for them, one line per
|
||||
declared input, with the declared value in brackets:
|
||||
|
||||
```text
|
||||
lr (float) [0.05]: 0.01
|
||||
epochs (int) [10]:
|
||||
dataset (artifact): @run:1758042000123-9f2ab41c.dataset
|
||||
```
|
||||
|
||||
Enter keeps what is in brackets, so pressing it through the lot runs the
|
||||
defaults. Nothing changes for a scripted run: passing any parameter, or piping
|
||||
the command, skips the questions, and `--defaults` skips them explicitly.
|
||||
|
||||
`--no-sync` runs what is already on the engine. Worth it in a tight loop where
|
||||
you know nothing changed, since syncing retires the workers and the next call
|
||||
pays its imports again. A directory that declares no flows syncs nothing and
|
||||
says nothing: a flow drawn on the canvas is run the same way.
|
||||
|
||||
`--no-cache` executes every node, including one an earlier run already
|
||||
answered. See [Stage caching](../concepts/runs.md#stage-caching).
|
||||
|
||||
`--local` boots the engine inside this process instead of talking to a served
|
||||
one, so there is no `fluksio serve` terminal to keep open. It is the same
|
||||
instance either way (the same `.fluksio`, the same database, artifacts and run
|
||||
history) so a run made this way and a run made through a served
|
||||
engine cache against each other. It always waits, because the engine it starts
|
||||
lives exactly as long as the command. Starting one costs a few seconds of
|
||||
worker pool and module reconcile, against the ~15 ms of submitting to an
|
||||
engine that is already up: `--local` is for "I just want to run it", not for a
|
||||
loop you are iterating in.
|
||||
`--local` uses the same data directory, history and artifacts as `serve`.
|
||||
It starts and stops the engine for each command, so keep `serve` running
|
||||
when iterating frequently.
|
||||
|
||||
### `fluksio status`
|
||||
|
||||
```sh
|
||||
fluksio status [--watch]
|
||||
fluksio status
|
||||
fluksio status --watch
|
||||
```
|
||||
|
||||
The home screen's top half in a terminal: whether the engine is healthy and
|
||||
what is wrong if not, whether it is paired with a portal and reaching it, then
|
||||
every flow with its state, its node count and whether it has unpublished
|
||||
changes, and the last few runs and failures under them.
|
||||
|
||||
A `resources` line names each machine the engine can run a node on and how much
|
||||
of it is in use, plus how many nodes are queued for one. It is absent on an
|
||||
engine that accounts for nothing.
|
||||
|
||||
The portal reads one of three ways. `no portal` means this instance was
|
||||
never enrolled. `portal hub.fluksio.com` means the link is up. `portal
|
||||
unreachable` names the error, and is the one worth acting on: the dashboard is
|
||||
served from the other end, so nobody can reach it while that is showing.
|
||||
|
||||
`--watch` keeps it on screen and refreshes every five seconds until Ctrl-C, the
|
||||
cadence the dashboard polls at, since nothing here moves faster. It needs a
|
||||
terminal; without one, run it without `--watch` and the output pipes cleanly.
|
||||
|
||||
`--local` reads the flows and history out of this directory with no engine
|
||||
served, the same way `runs --local` does. It cannot be watched: an in-process
|
||||
engine is the command itself, so nothing changes under it.
|
||||
Shows engine health, portal connectivity, flows, recent runs and available
|
||||
resources. `--watch` refreshes every five seconds. `--local` reads the local
|
||||
instance without a served engine.
|
||||
|
||||
### `fluksio runs`
|
||||
|
||||
```sh
|
||||
fluksio runs [--flow train] [--limit 20]
|
||||
fluksio runs --flow train --limit 10
|
||||
```
|
||||
|
||||
The runs an engine has recorded, newest first: id, status, flow, duration, how
|
||||
long ago it was submitted, the commit of the repository it came from, and the
|
||||
inputs it was given. Statuses are coloured when a terminal is reading the
|
||||
output: `ok` green, `error` red, `cached` cyan.
|
||||
|
||||
Only the inputs that *differ from what the flow declares* are shown, and they
|
||||
are clamped to what is left of the terminal's width. A run that took the
|
||||
defaults lists none at all, and a flow taking a few kB of JSON does not push
|
||||
everything else off the line. `Client.runs()` and
|
||||
[`fluksio export runs`](#fluksio-export) are where the whole value is read.
|
||||
`--local` reads the same history from an in-process engine, without one having
|
||||
to be served.
|
||||
Lists recent runs, newest first. Omit `--flow` for all flows.
|
||||
`--local` reads the local instance.
|
||||
|
||||
### `fluksio retry`
|
||||
|
||||
```sh
|
||||
fluksio retry <run-id> [<run-id> ...]
|
||||
fluksio retry <run-id>
|
||||
fluksio retry --group <sweep-id>
|
||||
```
|
||||
|
||||
Runs the same thing again as a run of its own, carrying the flow, the inputs,
|
||||
the seed and the group of the one it repeats, and recording it as its parent.
|
||||
What it is for is a run an engine restart interrupted: those are marked
|
||||
`abandoned` once their lease goes stale, and `--group` retries every run of a
|
||||
sweep that did not end `ok`, so the missing combinations are filled in rather
|
||||
than the whole grid being submitted again. The stage cache is what makes it
|
||||
cheap — the nodes that finished are restored rather than recomputed.
|
||||
Submits a new run with the previous inputs, seed and group, linked to its
|
||||
parent. `--group` retries finished runs in that group that did not end
|
||||
`ok`. Active runs must be cancelled before retrying.
|
||||
|
||||
A run that has not finished is refused; cancel it first.
|
||||
|
||||
Runs that were still *queued* when an engine stopped need none of this: the
|
||||
next engine reads them back out of its own history and picks them up.
|
||||
Queued runs survive engine restarts. Interrupted running runs become
|
||||
`abandoned` after their lease expires; retry them to resume with any
|
||||
available cached stages.
|
||||
|
||||
### `fluksio flavors`
|
||||
|
||||
@@ -469,137 +263,96 @@ next engine reads them back out of its own history and picks them up.
|
||||
fluksio flavors
|
||||
```
|
||||
|
||||
The named sizes a node can ask for, as in `@node(resources="gpu-small")`, with the
|
||||
cores, memory and cards each stands for. Editing them is the Workers screen or
|
||||
`POST /api/v1/flavors`; this is the read.
|
||||
Lists named resource sizes, usable as `@node(resources="gpu-small")`.
|
||||
Edit them in **Workers** or through the flavors API.
|
||||
|
||||
### `fluksio sweep`
|
||||
|
||||
```sh
|
||||
fluksio sweep train --param lr=0.1,0.01 --param epochs=10,50 --wait
|
||||
fluksio sweep train --param lr=0.01,0.1 --param epochs=10,50 --wait
|
||||
```
|
||||
|
||||
Every combination of the parameter lists, submitted as one group: four runs
|
||||
above, sharing a `group_id` and executing in parallel. Values are typed by the
|
||||
flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`,
|
||||
`--no-cache` and `--local` mean what they do there. `--wait` blocks until all
|
||||
of them are finished and exits non-zero if any failed.
|
||||
Submits every combination of the declared input values: four runs here,
|
||||
sharing a `group_id`. Runs execute concurrently within the engine's limits.
|
||||
To vary seeds, the flow must declare a `seed` input; use
|
||||
`--param seed=1,2,3`.
|
||||
|
||||
`--seed`, `--sync`, `--no-sync`, `--no-cache`, `--timeout` and
|
||||
`--local` work as for `run`. `--wait` exits non-zero if any run fails.
|
||||
|
||||
### `fluksio export`
|
||||
|
||||
```sh
|
||||
fluksio export metrics --flow train --list
|
||||
fluksio export metrics --flow train --name train.train_loss --stride 10 -o curves.csv
|
||||
fluksio export runs --flow train --status ok > arms.csv
|
||||
fluksio export metrics --flow train --name train.loss -o curves.csv
|
||||
fluksio export runs --flow train --status ok -o results.csv
|
||||
```
|
||||
|
||||
The two tables an analysis reads. `export metrics` is the long one, a row per
|
||||
run, metric and step, which is what a plotting library takes without
|
||||
reshaping; `--name` keeps the metrics it lists and `--stride` keeps every Nth
|
||||
point of *each* curve. `export runs` is the wide one: a row per run with its
|
||||
inputs as columns, its final numbers, its status, its duration and the commit
|
||||
and digest of the code it ran.
|
||||
`metrics` exports a row per run, metric and step. `--list` lists metric
|
||||
names; `--name A,B` selects them; `--stride N` keeps every Nth point.
|
||||
|
||||
Every input the selected runs recorded becomes a column, so the schema does
|
||||
not move with the selection and a filter written against one export keeps
|
||||
working on the next; `--params lr,seed` narrows it to the axis a comparison is
|
||||
read along. `--metrics` narrows the final numbers the same way.
|
||||
`runs` exports a row per run with inputs, final numeric results, status,
|
||||
duration and code identifiers. Use `--params A,B` and `--metrics A,B` to
|
||||
select columns, including dotted paths such as `final_metrics.train_loss`.
|
||||
|
||||
An input left out of a submit is recorded at the value the flow declares for
|
||||
it, so every row says what it was actually run with rather than leaving the
|
||||
cell blank.
|
||||
Both support `--flow`, repeated `--run ID`, `--group`, `--status`,
|
||||
`--since`, `--until` and `--local`. The run ID joins results to metrics.
|
||||
|
||||
A node usually returns a record rather than a scalar, so both sides take
|
||||
dotted paths into one: `--metrics final_metrics.train_loss,test_metrics.known.perfect`
|
||||
selects three fields rather than two blobs, and `--params model.ansatz` does
|
||||
the same for an input. The defaults reach the same depth: every number a
|
||||
result carries becomes a column wherever it sits, and a record's inputs are
|
||||
taken leaf by leaf rather than as one blob.
|
||||
Output defaults to CSV on stdout. `-o FILE` writes a file; `--format jsonl`
|
||||
writes JSON Lines. Parquet requires `pip install 'fluksio[parquet]'` and
|
||||
`--format parquet -o FILE`.
|
||||
|
||||
Metric names are flow-qualified (a node of `train` writing `train_loss` records
|
||||
`train.train_loss`) so `--list` prints the names the selected runs
|
||||
carry when the spelling is not obvious. The engine answers it over the whole
|
||||
selection, so a name only an older run ever recorded is listed too.
|
||||
|
||||
Both take `--flow`, `--run ID` (repeat it), `--group`, `--status`, `--since`,
|
||||
`--until` and `--local`, and both put the run id on every row: it is the join
|
||||
back to the run page and to what the run made.
|
||||
|
||||
`--format` is `csv` (the default), `jsonl` or `parquet`; output goes to stdout
|
||||
unless `-o FILE` names somewhere. Parquet keeps the types, needs pyarrow
|
||||
(`pip install 'fluksio[parquet]'`) and needs a file to write, since it is not a
|
||||
stream. In a notebook, `Client.export_metrics()` and `Client.export_runs()`
|
||||
answer the same rows as a list of dicts, which `pandas.DataFrame` takes
|
||||
directly.
|
||||
In Python, `Client.export_metrics()` and `Client.export_runs()` return the
|
||||
same rows as lists of dictionaries.
|
||||
|
||||
### `fluksio artifacts`
|
||||
|
||||
```sh
|
||||
fluksio artifacts 1758042000123-9f2ab41c
|
||||
fluksio artifacts 1758042000123-9f2ab41c weights -o model.npz
|
||||
fluksio artifacts <run-id>
|
||||
fluksio artifacts <run-id> train.weights -o weights.json
|
||||
```
|
||||
|
||||
The files a run produced: what `fluksio.save_artifact(...)` wrote, and any
|
||||
artifact a node returned. Named alone it lists them: the message each left on,
|
||||
its size, its media type and the filename the node gave it. Name one and it is
|
||||
written here, under that filename unless `-o` says otherwise.
|
||||
|
||||
The message name is the one to pass, since it is what addresses the bytes;
|
||||
`--local` reads them from this directory without an engine served.
|
||||
Lists a run's returned artifacts, including their message names, sizes and
|
||||
filenames. Pass a name from that list to download it. Output uses the stored
|
||||
filename unless `-o` selects another path. `--local` reads the local instance.
|
||||
|
||||
## What lives in the data directory
|
||||
|
||||
```text
|
||||
.fluksio/ (or ~/.fluksio, with `--global`)
|
||||
├── client.json the token `serve` wrote, mode 600
|
||||
├── serve.pid the engine serving this directory, while one is
|
||||
├── serve.log what the engine under the dashboard printed
|
||||
├── .gitignore `*`, so a database and a credential are ignored from within
|
||||
├── fluksio.db SQLite: users, runs, metrics, observability, agents
|
||||
├── flows/ a git repository, one directory per flow
|
||||
│ ├── house/
|
||||
│ │ ├── flow.json the published structure
|
||||
│ │ ├── nodes/*.py the published node code
|
||||
│ │ ├── flow.draft.json unpublished edits, if any
|
||||
│ │ └── nodes.draft/*.py
|
||||
│ ├── _lib/ shared node sources
|
||||
│ ├── _dashboards/ dashboards, drafts and all
|
||||
│ └── requirements.txt what the Modules screen installs
|
||||
├── artifacts/ content-addressed bytes, two levels deep
|
||||
├── user-venv/ the interpreter your node code runs on
|
||||
├── secrets.enc encrypted credentials, deliberately outside flows/
|
||||
├── alerts.json alert channels and rules
|
||||
├── panels.json wall-panel pairings
|
||||
├── oauth-key.pem signs agent tokens
|
||||
├── cloud.json the portal enrolment, if there is one
|
||||
├── secret_key signs sessions and derives the secrets key
|
||||
└── env optional settings file
|
||||
```
|
||||
The main files and directories are:
|
||||
|
||||
Two things follow from this layout and are worth internalising:
|
||||
| Path | Contents |
|
||||
|---|---|
|
||||
| `client.json` | Local API URL and token |
|
||||
| `fluksio.db` | SQLite database: accounts, runs, metrics and operations history |
|
||||
| `flows/` | Git repository of flows, node sources, dashboards and module requirements |
|
||||
| `artifacts/` | Files stored by content digest |
|
||||
| `user-venv/` | Managed node environment, when used |
|
||||
| `secrets.enc` | Encrypted credentials, outside the flow repository |
|
||||
| `cloud.json` | Portal enrollment, when paired |
|
||||
| `secret_key` | Session signing and encryption key |
|
||||
| `serve.pid`, `serve.log` | Running engine information and terminal dashboard logs |
|
||||
| `env` | Optional instance settings |
|
||||
|
||||
**`flows/` is a real git repository.** `git log` is the history of every change
|
||||
anyone made to any flow. A run records the commit it ran at, so `git show` on
|
||||
that hash is literally the code that produced the number.
|
||||
The flow repository records edits in Git. Runs record its commit and code
|
||||
digests; code-defined nodes also depend on your project files, so keep that
|
||||
project's history too.
|
||||
|
||||
**Backing up the data directory backs up the instance.** Everything else is
|
||||
rebuildable. Copy it while the engine is stopped, or use SQLite's online backup
|
||||
for the database if it is not.
|
||||
Back up the data directory while the engine is stopped. An adopted venv and
|
||||
your project files live outside it and need their own backup.
|
||||
|
||||
## Settings
|
||||
|
||||
Settings come from the environment, or from an `env` file in the data
|
||||
directory. The ones you are most likely to touch:
|
||||
Settings come from environment variables or an `env` file in the data
|
||||
directory. Common choices:
|
||||
|
||||
| Variable | Default | What it does |
|
||||
|---|---|---|
|
||||
| `DATA_DIR` | `./.fluksio` via the CLI; `~/.fluksio` with `--global` | everything below it derives from this |
|
||||
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
|
||||
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
|
||||
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
|
||||
| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links |
|
||||
| `ENVIRONMENT` | `local` | `production` closes the interactive API schema |
|
||||
| `MCP_ENABLED` | `false` | opens the [agent endpoint](agents.md) |
|
||||
| `SECRET_KEY` | generated once | signs sessions, derives the secrets key |
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `FLUKSIO_HOME` | Select the CLI's data directory |
|
||||
| `NODE_VENV` | `auto` adopts an existing venv; `managed` requests a separate one; a path selects an interpreter |
|
||||
| `REDIS_HOST` | Use Redis for flow state and queues |
|
||||
| `FRONTEND_HOST` | Browser address used in mail, OAuth metadata and panel links |
|
||||
| `ENVIRONMENT` | `production` disables interactive API documentation |
|
||||
| `MCP_ENABLED` | Enable the [agent endpoint](agents.md) |
|
||||
|
||||
The full list is in [Configuration](../reference/configuration.md).
|
||||
See [Configuration](../reference/configuration.md) for defaults and the full
|
||||
list.
|
||||
|
||||
+97
-113
@@ -1,146 +1,130 @@
|
||||
# Writing a connector
|
||||
|
||||
A walkthrough from an empty directory to a node on the canvas. The normative
|
||||
surface is in [the connector contract](../reference/connector-contract.md);
|
||||
this is how to use it.
|
||||
A connector adds a device or service integration to the node palette. It is
|
||||
a Python package with a `ConnectorNode` subclass and an entry point.
|
||||
This example works without hardware or another repository.
|
||||
|
||||
## Start from the skeleton
|
||||
## Create the package
|
||||
|
||||
A working connector, kept as small as a correct one can be, lives at
|
||||
`skeleton/` in the [connectors repository](https://git.stroblme.de/Fluksio/connectors).
|
||||
Copy it:
|
||||
Make this directory beside your project:
|
||||
|
||||
```bash
|
||||
git clone https://git.stroblme.de/Fluksio/connectors.git
|
||||
cp -r connectors/skeleton fluksio-connector-mydevice
|
||||
cd fluksio-connector-mydevice
|
||||
```text
|
||||
fluksio-connector-random/
|
||||
pyproject.toml
|
||||
src/
|
||||
fluksio_connector_random/
|
||||
__init__.py
|
||||
```
|
||||
|
||||
Rename the package (`src/fluksio_connector_skeleton/` and the `[project]` name
|
||||
in `pyproject.toml`), then point the entry point at your class:
|
||||
Put this in `pyproject.toml`:
|
||||
|
||||
```toml
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "fluksio-connector-random"
|
||||
version = "0.1.0"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = ["fluksio>=0.1,<0.2"]
|
||||
|
||||
[project.entry-points."fluksio.node_types"]
|
||||
my_device = "fluksio_connector_mydevice:MyDevice"
|
||||
random_sensor = "fluksio_connector_random:RandomSensor"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/fluksio_connector_random"]
|
||||
```
|
||||
|
||||
## Install it into the engine
|
||||
|
||||
Connectors are found through installed package metadata, so the engine has to
|
||||
have yours installed. During development, install it as an editable path
|
||||
dependency of the backend:
|
||||
|
||||
```bash
|
||||
cd app/backend
|
||||
uv add --editable ../../fluksio-connector-mydevice
|
||||
```
|
||||
|
||||
Editing your connector's **code** then takes effect on the next engine restart,
|
||||
with no reinstall. Changing its **metadata** (the entry point, the version,
|
||||
the package name) needs `uv sync` to regenerate the installed metadata before
|
||||
a restart picks it up.
|
||||
|
||||
Two things to know about this:
|
||||
|
||||
- `uv add` writes the path dependency into `backend/pyproject.toml` and
|
||||
`uv.lock`. That path does not exist inside the backend's Docker image, so
|
||||
**revert both files before committing**; the dependency belongs on your
|
||||
machine, not in the repo. Distribution is a later milestone.
|
||||
- For the same reason, develop against the **standalone backend** rather than
|
||||
the Docker stack:
|
||||
|
||||
```bash
|
||||
cd app && make dev-backend # FastAPI on :8000
|
||||
make dev-frontend # Vite on :5173
|
||||
```
|
||||
|
||||
Restart the backend and your node type appears in the add-node palette (⌘P),
|
||||
labelled with the package it came from.
|
||||
|
||||
## Write the node
|
||||
Put this in `src/fluksio_connector_random/__init__.py`:
|
||||
|
||||
```python
|
||||
from typing import Any
|
||||
import random
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
from fluksio.flow.connector import CONTRACT_VERSION, ConnectorNode
|
||||
|
||||
|
||||
class MyDevice(ConnectorNode):
|
||||
class RandomSensor(ConnectorNode):
|
||||
contract = CONTRACT_VERSION
|
||||
title = "My device"
|
||||
description = "Reads temperature and humidity over the device's HTTP API."
|
||||
title = "Random sensor"
|
||||
description = "Emits a random reading. No sensor required."
|
||||
|
||||
class Params(ConnectorNode.Params):
|
||||
host: str = "192.168.1.50"
|
||||
api_key: str | None = Field(
|
||||
default=None, json_schema_extra={"x-secret": True}
|
||||
)
|
||||
ceiling: float = Field(default=1.0, gt=0)
|
||||
|
||||
async def poll(self) -> dict[str, Any] | None:
|
||||
reading = await self._read()
|
||||
return {"temperature": reading.temp, "humidity": reading.humidity}
|
||||
```
|
||||
|
||||
Then, on the canvas: place the node, set `poll_interval` to how often the
|
||||
device should be read, pick the API key from the secrets store, and add output
|
||||
ports named `temperature` and `humidity`. The port names are the wiring: any
|
||||
node consuming those messages is downstream of this one.
|
||||
|
||||
## If the device is told rather than asked
|
||||
|
||||
A connector that commands something implements `write()` instead of, or
|
||||
alongside, `poll()`. Input ports carry the values in:
|
||||
|
||||
```python
|
||||
def write(self, **ports: Any) -> None:
|
||||
self._device.set(ports["level"])
|
||||
```
|
||||
|
||||
Place the node with *input* ports named `level`, and whatever publishes that
|
||||
message drives the device. Set `idempotent = False` on the class: a write is a
|
||||
command, and a redelivery after a crash should not undo a newer one.
|
||||
|
||||
Keep the first version off the wire. A boolean setting the code checks before
|
||||
it sends (`artnet`'s `transmit` is the example) lets a flow be built and
|
||||
watched in the logs before anything physically moves.
|
||||
|
||||
## If the reading is bytes
|
||||
|
||||
A camera or a microphone publishes a reference rather than the bytes:
|
||||
|
||||
```python
|
||||
async def poll(self):
|
||||
jpeg = await asyncio.to_thread(self._grab)
|
||||
return {"frame": self.save_artifact(jpeg, "f.jpg", media_type="image/jpeg")}
|
||||
return {"reading": random.random() * self.config.ceiling}
|
||||
```
|
||||
|
||||
Place the node with an `image`-typed output port and a Media widget draws each
|
||||
frame as it lands. `fluksio-connector-test-media` publishes test frames and
|
||||
tones this way, so the whole path can be wired up with no camera in the room.
|
||||
Copy it if yours is a media device.
|
||||
The entry point name, `random_sensor`, becomes the stored node type.
|
||||
Choose a unique name; the engine skips names that are already registered.
|
||||
|
||||
## Try it without a device
|
||||
## Install and try it
|
||||
|
||||
Give `poll` something predictable first and confirm the values reach the canvas
|
||||
before pointing it at hardware:
|
||||
Use the **engine's Python environment**, the one where you installed Fluksio:
|
||||
|
||||
```python
|
||||
async def poll(self):
|
||||
return {"temperature": 21.5}
|
||||
```sh
|
||||
pip install -e /path/to/fluksio-connector-random
|
||||
```
|
||||
|
||||
Press Run, or wait for the poll interval, and the value appears on the edge.
|
||||
Anything the connector prints shows up in the logs panel in the dock, and a
|
||||
raised exception shows there with its traceback and marks the node down.
|
||||
Restart the engine from its original project directory so it keeps using
|
||||
the same instance data:
|
||||
|
||||
## Checklist before sharing it
|
||||
```sh
|
||||
fluksio serve
|
||||
```
|
||||
|
||||
- `contract = CONTRACT_VERSION` declared on the class itself.
|
||||
- Credentials marked `x-secret`, never read from the environment.
|
||||
- `stop()` is safe to call twice, and after a failed `start()`.
|
||||
- Blocking I/O wrapped in `asyncio.to_thread`.
|
||||
- `poll()` returns `None` rather than raising when there is simply nothing new.
|
||||
- `write()` returns quickly; the thread it runs on is the scheduler's.
|
||||
- Health reported when the connection comes and goes.
|
||||
- Anything that has to outlive a restart in `self.store`, keyed by device.
|
||||
Restart an already running engine after installing. In the browser:
|
||||
|
||||
1. Add a **Random sensor** node to a live flow.
|
||||
2. Set `poll_interval` to `5` seconds.
|
||||
3. Add an output port named `reading`, type `float`.
|
||||
4. Publish the flow and watch the value change.
|
||||
|
||||
The palette shows the package name and version. Code edits need an engine
|
||||
restart; metadata changes also need another editable install.
|
||||
|
||||
For Docker, install the package in the backend image and rebuild it.
|
||||
The Modules screen installs packages for **node code**, so it does not
|
||||
install connectors into the engine.
|
||||
|
||||
## Connect a real device
|
||||
|
||||
Replace `poll()` with your device read. Return a dictionary keyed by declared
|
||||
output port, or `None` when there is nothing new. The base class polls at
|
||||
`poll_interval`, publishes changed values, and reports failures.
|
||||
|
||||
Keep blocking I/O off the event loop with `asyncio.to_thread`. For pushed
|
||||
updates, manage the connection in `start()` and `stop()`, leave
|
||||
`poll_interval=0`, and call `self.inject({...})` when data arrives.
|
||||
|
||||
Mark credential parameters with
|
||||
`Field(default=None, json_schema_extra={"x-secret": True})`. The editor
|
||||
then stores a secret reference rather than the credential in the flow.
|
||||
|
||||
To send commands, implement `write(self, **ports)`: it receives values keyed
|
||||
by input port. Set `idempotent = False` for commands that must not be replayed
|
||||
after a crash.
|
||||
|
||||
## Media and persistent state
|
||||
|
||||
Use `self.save_artifact(data, name, media_type=...)` for bytes such as camera
|
||||
frames. Return the reference through an `image`, `audio`, or `video` port.
|
||||
For disposable live frames, pass `volatile=True`.
|
||||
|
||||
Use `self.store` for short strings that must survive a restart, such as a
|
||||
pairing token. Include the device address in the key: all nodes from the
|
||||
same top-level package share this store.
|
||||
|
||||
## Before sharing
|
||||
|
||||
- Declare `contract = CONTRACT_VERSION` on the class itself.
|
||||
- Return only declared output ports, with values matching their types.
|
||||
- Mark credentials as secrets.
|
||||
- Keep `stop()` safe after a failed start and when called twice.
|
||||
- Report connection health if you manage your own subscription.
|
||||
|
||||
See [the connector contract](../reference/connector-contract.md) for lifecycle,
|
||||
storage limits, and certificate pinning.
|
||||
|
||||
+132
-218
@@ -1,323 +1,237 @@
|
||||
# Writing node code
|
||||
|
||||
A Function node is a Python file: no base class, no decorator, no framework
|
||||
import unless you want one.
|
||||
A Function node runs Python in a worker process. Write its source in the
|
||||
canvas, or declare functions in your project with `@node` and
|
||||
[`fluksio sync`](cli.md#fluksio-sync).
|
||||
|
||||
```python
|
||||
def process(temperature, setpoint=21.0):
|
||||
"""Ask for heat when the room is below the comfort point."""
|
||||
return {"heat": temperature < setpoint}
|
||||
```
|
||||
|
||||
Declare `temperature` as an input port, `heat` as a Boolean output port, and
|
||||
`setpoint` as a setting. Now you've got a tiny thermostat.
|
||||
|
||||
## The rules
|
||||
|
||||
**One function called `process`.** If the file defines exactly one public
|
||||
function under another name, that one is used instead. Two, and the node
|
||||
refuses to load rather than guessing.
|
||||
- **Define `process`.** Without it, the worker accepts exactly one public
|
||||
function defined in the source. Multiple candidates produce an error.
|
||||
- **Arguments arrive by name** from input ports and settings. A setting cannot
|
||||
share a name with a port. See [Where values come from](../concepts/values.md).
|
||||
- **Return a dictionary keyed by output port.** Undeclared keys and values of
|
||||
the wrong type fail validation. With `@node(provides=Port("heat", "bool"))`,
|
||||
the SDK also accepts a bare value and wraps it for that single port.
|
||||
- **Use JSON-compatible values.** For files, use artifacts. For persistent
|
||||
state, use [flow state](../concepts/state.md); worker globals are not a
|
||||
reliable store.
|
||||
|
||||
**Arguments come from ports and settings, by name.** `temperature` above is an
|
||||
input port; `setpoint` is a setting typed into the node's panel. Both arrive as
|
||||
keyword arguments, which is why a setting may not share a name with a port.
|
||||
See [Where a node's values come from](../concepts/values.md).
|
||||
|
||||
**The return value is a dict keyed by output ports.** Every value is checked
|
||||
against the port's declared type before it is published. A key that is not a
|
||||
declared port is an error, not a silent drop. Nothing leaves a node except
|
||||
through a port it declared.
|
||||
|
||||
**Nothing else is importable from the engine.** Node code runs in a separate
|
||||
process, on a separate interpreter, with none of Fluksio's own modules on its
|
||||
path. What it can import is what the [Modules](../interface/operations.md)
|
||||
screen installed, which can include [your own project](#your-own-code-as-a-package),
|
||||
so a node need not be a self-contained file.
|
||||
|
||||
**A node is a pure function of its inputs.** No context object, no global
|
||||
store, no handle to reach for. A running total or a debounce timer has a
|
||||
specific shape; see [Keeping state in a flow](../concepts/state.md).
|
||||
Inside a node, `import fluksio` exposes reporting and artifact helpers. It
|
||||
does not expose the engine's internal modules. Other imports come from the
|
||||
node's Python environment; see [Packages](#packages).
|
||||
|
||||
## Producing values over time
|
||||
|
||||
A node that produces values *during* its execution is a generator. Every
|
||||
`yield` is a dict keyed by output port, published the instant it happens:
|
||||
Yield dictionaries to publish outputs before the node finishes:
|
||||
|
||||
```python
|
||||
import math
|
||||
|
||||
|
||||
def process(lr, steps):
|
||||
loss = 1.0
|
||||
for _ in range(steps):
|
||||
loss = train_one_step(lr)
|
||||
yield {"loss": loss} # published now
|
||||
for step in range(steps):
|
||||
loss = math.exp(-lr * (step + 1))
|
||||
yield {"loss": loss}
|
||||
return {"final_loss": loss}
|
||||
```
|
||||
|
||||
Whatever the generator `return`s at the end is the node's result, which is what
|
||||
downstream nodes read. If you never `return`, the last thing you yield is the
|
||||
result instead.
|
||||
|
||||
Mark the port so the flow says what it does:
|
||||
Declare `loss` as a streaming float port:
|
||||
|
||||
```json
|
||||
{"name": "loss", "dtype": "float", "stream": true}
|
||||
```
|
||||
|
||||
Two consequences. In a [run](../concepts/runs.md), the whole series is kept as
|
||||
that run's metrics, which is why there is no `log_metric()` anywhere in the
|
||||
API. And **the node's timeout starts measuring silence rather than duration**:
|
||||
each emission resets the deadline, so a node yielding every few seconds can run
|
||||
for hours under a timeout of 300.
|
||||
Each yield is type-checked and published immediately. The final return is
|
||||
the node's result; without a return, the last yield is the result. A
|
||||
[run](../concepts/runs.md) keeps numeric emissions as metric series and waits
|
||||
for the result before running downstream nodes. A live flow can also trigger
|
||||
downstream nodes from emissions.
|
||||
|
||||
### `fluksio.emit`
|
||||
|
||||
Where a `yield` cannot reach, because the value comes from inside somebody
|
||||
else's callback and they call you rather than the other way round:
|
||||
Use `emit` when a callback produces the value:
|
||||
|
||||
```python
|
||||
import fluksio
|
||||
|
||||
|
||||
def on_epoch_end(epoch, logs):
|
||||
fluksio.emit(loss=logs["loss"])
|
||||
```
|
||||
|
||||
Pass this callback to your training framework. It publishes through the same
|
||||
declared ports as `yield`. Outside a node, `emit` does nothing, so ordinary
|
||||
tests can still call your code.
|
||||
|
||||
## Bytes: artifacts
|
||||
|
||||
Save files or bytes with `save_artifact`, then return the reference through an
|
||||
`artifact` port:
|
||||
|
||||
```python
|
||||
import fluksio
|
||||
|
||||
|
||||
def process():
|
||||
model.fit(callbacks=[LambdaCallback(
|
||||
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
|
||||
)])
|
||||
return {"weights": ...}
|
||||
```
|
||||
|
||||
Same ports, same type checking, same publication. Prefer `yield` where you can
|
||||
reach it; `emit` where you cannot.
|
||||
|
||||
## Bytes: artifacts
|
||||
|
||||
Messages are JSON, which is what lets the same value pass through Redis, the
|
||||
work queue and the worker protocol unchanged. A checkpoint is not that.
|
||||
|
||||
```python
|
||||
import fluksio
|
||||
|
||||
|
||||
def process(dataset):
|
||||
path = fluksio.load_artifact(dataset) # → a local path to read
|
||||
...
|
||||
return {
|
||||
"weights": fluksio.save_artifact("model.pt", media_type="application/octet-stream"),
|
||||
"score": 0.94,
|
||||
"report": fluksio.save_artifact(
|
||||
b"Experiment complete\n", "report.txt", "text/plain"
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
`save_artifact` takes bytes or a path, stores them by their SHA-256 digest, and
|
||||
returns a small reference (digest, size, media type, name) which is what an
|
||||
`artifact`-typed port carries.
|
||||
Another node can read it with `fluksio.load_artifact(report)`, which returns
|
||||
a local file path. These two helpers require a running node; for tests,
|
||||
keep the underlying calculation in ordinary Python functions.
|
||||
|
||||
Because the address is the content's hash, a sweep whose fifty configs share
|
||||
one preprocessed input stores it once, and a reference stays valid wherever the
|
||||
store is reachable from, including on another machine.
|
||||
|
||||
`fluksio artifacts RUN NAME` is how one comes back out at a terminal.
|
||||
|
||||
### `fluksio.logger`
|
||||
|
||||
A node's `print` is kept as that node's logs, and so is anything on
|
||||
`fluksio.logger` is the same logger the SDK exports at top level, so code that
|
||||
runs both inside a node and outside one says it the same way:
|
||||
|
||||
```python
|
||||
fluksio.logger.info("resuming from epoch %d", start)
|
||||
```
|
||||
Artifacts are stored by their SHA-256 digest, so identical bytes share storage.
|
||||
Use [`fluksio artifacts`](cli.md#fluksio-artifacts) to list and download a
|
||||
run's files.
|
||||
|
||||
### Media
|
||||
|
||||
Say what the bytes are and the port can be typed for them:
|
||||
`image`, `audio` and `video` ports carry artifact references with a matching
|
||||
media type. For example, save JPEG bytes with `media_type="image/jpeg"` and
|
||||
return the reference through an `image` port. See
|
||||
[Payload types](../reference/payload-types.md#image-audio-video).
|
||||
|
||||
```python
|
||||
def process(speech): # an `audio` port
|
||||
clip = fluksio.load_artifact(speech)
|
||||
words = transcribe(clip)
|
||||
return {"transcript": words} # a `str` port
|
||||
```
|
||||
|
||||
```python
|
||||
def process(camera_url):
|
||||
for index, jpeg in enumerate(grab(camera_url)): # a generator
|
||||
frame = fluksio.save_artifact(
|
||||
jpeg, f"frame-{index:05d}.jpg",
|
||||
media_type="image/jpeg", volatile=True,
|
||||
)
|
||||
frame["meta"] = {"seq": index}
|
||||
yield {"frame": frame} # an `image` stream port
|
||||
```
|
||||
|
||||
An `image`, `audio` or `video` port is an artifact reference whose media type
|
||||
has to match, so a node declaring `audio` never receives a video by accident.
|
||||
See [Payload types](../reference/payload-types.md#image-audio-video) for what
|
||||
each carries and what rates are realistic.
|
||||
|
||||
`volatile=True` is for a frame rather than a result. The bytes go to a ring in
|
||||
memory instead of the data volume, and the engine pushes them down the
|
||||
websocket to whichever screens are drawing that message — so a camera runs at
|
||||
ten frames a second without writing anything to disk. They last as long as it
|
||||
takes newer frames to need the room. Leave it off for a clip somebody asked to
|
||||
keep.
|
||||
|
||||
!!! warning "Emitted media is not kept; returned media is"
|
||||
|
||||
Only what a node *returns* is recorded against its run. Frames yielded
|
||||
along the way are replaced in state by the next one, and the artifact sweep
|
||||
removes bytes nothing refers to any more, which is what stops a camera
|
||||
filling the disk. If a particular frame matters, return it: a volatile one
|
||||
is copied out of the ring when the run records it, so returning it is also
|
||||
what makes it outlive the next few seconds.
|
||||
For transient frames, `save_artifact(..., volatile=True)` uses an in-memory
|
||||
ring and streams bytes to connected screens. New frames eventually replace
|
||||
old ones. A volatile artifact returned as a run result is copied to persistent
|
||||
storage; emitted frames alone are not recorded as run artifacts.
|
||||
|
||||
## Printing
|
||||
|
||||
`print` works and is captured. The first 16 KB per call is kept and shown in
|
||||
the flow editor's log panel and on the run's per-node record; the rest is
|
||||
dropped, so a node printing in a loop cannot fill anything up.
|
||||
|
||||
Use it to debug, not to record results. A number worth keeping is an output
|
||||
port, not a line of text.
|
||||
`print` and `fluksio.logger.info(...)` are captured in the node logs. The
|
||||
worker retains the first 16,384 characters per call. Use logs to debug and
|
||||
output ports for numbers you want to plot or compare.
|
||||
|
||||
## Errors
|
||||
|
||||
An exception fails that node's execution, not the flow. The message you see is
|
||||
one line from the frame in *your* code, not a stack through the engine.
|
||||
|
||||
The node keeps its last error visible after it recovers, so a failure that
|
||||
fired an alert at 03:00 still says what it was at 09:00. It can also be
|
||||
acknowledged from the canvas.
|
||||
Exceptions appear on the node, with a short message and a traceback. In a live
|
||||
flow, one failed execution does not stop the whole flow. In a batch run, node
|
||||
failure makes the run fail. The canvas keeps the last error visible after
|
||||
recovery until you acknowledge it.
|
||||
|
||||
## Timeouts
|
||||
|
||||
`timeout` on a node is how many seconds its code may be *silent* before it is
|
||||
stopped. A yield or an `emit` resets the clock, and the first call's imports are
|
||||
not charged to it: a node importing torch is not being slow, it is loading.
|
||||
`timeout` is the allowed number of seconds without an emission. A `yield` or
|
||||
`emit` resets the deadline; initial compilation and imports are excluded.
|
||||
|
||||
There is no timeout by default. Training runs for hours and a node that reports
|
||||
nothing is usually working, so the engine waits: what fails a call is the worker
|
||||
dying, which arrives at once rather than after a deadline. Set a timeout on the
|
||||
nodes where silence means stuck (an HTTP call that should answer in seconds, a
|
||||
loop that can spin) either per node or, for all of them,
|
||||
with `FLOW_NODE_TIMEOUT`. `timeout = 0` says explicitly that this node has none.
|
||||
There is no timeout by default. Set one on a node where silence means stuck,
|
||||
or use `FLOW_NODE_TIMEOUT` for the instance default. `timeout=0` explicitly
|
||||
disables it for a node. A worker disappearing fails its active call without
|
||||
waiting for that deadline.
|
||||
|
||||
## Running a node somewhere else
|
||||
|
||||
A node declares the label of the machine it needs:
|
||||
Set a device label to select an attached worker:
|
||||
|
||||
```json
|
||||
{"id": "train", "device": "gpu", "device_policy": "require", "timeout": 7200}
|
||||
```
|
||||
|
||||
`require` (the default) waits for a worker carrying that label; `prefer` runs
|
||||
locally when none is attached. A node bound to a device is compiled *on that
|
||||
machine*: a node importing `torch` is correct on the GPU box and a missing
|
||||
module on the engine, so checking it here would fail something that is fine.
|
||||
|
||||
See [Remote workers](workers.md).
|
||||
`require` is the default and waits for a worker with that label. `prefer` uses
|
||||
the local engine when none is attached. Code is compiled on the selected
|
||||
machine, which needs the packages your node imports. See
|
||||
[Remote workers](workers.md) for setup and resource requests.
|
||||
|
||||
## Sharing code between flows
|
||||
|
||||
A node's source can be promoted to the shared library from its panel, and other
|
||||
flows can then use it by reference. There is one copy and one place to edit, and
|
||||
every flow using it runs the edit.
|
||||
Promote a node's source to the shared library from its panel. Other flows can
|
||||
reference that source, and edits apply to all of them. Shared sources live in
|
||||
`_lib/` in the instance's flow repository.
|
||||
|
||||
Shared sources live in `_lib/` in the flow repository, so they are versioned
|
||||
with everything else.
|
||||
For code-defined flows, reuse the same decorated function. `use(fn, ...)`
|
||||
changes its wiring or settings for one flow:
|
||||
|
||||
```python
|
||||
use(fit, wire={"dataset": "augmented"}, epochs=3)
|
||||
```
|
||||
|
||||
## Packages
|
||||
|
||||
Node code runs in a virtual environment of its own, on the instance's data
|
||||
volume, separate from the one Fluksio itself runs on.
|
||||
The startup banner shows the interpreter used for local nodes:
|
||||
|
||||
Declare what you import in [Modules](../interface/operations.md), or over the
|
||||
API:
|
||||
| Environment | How to add packages |
|
||||
|---|---|
|
||||
| Fluksio installed in an existing venv | Use pip or uv in that venv, then `fluksio sync` to refresh workers. Modules is read-only. |
|
||||
| Managed node venv | Edit the manifest in **Modules** and apply it. |
|
||||
| Remote worker | Install packages in the interpreter selected by `--python`. |
|
||||
|
||||
With `NODE_VENV=auto`, Fluksio adopts its existing venv unless this instance
|
||||
already has a managed one. Without an existing venv, it creates a managed
|
||||
environment. `NODE_VENV=managed` always requests one; the Docker image uses
|
||||
this setting.
|
||||
|
||||
In managed mode, the Modules manifest is installed with `uv pip sync` and
|
||||
versioned with the flows. Applying it refreshes local workers. The API accepts
|
||||
the same manifest:
|
||||
|
||||
```sh
|
||||
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
|
||||
curl -X POST "$FLUKSIO/modules/apply" -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"requirements": "numpy>=2\npandas\n"}'
|
||||
```
|
||||
|
||||
It is a pip manifest installed with `uv pip sync`, versioned alongside your
|
||||
flows. An install takes effect immediately; nothing restarts.
|
||||
Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN` is an
|
||||
access token; see [The HTTP API](api.md).
|
||||
|
||||
### Your own code as a package
|
||||
|
||||
!!! tip "If Fluksio is installed in the venv you work in, skip this"
|
||||
|
||||
Node code then runs on that environment, so your project and everything it
|
||||
imports are already importable; see
|
||||
[Getting started: data science](../getting-started/data-science.md). What
|
||||
follows is for a Fluksio with a venv of its own, which is what a container
|
||||
always has.
|
||||
|
||||
A manifest line can name a directory, so the project you already have is
|
||||
installable like any other dependency:
|
||||
`fluksio sync` generates an import shim pointing at your local project. A
|
||||
managed environment can also install your project from a Modules manifest:
|
||||
|
||||
```text
|
||||
-e /home/you/my-research
|
||||
numpy>=2
|
||||
```
|
||||
|
||||
A node body then imports it, and the logic stays where it already lives: in
|
||||
your repository, under your own version control, importing its own siblings:
|
||||
Then a canvas node can import ordinary helpers from it:
|
||||
|
||||
```python
|
||||
from myresearch.train import fit
|
||||
from myresearch.training import train_model
|
||||
|
||||
|
||||
def process(lr, epochs):
|
||||
return fit(lr, epochs)
|
||||
return {"score": train_model(lr, epochs)}
|
||||
```
|
||||
|
||||
That is the whole of it. The node is three lines, `myresearch` can be as many
|
||||
modules as it likes, and nothing was copied.
|
||||
Editable installs expose changed files, but warm local workers still hold
|
||||
imports in memory. Apply the Modules manifest or run `fluksio sync` after
|
||||
edits. Attached workers start a process per call and read the files afresh.
|
||||
|
||||
!!! tip "You can have those three lines written for you"
|
||||
|
||||
Decorate `fit` with `@node(...)` where it is defined, say which nodes make
|
||||
a flow with `Flow(...)`, and `fluksio sync` generates the body above,
|
||||
along with the flow document, so there is nothing to PUT by hand. The
|
||||
declaration lives beside the function it describes and is checked against
|
||||
its signature. See
|
||||
[Getting started: data science](../getting-started/data-science.md).
|
||||
|
||||
!!! warning "Editable, but not live"
|
||||
|
||||
`-e` means edits reach the venv without reinstalling, but a node's process
|
||||
already holds the imported module in memory. The engine's workers are
|
||||
long-lived, so a change to your code is picked up when they are retired,
|
||||
which is what **Apply** does. Pressing it after an edit is the loop.
|
||||
|
||||
A [worker](workers.md) you attach yourself is the exception: it starts a
|
||||
process per call, so it reads your code fresh every run. If you are
|
||||
iterating on the code many times an hour, point one at your own
|
||||
interpreter (`fluksio-worker --python "$(which python)"`) and mark the
|
||||
node with its label.
|
||||
|
||||
!!! note "The path is a deployment detail"
|
||||
|
||||
It is resolved on whichever machine runs the node, and the manifest is
|
||||
committed to the flow repository, so an absolute path from your laptop
|
||||
means nothing inside a container or on a GPU box. Those need their own
|
||||
install of the same project; a VCS requirement
|
||||
(`myresearch @ git+ssh://…@a1b2c3d`) travels where a path does not.
|
||||
Paths are resolved on the machine running the node. A remote worker or
|
||||
container needs its own copy or installation of your project; syncing a shim
|
||||
does not ship the package.
|
||||
|
||||
## A worked example
|
||||
|
||||
The repository ships a small supervised fit as a seedable demo: three nodes,
|
||||
a batch flow, streaming metrics, artifacts between stages, and a GPU-labelled
|
||||
node that falls back to the engine when no worker is attached. It is the
|
||||
shortest complete thing to read:
|
||||
[Getting started: data science](../getting-started/data-science.md) contains a
|
||||
complete runnable flow. The public repository also includes
|
||||
`examples/myresearch/`, with separate preparation, training and evaluation
|
||||
modules. From the repository root:
|
||||
|
||||
```text
|
||||
prepare ──dataset(artifact)──▶ train ──weights(artifact)──▶ evaluate
|
||||
│
|
||||
└── loss (streaming float) ──▶ chart
|
||||
```sh
|
||||
fluksio sync examples/myresearch
|
||||
fluksio run train --lr 0.05 --wait
|
||||
```
|
||||
|
||||
The example is built by a seed script the maintainers run against a live
|
||||
stack; the flow above is what it produces.
|
||||
Start an instance first, as shown in the getting started guide.
|
||||
|
||||
## See also
|
||||
|
||||
- [Where a node's values come from](../concepts/values.md)
|
||||
- [Keeping state in a flow](../concepts/state.md)
|
||||
- [Runs: pipelines that finish](../concepts/runs.md)
|
||||
- [Node types](../reference/node-types.md) — the ones you do not have to write
|
||||
- [Node types](../reference/node-types.md)
|
||||
|
||||
+120
-207
@@ -1,119 +1,77 @@
|
||||
# Remote workers
|
||||
|
||||
The engine runs where the automations are. The GPU is somewhere else, the
|
||||
Raspberry Pi with the relays is in a shed, and neither of them is on the same
|
||||
network as the other.
|
||||
A worker runs node code on another machine: a GPU host, a Raspberry Pi or a
|
||||
batch allocation. It connects outward to the engine over an authenticated
|
||||
WebSocket. Keep one engine; add workers where the hardware lives.
|
||||
|
||||
A **worker** is a process that runs the code of nodes marked for it. It dials
|
||||
*out* to the engine over one authenticated websocket, so nothing on that
|
||||
machine has to be reachable, and nothing has to expose the engine's state
|
||||
backend across hosts, which it never should.
|
||||
## Create a worker token
|
||||
|
||||
## Install and attach
|
||||
On the engine, use a superuser access token. In these examples:
|
||||
|
||||
- `API_URL` is the engine's API base, including `/api/v1`, such as
|
||||
`https://api.example.com/api/v1` or `http://localhost:8080/api/v1` for a LAN stack.
|
||||
- `TOKEN` is your superuser bearer token. The login endpoint
|
||||
`POST /api/v1/login/access-token` accepts form fields `username` and `password`.
|
||||
|
||||
Mint a token with the labels this machine should carry:
|
||||
|
||||
```sh
|
||||
pip install fluksio-worker
|
||||
|
||||
fluksio-worker \
|
||||
--url wss://api.${DOMAIN}/api/v1/workers/attach \
|
||||
--labels gpu,cuda12 \
|
||||
--python /opt/torch-venv/bin/python
|
||||
```
|
||||
|
||||
`fluksio-worker` is its own distribution: the agent, the node runner, and
|
||||
`websockets`. Nothing of the engine, so a GPU box does not install a database
|
||||
driver in order to run a training step. An engine host already has it, and
|
||||
`fluksio worker …` is the same program.
|
||||
|
||||
| Option | Default | What it does |
|
||||
|---|---|---|
|
||||
| `--url` | **required** | `wss://…/api/v1/workers/attach` |
|
||||
| `--token` | `$FLUKSIO_WORKER_TOKEN` | the credential, minted on the engine; on the command line any user of the machine reads it from `ps`, so prefer the variable |
|
||||
| `--name` | this host's name | sent along; the worker list shows the token's name |
|
||||
| `--labels` | the token's | comma-separated; must match the labels the token was minted with |
|
||||
| `--python` | this interpreter | the interpreter node code runs on |
|
||||
| `--parallel` | `1` | how many node calls it will take at once |
|
||||
| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |
|
||||
| `--cpus` | what the job or the machine has | cores to advertise |
|
||||
| `--gpus` | what the job says, else counted | GPUs to advertise |
|
||||
| `--ram-mb` | what the job or the machine has | memory to advertise, in MB |
|
||||
| `--max-idle` | never | stop after this many seconds with nothing running |
|
||||
| `--insecure` | off | allow `ws://` and `http://` to an engine on another machine |
|
||||
|
||||
The worker refuses a plain `ws://` or `http://` address unless the engine is on
|
||||
this machine (`localhost`, `127.0.0.0/8`, `::1`): its token and every secret a
|
||||
node is handed would cross the network in clear. `--insecure` is for a network
|
||||
you trust, such as an engine on the same LAN without TLS. Node code gets this
|
||||
host's environment less anything named like a credential (`*TOKEN*`,
|
||||
`*SECRET*`, `*PASSWORD*`), the same as on the engine; hand a node a credential
|
||||
as a [secret](../interface/operations.md#secrets) instead.
|
||||
|
||||
`--python` is the important one. It is how this machine keeps its own wheels
|
||||
(the CUDA build, the vendor SDK, the thing that will not install anywhere else)
|
||||
without the engine ever installing them or knowing about them.
|
||||
|
||||
### What it says it has
|
||||
|
||||
A worker reports its inventory when it attaches (cores, GPUs and memory) and
|
||||
the engine schedules against it: a node asking for two cores and a GPU goes to
|
||||
a machine that has them free, not merely to one carrying the right label.
|
||||
|
||||
Cores and memory are read off the machine, or off the batch job that started
|
||||
this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **What the job says
|
||||
it was given always wins for GPUs** (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`,
|
||||
or `FLUKSIO_WORKER_GPUS`): a node with eight cards may have granted this job
|
||||
one, and advertising eight would be a lie the scheduler acts on. With nothing
|
||||
said, NVIDIA's device nodes are counted, the same as the engine does for its
|
||||
own machine, and `--gpus` overrides either. No vendor tool is asked, which is
|
||||
what keeps the one dependency from becoming two. A worker that reports nothing
|
||||
still attaches and is scheduled by its label alone, as every worker was before
|
||||
any of them reported anything.
|
||||
|
||||
The engine tells each call what it may use: thread caps, and the devices it
|
||||
may see. The worker starts a process per call, so it applies them at the only
|
||||
moment a numerical library still reads them: before the import.
|
||||
|
||||
`--max-idle` is for a worker something else started for one job, a batch
|
||||
scheduler say. It exits when nothing has run for that long, so the allocation
|
||||
goes back rather than idling until its walltime.
|
||||
|
||||
## Mint the token
|
||||
|
||||
On the engine, as a superuser:
|
||||
|
||||
```sh
|
||||
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
|
||||
curl -X POST "$API_URL/workers/tokens" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}'
|
||||
```
|
||||
|
||||
Shown once, valid for a year, since a worker is a machine somebody sets up and
|
||||
leaves running. The name and labels are signed into it: labels decide which
|
||||
nodes, with their source and secrets, are sent to a worker, so a worker cannot
|
||||
choose its own. It shows up under the token's name, and a worker whose
|
||||
`--labels` differ from the token's is refused.
|
||||
The response contains `token`, `jti` and the expiry. Save the token as
|
||||
`FLUKSIO_WORKER_TOKEN` on the worker host. It is returned once and is valid
|
||||
for 365 days. The signed name and labels control which nodes and secrets the
|
||||
engine sends to that worker.
|
||||
|
||||
## Install and attach
|
||||
|
||||
Use Python 3.12 or later on the worker host:
|
||||
|
||||
```sh
|
||||
curl -s $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" | jq
|
||||
curl -X POST $FLUKSIO/workers/tokens/$JTI/revoke -H "Authorization: Bearer $TOKEN"
|
||||
pip install fluksio-worker
|
||||
export FLUKSIO_WORKER_TOKEN='<token from the engine>'
|
||||
|
||||
fluksio-worker \
|
||||
--url wss://api.example.com/api/v1/workers/attach \
|
||||
--labels gpu,cuda12 \
|
||||
--python /opt/torch-venv/bin/python
|
||||
```
|
||||
|
||||
The first lists what was minted, by `jti` (the tokens themselves are not
|
||||
kept); the second refuses one from then on, here and at the artifact
|
||||
endpoint, and drops a worker attached with it. Rotating the keypair agent
|
||||
tokens are signed with still revokes every worker at once. A token minted
|
||||
before tokens carried their labels is refused: mint a new one.
|
||||
Use your engine's hostname. `--python` selects the environment containing the
|
||||
node's dependencies, such as PyTorch or a device SDK. If you install the worker
|
||||
in that environment, you can omit the option. An engine installation also
|
||||
provides the same agent as `fluksio worker`.
|
||||
|
||||
??? note "A host where pip is not an option"
|
||||
| Option | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `--url` | required | engine's `/api/v1/workers/attach` WebSocket |
|
||||
| `--token` | `$FLUKSIO_WORKER_TOKEN` | credential; prefer the environment variable to keep it out of process arguments |
|
||||
| `--name` | hostname | suggested name; the engine uses the token's name |
|
||||
| `--labels` | token's labels | comma-separated; when supplied, must match the token |
|
||||
| `--python` | worker's interpreter | Python used for node code |
|
||||
| `--parallel` | `1` | concurrent calls |
|
||||
| `--artifact-url` | derived from `--url` | artifact endpoint, if hosted separately |
|
||||
| `--cpus` | detected | advertised CPU count |
|
||||
| `--gpus` | detected | advertised GPU count; explicit `0` means none |
|
||||
| `--ram-mb` | detected | advertised memory in MB |
|
||||
| `--max-idle` | `0` | exit after this many idle seconds; `0` disables |
|
||||
| `--insecure` | off | allow unencrypted connections to a non-loopback host |
|
||||
|
||||
The two files work copied into one directory and run with `python agent.py
|
||||
…`. The engine serves the runner itself at `GET /api/v1/workers/runtime`.
|
||||
It is the same module its own local workers run, deliberately standard
|
||||
library only.
|
||||
For a trusted LAN stack, the URL is
|
||||
`ws://<host-ip>:8080/api/v1/workers/attach` and requires `--insecure`.
|
||||
Outside loopback, the worker otherwise requires encrypted URLs (`wss://` and
|
||||
`https://`): node secrets and the worker token travel over the connection.
|
||||
|
||||
Node code receives a filtered host environment. Credential-like variables are
|
||||
removed; pass credentials through [Secrets](../interface/operations.md#secrets).
|
||||
|
||||
## Send a node to it
|
||||
|
||||
A node declares the label of the machine it needs:
|
||||
Set these fields on a node through `PUT /api/v1/flows/{name}`:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -124,94 +82,73 @@ A node declares the label of the machine it needs:
|
||||
}
|
||||
```
|
||||
|
||||
| `device_policy` | Behaviour when nothing carrying the label is attached |
|
||||
This is a node fragment; keep the node's other fields and the rest of the flow
|
||||
when updating it. `device` and `device_policy` are not in the node panel yet.
|
||||
|
||||
| `device_policy` | If no matching worker is attached |
|
||||
|---|---|
|
||||
| `require` (default) | the run stays `queued` and says what it is waiting for |
|
||||
| `prefer` | it runs on the engine instead |
|
||||
| `require` (default) | queues until a matching worker is available |
|
||||
| `prefer` | falls back to the engine |
|
||||
|
||||
`prefer` is what makes a flow work before the GPU box exists. `require` is what
|
||||
you want once it does.
|
||||
Source code travels with each call and compiles on the worker, so remote-only
|
||||
dependencies do not need to be installed on the engine. Calls to `emit`,
|
||||
`save_artifact` and `load_artifact` work there too.
|
||||
|
||||
!!! note "Not in the panel yet"
|
||||
Cancelling a run stops its active node processes. A disconnected worker fails
|
||||
in-flight calls. While executing, the agent sends heartbeats every ten seconds;
|
||||
90 seconds without contact fails the call. Heartbeats do not reset the node's
|
||||
own silence timeout.
|
||||
|
||||
`device` and `device_policy`, which machine a node's code runs on, are set
|
||||
through the API rather than the panel, with `PUT /flows/{name}`.
|
||||
## Resources and artifacts
|
||||
|
||||
## What follows from this
|
||||
Workers report CPU, GPU and memory inventory, and scheduling checks both labels
|
||||
and available resources. CPU and memory detection uses Slurm allocations when
|
||||
present, otherwise host figures. GPU detection uses `SLURM_GPUS_ON_NODE`,
|
||||
`FLUKSIO_WORKER_GPUS` or `SLURM_JOB_GPUS` before counting NVIDIA device nodes.
|
||||
Explicit flags override detection.
|
||||
|
||||
- **The node's source travels with every call.** Nothing has to be deployed to
|
||||
the worker, and changing a node's code takes effect on the next execution.
|
||||
- **A node bound to a device is compiled on that machine.** A node importing
|
||||
`torch` is correct on the GPU box and a missing module on the engine, so
|
||||
checking it here would fail something that is fine.
|
||||
- **`import fluksio` inside a node is the worker's own reporter.** `emit`,
|
||||
`save_artifact`, `load_artifact`, installed before your code runs, so an
|
||||
installed `fluksio` package on that box never shadows it.
|
||||
- **Cancelling a run kills what it is executing**, there or here, and leaves
|
||||
other runs of the same node alone.
|
||||
- **If the worker disappears mid-call**, the run fails in seconds with
|
||||
`worker went away mid-call` rather than waiting out its timeout.
|
||||
- **A worker sends a heartbeat every ten seconds while it executes**, so a long
|
||||
node is distinguishable from a dead socket. Ninety seconds of nothing at all,
|
||||
not even a heartbeat, fails the call as gone. A heartbeat says the *agent* is
|
||||
alive and nothing about the node, so it never satisfies a node's own timeout:
|
||||
one set to thirty seconds fires after thirty seconds of the node reporting
|
||||
nothing, wherever it runs.
|
||||
Artifacts are identified by content hash. Remote workers fetch and upload over
|
||||
HTTP beside the WebSocket endpoint; node code uses the same artifact helpers
|
||||
as on the engine. Downloads are cached by digest:
|
||||
|
||||
## Artifacts across machines
|
||||
| Variable | Default |
|
||||
|---|---|
|
||||
| `FLUKSIO_ARTIFACT_CACHE` | `fluksio-artifacts` in the temporary directory |
|
||||
| `FLUKSIO_ARTIFACT_CACHE_BYTES` | `1073741824` (1 GiB) |
|
||||
|
||||
An artifact reference names content by its hash, not a location, so it stays
|
||||
valid wherever the store is reachable from. A worker that shares the engine's
|
||||
filesystem writes to it directly; one that does not fetches and uploads over
|
||||
HTTP, using the artifact endpoint beside the socket it already has. Either way
|
||||
your node code is the same two calls.
|
||||
Oldest files are removed when the cache exceeds its limit. Raise it if workers
|
||||
repeatedly use large inputs.
|
||||
|
||||
A fetch is cached on the worker by digest, since content addressing means an
|
||||
entry is never stale. Nothing expires on its own, so the cache is bounded by
|
||||
size and the oldest fall out: `FLUKSIO_ARTIFACT_CACHE` says where it lives
|
||||
(default a directory in the temporary directory) and
|
||||
`FLUKSIO_ARTIFACT_CACHE_BYTES` how much it holds (default 1 GiB). Worth raising
|
||||
where a worker reads the same large inputs repeatedly, and worth leaving alone
|
||||
where it reads a media stream — those are chunks nothing asks for twice.
|
||||
|
||||
## Seeing what is attached
|
||||
The **Workers** screen shows attached hosts, available resources and queued
|
||||
nodes. The API exposes the same information:
|
||||
|
||||
```sh
|
||||
curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq
|
||||
curl "$API_URL/workers" -H "Authorization: Bearer $TOKEN"
|
||||
curl "$API_URL/workers/resources" -H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
|
||||
Name, labels, how many calls it will take at once, how many are in flight, when
|
||||
it attached, when it was last seen, its Python version, a digest of its
|
||||
environment, and what it says it has: cores, GPUs and memory.
|
||||
## Revoke or upgrade
|
||||
|
||||
## Upgrading
|
||||
List tokens and revoke one by its `jti`:
|
||||
|
||||
The engine and the worker speak a version-matched protocol, and a worker
|
||||
announcing anything else is refused rather than half-understood. Protocol 2,
|
||||
the one that carries inventory, is `fluksio-worker` 0.2.0. An older agent is
|
||||
told so on the socket and stops, rather than retrying against an engine that
|
||||
will never accept it; `pip install -U fluksio-worker` on that host is the whole
|
||||
upgrade. Nothing changed in the runner served at `GET /api/v1/workers/runtime`,
|
||||
so a host that copies its two files copies the same one as before.
|
||||
```sh
|
||||
curl "$API_URL/workers/tokens" -H "Authorization: Bearer $TOKEN"
|
||||
curl -X POST "$API_URL/workers/tokens/$JTI/revoke" \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
|
||||
A token minted before tokens carried their labels and a `jti` is refused. Mint
|
||||
each existing worker host a new one with `POST /workers/tokens`, naming the
|
||||
labels it should carry, and swap it in for `FLUKSIO_WORKER_TOKEN`.
|
||||
Revocation also disconnects attached workers and blocks artifact access.
|
||||
Replacing the OAuth signing key revokes all worker tokens.
|
||||
|
||||
**Upgrade the engine before its workers.** A worker sends its token as an
|
||||
`Authorization` header on the handshake, where it stays out of access logs, and
|
||||
an engine from before that reads it only from the URL: it refuses the newer
|
||||
worker at the handshake. The engine still takes the old `?token=` for one
|
||||
release, so older workers keep attaching to a newer engine meanwhile.
|
||||
Upgrade the engine before its workers, then run `pip install -U fluksio-worker`
|
||||
on each host. Engine and worker must speak the same protocol; a mismatch is
|
||||
reported and refused. Older tokens without signed labels and a `jti` need to
|
||||
be replaced.
|
||||
|
||||
## Machines from a batch scheduler
|
||||
## Slurm provisioning
|
||||
|
||||
A cluster is not a machine that attaches and stays; it is a queue somebody else
|
||||
owns. So Fluksio does not submit *nodes* to Slurm. It submits a job whose payload
|
||||
is an ordinary worker dialling back in, and from there everything works the way
|
||||
it already does: the same protocol, the same artifacts, the same cancellation.
|
||||
|
||||
Write the clusters into `provisioners.json` beside the flows:
|
||||
The engine can submit a Slurm job that starts a worker. Put this configuration
|
||||
in `$DATA_DIR/provisioners.json`, replacing the example addresses and paths:
|
||||
|
||||
```json
|
||||
[{
|
||||
@@ -219,58 +156,34 @@ Write the clusters into `provisioners.json` beside the flows:
|
||||
"name": "hpc",
|
||||
"login": "me@login.cluster",
|
||||
"ssh_key": "/secrets/hpc_ed25519",
|
||||
"engine_url": "wss://api.${DOMAIN}/api/v1/workers/attach",
|
||||
"engine_url": "wss://api.example.com/api/v1/workers/attach",
|
||||
"max_idle_s": 300,
|
||||
"provision_timeout_s": 900,
|
||||
"profiles": [{
|
||||
"name": "gpu-small",
|
||||
"cpus": 8, "gpus": 1, "ram_mb": 65536,
|
||||
"cpus": 8,
|
||||
"gpus": 1,
|
||||
"ram_mb": 65536,
|
||||
"labels": ["gpu"],
|
||||
"sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"],
|
||||
"prerun": ["module load cuda/12", "source ~/venvs/flux/bin/activate"]
|
||||
"prerun": ["module load cuda/12", "source ~/venvs/fluksio/bin/activate"]
|
||||
}]
|
||||
}]
|
||||
```
|
||||
|
||||
A **profile** is what the scheduler is asked for, where a flavor is what a node
|
||||
asks for. They are separate on purpose, and agree when you set them up to.
|
||||
The engine uses system SSH to submit `sbatch`. Prepare the environment named
|
||||
by `prerun` with `fluksio-worker` and your node dependencies; the generated job
|
||||
does not install packages. Compute nodes must be able to connect to the
|
||||
engine. Each job receives its own worker token, valid for 24 hours.
|
||||
|
||||
When a node needs a machine nothing attached can give, and a profile fits, the
|
||||
engine `sbatch`es one over the system `ssh`, and the run waits meanwhile, saying
|
||||
so. `prerun` owns the environment: a `module load`, or a venv with
|
||||
`fluksio-worker` already in it. The generated script runs no `pip install`.
|
||||
It hands the worker a token of its own, good for 24 hours, as
|
||||
`FLUKSIO_WORKER_TOKEN` rather than `--token`, so other users of a shared
|
||||
compute node cannot read it from `ps`.
|
||||
|
||||
One outstanding request per profile, however often it is asked for. A job that
|
||||
never attaches within `provision_timeout_s` is `scancel`led, as is anything
|
||||
outstanding when the engine stops. `--max-idle` is what ends the job at the
|
||||
other end, so an allocation goes back rather than idling to its walltime.
|
||||
|
||||
Both take `0` for "no limit": `provision_timeout_s: 0` waits for as long as the
|
||||
queue does, which is what a cluster that queues overnight needs, and
|
||||
`max_idle_s: 0` keeps the machine for the job's whole walltime.
|
||||
|
||||
`GET /workers/resources` reports what is outstanding and what last went wrong.
|
||||
|
||||
!!! note "It needs a route out"
|
||||
|
||||
A compute node must be able to open a connection to the engine. That is
|
||||
true of most clusters and false of air-gapped ones; there is no staging
|
||||
path today.
|
||||
|
||||
## What a worker is not
|
||||
|
||||
It is not a second engine. Subscriptions, schedules, webhooks, the dashboards
|
||||
and the run queue all stay in one process, which keeps a message having one
|
||||
definition and a cron tick happening once. A worker executes node bodies.
|
||||
|
||||
Running two engines against one data directory is not supported. Distribute
|
||||
work with workers.
|
||||
Only one request per profile is outstanding at a time. Jobs that fail to attach
|
||||
within `provision_timeout_s` are cancelled, as are pending requests when the
|
||||
engine shuts down. `max_idle_s` releases idle workers. Either setting accepts
|
||||
`0` to disable its limit. Provisioning status appears under
|
||||
`GET /api/v1/workers/resources`.
|
||||
|
||||
## See also
|
||||
|
||||
- [Runs: pipelines that finish](../concepts/runs.md#running-a-node-somewhere-else)
|
||||
- [Writing node code](nodes.md)
|
||||
- [Getting started: data science](../getting-started/data-science.md)
|
||||
- [Runs](../concepts/runs.md#running-a-node-somewhere-else): remote execution.
|
||||
- [Writing node code](nodes.md): node functions and artifact helpers.
|
||||
- [Getting started: data science](../getting-started/data-science.md): resource declarations.
|
||||
|
||||
Reference in New Issue
Block a user