Publish the documentation site: docs.fluksio.com
A zensical site under docs/, served by a new `docs` compose service behind Traefik, built with --strict in CI. Same pattern the sibling n3xd workspace uses. Getting started splits the way the landing page does — one path is `pip install fluksio` and a training script, the other is a Docker stack and an afternoon in the browser — because the two audiences will not spend the same amount of time. Everything after that is shared: the concepts, the web interface (app and portal), the CLI and the API, and a reference for node types, payload types and configuration. The three flow guides move here from the docs submodule rather than being copied, so there is one version of them. Styling mirrors DESIGN-GUIDELINES.md: the app's token palette remapped onto Material's variables in both schemes, Inter, the 16px panel radius, and the one terracotta accent spent on the facility lane of the audience split. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M7Xv3cJEW5c8AXxn2hoojV
This commit is contained in:
@@ -0,0 +1,137 @@
|
||||
# Remote workers
|
||||
|
||||
The engine runs where the automations are. The GPU is somewhere else, the
|
||||
Raspberry Pi with the relays is in a shed, and neither of them is on the same
|
||||
network as the other.
|
||||
|
||||
A **worker** is a process that runs the code of nodes marked for it. It dials
|
||||
*out* to the engine over one authenticated websocket, so nothing on that
|
||||
machine has to be reachable — and nothing has to expose the engine's state
|
||||
backend across hosts, which it never should.
|
||||
|
||||
## Install and attach
|
||||
|
||||
```sh
|
||||
pip install fluksio-worker
|
||||
|
||||
fluksio-worker \
|
||||
--url wss://api.example.com/api/v1/workers/attach \
|
||||
--token "$FLUKSIO_WORKER_TOKEN" \
|
||||
--labels gpu,cuda12 \
|
||||
--python /opt/torch-venv/bin/python
|
||||
```
|
||||
|
||||
`fluksio-worker` is its own distribution — the agent, the node runner, and
|
||||
`websockets`. Nothing of the engine, so a GPU box does not install a database
|
||||
driver in order to run a training step. An engine host already has it, and
|
||||
`fluksio worker …` is the same program.
|
||||
|
||||
| Option | Default | What it does |
|
||||
|---|---|---|
|
||||
| `--url` | **required** | `wss://…/api/v1/workers/attach` |
|
||||
| `--token` | `$FLUKSIO_WORKER_TOKEN` | the credential, minted on the engine |
|
||||
| `--name` | this host's name | how it shows up in the worker list |
|
||||
| `--labels` | none | comma-separated; what a node's `device` matches |
|
||||
| `--python` | this interpreter | the interpreter node code runs on |
|
||||
| `--parallel` | `1` | how many node calls it will take at once |
|
||||
| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |
|
||||
|
||||
`--python` is the important one. It is how this machine keeps its own wheels —
|
||||
the CUDA build, the vendor SDK, the thing that will not install anywhere else —
|
||||
without the engine ever installing them or knowing about them.
|
||||
|
||||
## Mint the token
|
||||
|
||||
On the engine, as a superuser:
|
||||
|
||||
```sh
|
||||
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
|
||||
-H 'Content-Type: application/json' -d '{"name": "gpu-dev"}'
|
||||
```
|
||||
|
||||
Shown once, valid for a year — a worker is a machine somebody sets up and
|
||||
leaves running. It is signed with the same keypair agent tokens use, so
|
||||
rotating that key revokes every worker along with them.
|
||||
|
||||
??? note "A host where pip is not an option"
|
||||
|
||||
The two files work copied into one directory and run with `python agent.py
|
||||
…`. The engine serves the runner itself at `GET /api/v1/workers/runtime` —
|
||||
it is the same module its own local workers run, deliberately standard
|
||||
library only.
|
||||
|
||||
## Send a node to it
|
||||
|
||||
A node declares the label of the machine it needs:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "train",
|
||||
"device": "gpu",
|
||||
"device_policy": "require",
|
||||
"timeout": 7200
|
||||
}
|
||||
```
|
||||
|
||||
| `device_policy` | Behaviour when nothing carrying the label is attached |
|
||||
|---|---|
|
||||
| `require` (default) | the run stays `queued` and says what it is waiting for |
|
||||
| `prefer` | it runs on the engine instead |
|
||||
|
||||
`prefer` is what makes a flow work before the GPU box exists. `require` is what
|
||||
you want once it does.
|
||||
|
||||
!!! note "Set from the API"
|
||||
|
||||
`device` and `device_policy` are not yet fields in the node panel. Set them
|
||||
with `PUT /flows/{name}`.
|
||||
|
||||
## What follows from this
|
||||
|
||||
- **The node's source travels with every call.** Nothing has to be deployed to
|
||||
the worker, and changing a node's code takes effect on the next execution.
|
||||
- **A node bound to a device is compiled on that machine.** A node importing
|
||||
`torch` is correct on the GPU box and a missing module on the engine, so
|
||||
checking it here would fail something that is fine.
|
||||
- **`import fluksio` inside a node is the worker's own reporter.** `emit`,
|
||||
`save_artifact`, `load_artifact` — installed before your code runs, so an
|
||||
installed `fluksio` package on that box never shadows it.
|
||||
- **Cancelling a run kills what it is executing**, there or here, and leaves
|
||||
other runs of the same node alone.
|
||||
- **If the worker disappears mid-call**, the run fails in seconds with
|
||||
`worker went away mid-call` rather than waiting out its timeout.
|
||||
- **A worker sends a heartbeat while it executes**, so a long node is
|
||||
distinguishable from a dead socket. Ninety seconds of silence is gone.
|
||||
|
||||
## Artifacts across machines
|
||||
|
||||
An artifact reference names content by its hash, not a location, so it stays
|
||||
valid wherever the store is reachable from. A worker that shares the engine's
|
||||
filesystem writes to it directly; one that does not fetches and uploads over
|
||||
HTTP, using the artifact endpoint beside the socket it already has. Either way
|
||||
your node code is the same two calls.
|
||||
|
||||
## Seeing what is attached
|
||||
|
||||
```sh
|
||||
curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq
|
||||
```
|
||||
|
||||
Name, labels, how many calls it will take at once, how many are in flight, when
|
||||
it attached, when it was last seen, its Python version, and a digest of its
|
||||
environment.
|
||||
|
||||
## What a worker is not
|
||||
|
||||
It is not a second engine. Subscriptions, schedules, webhooks, the dashboards
|
||||
and the run queue all stay in one process — that is what keeps a message having
|
||||
one definition and a cron tick happening once. A worker executes node bodies.
|
||||
|
||||
Running two engines against one data directory is not supported. Distribute
|
||||
work with workers.
|
||||
|
||||
## See also
|
||||
|
||||
- [Runs: pipelines that finish](../concepts/runs.md#running-a-node-somewhere-else)
|
||||
- [Writing node code](nodes.md)
|
||||
- [Getting started: data science](../getting-started/data-science.md)
|
||||
Reference in New Issue
Block a user