Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs
Docs / docs (push) Successful in 26s
Details
Playwright Tests / test-playwright (1, 2) (push) Failing after 4m7s
Details
Playwright Tests / test-playwright (2, 2) (push) Failing after 2m30s
Details
pre-commit / pre-commit (push) Failing after 2m42s
Details
Test Backend / test-backend (push) Failing after 3m3s
Details
Compose Smoke Test / test-compose (push) Successful in 32s
Details
Playwright Tests / merge-reports (push) Successful in 1m51s
Details

Browse Source
This commit is contained in:
stroblme
2026-10-13 10:02:20 +02:00
parent 8d97b8ffaa
commit b2aa932b6d
24 changed files with 2210 additions and 4310 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/code/workers.md
+120 -207
View File
@@ -1,119 +1,77 @@
# Remote workers
The engine runs where the automations are. The GPU is somewhere else, the
Raspberry Pi with the relays is in a shed, and neither of them is on the same
network as the other.
A worker runs node code on another machine: a GPU host, a Raspberry Pi or a
batch allocation. It connects outward to the engine over an authenticated
WebSocket. Keep one engine; add workers where the hardware lives.
A **worker** is a process that runs the code of nodes marked for it. It dials
*out* to the engine over one authenticated websocket, so nothing on that
machine has to be reachable, and nothing has to expose the engine's state
backend across hosts, which it never should.
## Create a worker token
## Install and attach
On the engine, use a superuser access token. In these examples:
- `API_URL` is the engine's API base, including `/api/v1`, such as
`https://api.example.com/api/v1` or `http://localhost:8080/api/v1` for a LAN stack.
- `TOKEN` is your superuser bearer token. The login endpoint
`POST /api/v1/login/access-token` accepts form fields `username` and `password`.
Mint a token with the labels this machine should carry:
```sh
pip install fluksio-worker
fluksio-worker \
--url wss://api.${DOMAIN}/api/v1/workers/attach \
--labels gpu,cuda12 \
--python /opt/torch-venv/bin/python
```
`fluksio-worker` is its own distribution: the agent, the node runner, and
`websockets`. Nothing of the engine, so a GPU box does not install a database
driver in order to run a training step. An engine host already has it, and
`fluksio worker …` is the same program.
| Option | Default | What it does |
|---|---|---|
| `--url` | **required** | `wss://…/api/v1/workers/attach` |
| `--token` | `$FLUKSIO_WORKER_TOKEN` | the credential, minted on the engine; on the command line any user of the machine reads it from `ps`, so prefer the variable |
| `--name` | this host's name | sent along; the worker list shows the token's name |
| `--labels` | the token's | comma-separated; must match the labels the token was minted with |
| `--python` | this interpreter | the interpreter node code runs on |
| `--parallel` | `1` | how many node calls it will take at once |
| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |
| `--cpus` | what the job or the machine has | cores to advertise |
| `--gpus` | what the job says, else counted | GPUs to advertise |
| `--ram-mb` | what the job or the machine has | memory to advertise, in MB |
| `--max-idle` | never | stop after this many seconds with nothing running |
| `--insecure` | off | allow `ws://` and `http://` to an engine on another machine |
The worker refuses a plain `ws://` or `http://` address unless the engine is on
this machine (`localhost`, `127.0.0.0/8`, `::1`): its token and every secret a
node is handed would cross the network in clear. `--insecure` is for a network
you trust, such as an engine on the same LAN without TLS. Node code gets this
host's environment less anything named like a credential (`*TOKEN*`,
`*SECRET*`, `*PASSWORD*`), the same as on the engine; hand a node a credential
as a [secret](../interface/operations.md#secrets) instead.
`--python` is the important one. It is how this machine keeps its own wheels
(the CUDA build, the vendor SDK, the thing that will not install anywhere else)
without the engine ever installing them or knowing about them.
### What it says it has
A worker reports its inventory when it attaches (cores, GPUs and memory) and
the engine schedules against it: a node asking for two cores and a GPU goes to
a machine that has them free, not merely to one carrying the right label.
Cores and memory are read off the machine, or off the batch job that started
this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **What the job says
it was given always wins for GPUs** (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`,
or `FLUKSIO_WORKER_GPUS`): a node with eight cards may have granted this job
one, and advertising eight would be a lie the scheduler acts on. With nothing
said, NVIDIA's device nodes are counted, the same as the engine does for its
own machine, and `--gpus` overrides either. No vendor tool is asked, which is
what keeps the one dependency from becoming two. A worker that reports nothing
still attaches and is scheduled by its label alone, as every worker was before
any of them reported anything.
The engine tells each call what it may use: thread caps, and the devices it
may see. The worker starts a process per call, so it applies them at the only
moment a numerical library still reads them: before the import.
`--max-idle` is for a worker something else started for one job, a batch
scheduler say. It exits when nothing has run for that long, so the allocation
goes back rather than idling until its walltime.
## Mint the token
On the engine, as a superuser:
```sh
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
curl -X POST "$API_URL/workers/tokens" \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}'
```
Shown once, valid for a year, since a worker is a machine somebody sets up and
leaves running. The name and labels are signed into it: labels decide which
nodes, with their source and secrets, are sent to a worker, so a worker cannot
choose its own. It shows up under the token's name, and a worker whose
`--labels` differ from the token's is refused.
The response contains `token`, `jti` and the expiry. Save the token as
`FLUKSIO_WORKER_TOKEN` on the worker host. It is returned once and is valid
for 365 days. The signed name and labels control which nodes and secrets the
engine sends to that worker.
## Install and attach
Use Python 3.12 or later on the worker host:
```sh
curl -s $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" | jq
curl -X POST $FLUKSIO/workers/tokens/$JTI/revoke -H "Authorization: Bearer $TOKEN"
pip install fluksio-worker
export FLUKSIO_WORKER_TOKEN='<token from the engine>'
fluksio-worker \
--url wss://api.example.com/api/v1/workers/attach \
--labels gpu,cuda12 \
--python /opt/torch-venv/bin/python
```
The first lists what was minted, by `jti` (the tokens themselves are not
kept); the second refuses one from then on, here and at the artifact
endpoint, and drops a worker attached with it. Rotating the keypair agent
tokens are signed with still revokes every worker at once. A token minted
before tokens carried their labels is refused: mint a new one.
Use your engine's hostname. `--python` selects the environment containing the
node's dependencies, such as PyTorch or a device SDK. If you install the worker
in that environment, you can omit the option. An engine installation also
provides the same agent as `fluksio worker`.
??? note "A host where pip is not an option"
| Option | Default | Purpose |
|---|---|---|
| `--url` | required | engine's `/api/v1/workers/attach` WebSocket |
| `--token` | `$FLUKSIO_WORKER_TOKEN` | credential; prefer the environment variable to keep it out of process arguments |
| `--name` | hostname | suggested name; the engine uses the token's name |
| `--labels` | token's labels | comma-separated; when supplied, must match the token |
| `--python` | worker's interpreter | Python used for node code |
| `--parallel` | `1` | concurrent calls |
| `--artifact-url` | derived from `--url` | artifact endpoint, if hosted separately |
| `--cpus` | detected | advertised CPU count |
| `--gpus` | detected | advertised GPU count; explicit `0` means none |
| `--ram-mb` | detected | advertised memory in MB |
| `--max-idle` | `0` | exit after this many idle seconds; `0` disables |
| `--insecure` | off | allow unencrypted connections to a non-loopback host |
The two files work copied into one directory and run with `python agent.py
…`. The engine serves the runner itself at `GET /api/v1/workers/runtime`.
It is the same module its own local workers run, deliberately standard
library only.
For a trusted LAN stack, the URL is
`ws://<host-ip>:8080/api/v1/workers/attach` and requires `--insecure`.
Outside loopback, the worker otherwise requires encrypted URLs (`wss://` and
`https://`): node secrets and the worker token travel over the connection.
Node code receives a filtered host environment. Credential-like variables are
removed; pass credentials through [Secrets](../interface/operations.md#secrets).
## Send a node to it
A node declares the label of the machine it needs:
Set these fields on a node through `PUT /api/v1/flows/{name}`:
```json
{
@@ -124,94 +82,73 @@ A node declares the label of the machine it needs:
}
```
| `device_policy` | Behaviour when nothing carrying the label is attached |
This is a node fragment; keep the node's other fields and the rest of the flow
when updating it. `device` and `device_policy` are not in the node panel yet.
| `device_policy` | If no matching worker is attached |
|---|---|
| `require` (default) | the run stays `queued` and says what it is waiting for |
| `prefer` | it runs on the engine instead |
| `require` (default) | queues until a matching worker is available |
| `prefer` | falls back to the engine |
`prefer` is what makes a flow work before the GPU box exists. `require` is what
you want once it does.
Source code travels with each call and compiles on the worker, so remote-only
dependencies do not need to be installed on the engine. Calls to `emit`,
`save_artifact` and `load_artifact` work there too.
!!! note "Not in the panel yet"
Cancelling a run stops its active node processes. A disconnected worker fails
in-flight calls. While executing, the agent sends heartbeats every ten seconds;
90 seconds without contact fails the call. Heartbeats do not reset the node's
own silence timeout.
`device` and `device_policy`, which machine a node's code runs on, are set
through the API rather than the panel, with `PUT /flows/{name}`.
## Resources and artifacts
## What follows from this
Workers report CPU, GPU and memory inventory, and scheduling checks both labels
and available resources. CPU and memory detection uses Slurm allocations when
present, otherwise host figures. GPU detection uses `SLURM_GPUS_ON_NODE`,
`FLUKSIO_WORKER_GPUS` or `SLURM_JOB_GPUS` before counting NVIDIA device nodes.
Explicit flags override detection.
- **The node's source travels with every call.** Nothing has to be deployed to
the worker, and changing a node's code takes effect on the next execution.
- **A node bound to a device is compiled on that machine.** A node importing
`torch` is correct on the GPU box and a missing module on the engine, so
checking it here would fail something that is fine.
- **`import fluksio` inside a node is the worker's own reporter.** `emit`,
`save_artifact`, `load_artifact`, installed before your code runs, so an
installed `fluksio` package on that box never shadows it.
- **Cancelling a run kills what it is executing**, there or here, and leaves
other runs of the same node alone.
- **If the worker disappears mid-call**, the run fails in seconds with
`worker went away mid-call` rather than waiting out its timeout.
- **A worker sends a heartbeat every ten seconds while it executes**, so a long
node is distinguishable from a dead socket. Ninety seconds of nothing at all,
not even a heartbeat, fails the call as gone. A heartbeat says the *agent* is
alive and nothing about the node, so it never satisfies a node's own timeout:
one set to thirty seconds fires after thirty seconds of the node reporting
nothing, wherever it runs.
Artifacts are identified by content hash. Remote workers fetch and upload over
HTTP beside the WebSocket endpoint; node code uses the same artifact helpers
as on the engine. Downloads are cached by digest:
## Artifacts across machines
| Variable | Default |
|---|---|
| `FLUKSIO_ARTIFACT_CACHE` | `fluksio-artifacts` in the temporary directory |
| `FLUKSIO_ARTIFACT_CACHE_BYTES` | `1073741824` (1 GiB) |
An artifact reference names content by its hash, not a location, so it stays
valid wherever the store is reachable from. A worker that shares the engine's
filesystem writes to it directly; one that does not fetches and uploads over
HTTP, using the artifact endpoint beside the socket it already has. Either way
your node code is the same two calls.
Oldest files are removed when the cache exceeds its limit. Raise it if workers
repeatedly use large inputs.
A fetch is cached on the worker by digest, since content addressing means an
entry is never stale. Nothing expires on its own, so the cache is bounded by
size and the oldest fall out: `FLUKSIO_ARTIFACT_CACHE` says where it lives
(default a directory in the temporary directory) and
`FLUKSIO_ARTIFACT_CACHE_BYTES` how much it holds (default 1 GiB). Worth raising
where a worker reads the same large inputs repeatedly, and worth leaving alone
where it reads a media stream — those are chunks nothing asks for twice.
## Seeing what is attached
The **Workers** screen shows attached hosts, available resources and queued
nodes. The API exposes the same information:
```sh
curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq
curl "$API_URL/workers" -H "Authorization: Bearer $TOKEN"
curl "$API_URL/workers/resources" -H "Authorization: Bearer $TOKEN"
```
Name, labels, how many calls it will take at once, how many are in flight, when
it attached, when it was last seen, its Python version, a digest of its
environment, and what it says it has: cores, GPUs and memory.
## Revoke or upgrade
## Upgrading
List tokens and revoke one by its `jti`:
The engine and the worker speak a version-matched protocol, and a worker
announcing anything else is refused rather than half-understood. Protocol 2,
the one that carries inventory, is `fluksio-worker` 0.2.0. An older agent is
told so on the socket and stops, rather than retrying against an engine that
will never accept it; `pip install -U fluksio-worker` on that host is the whole
upgrade. Nothing changed in the runner served at `GET /api/v1/workers/runtime`,
so a host that copies its two files copies the same one as before.
```sh
curl "$API_URL/workers/tokens" -H "Authorization: Bearer $TOKEN"
curl -X POST "$API_URL/workers/tokens/$JTI/revoke" \
-H "Authorization: Bearer $TOKEN"
```
A token minted before tokens carried their labels and a `jti` is refused. Mint
each existing worker host a new one with `POST /workers/tokens`, naming the
labels it should carry, and swap it in for `FLUKSIO_WORKER_TOKEN`.
Revocation also disconnects attached workers and blocks artifact access.
Replacing the OAuth signing key revokes all worker tokens.
**Upgrade the engine before its workers.** A worker sends its token as an
`Authorization` header on the handshake, where it stays out of access logs, and
an engine from before that reads it only from the URL: it refuses the newer
worker at the handshake. The engine still takes the old `?token=` for one
release, so older workers keep attaching to a newer engine meanwhile.
Upgrade the engine before its workers, then run `pip install -U fluksio-worker`
on each host. Engine and worker must speak the same protocol; a mismatch is
reported and refused. Older tokens without signed labels and a `jti` need to
be replaced.
## Machines from a batch scheduler
## Slurm provisioning
A cluster is not a machine that attaches and stays; it is a queue somebody else
owns. So Fluksio does not submit *nodes* to Slurm. It submits a job whose payload
is an ordinary worker dialling back in, and from there everything works the way
it already does: the same protocol, the same artifacts, the same cancellation.
Write the clusters into `provisioners.json` beside the flows:
The engine can submit a Slurm job that starts a worker. Put this configuration
in `$DATA_DIR/provisioners.json`, replacing the example addresses and paths:
```json
[{
@@ -219,58 +156,34 @@ Write the clusters into `provisioners.json` beside the flows:
"name": "hpc",
"login": "me@login.cluster",
"ssh_key": "/secrets/hpc_ed25519",
"engine_url": "wss://api.${DOMAIN}/api/v1/workers/attach",
"engine_url": "wss://api.example.com/api/v1/workers/attach",
"max_idle_s": 300,
"provision_timeout_s": 900,
"profiles": [{
"name": "gpu-small",
"cpus": 8, "gpus": 1, "ram_mb": 65536,
"cpus": 8,
"gpus": 1,
"ram_mb": 65536,
"labels": ["gpu"],
"sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"],
"prerun": ["module load cuda/12", "source ~/venvs/flux/bin/activate"]
"prerun": ["module load cuda/12", "source ~/venvs/fluksio/bin/activate"]
}]
}]
```
A **profile** is what the scheduler is asked for, where a flavor is what a node
asks for. They are separate on purpose, and agree when you set them up to.
The engine uses system SSH to submit `sbatch`. Prepare the environment named
by `prerun` with `fluksio-worker` and your node dependencies; the generated job
does not install packages. Compute nodes must be able to connect to the
engine. Each job receives its own worker token, valid for 24 hours.
When a node needs a machine nothing attached can give, and a profile fits, the
engine `sbatch`es one over the system `ssh`, and the run waits meanwhile, saying
so. `prerun` owns the environment: a `module load`, or a venv with
`fluksio-worker` already in it. The generated script runs no `pip install`.
It hands the worker a token of its own, good for 24 hours, as
`FLUKSIO_WORKER_TOKEN` rather than `--token`, so other users of a shared
compute node cannot read it from `ps`.
One outstanding request per profile, however often it is asked for. A job that
never attaches within `provision_timeout_s` is `scancel`led, as is anything
outstanding when the engine stops. `--max-idle` is what ends the job at the
other end, so an allocation goes back rather than idling to its walltime.
Both take `0` for "no limit": `provision_timeout_s: 0` waits for as long as the
queue does, which is what a cluster that queues overnight needs, and
`max_idle_s: 0` keeps the machine for the job's whole walltime.
`GET /workers/resources` reports what is outstanding and what last went wrong.
!!! note "It needs a route out"
A compute node must be able to open a connection to the engine. That is
true of most clusters and false of air-gapped ones; there is no staging
path today.
## What a worker is not
It is not a second engine. Subscriptions, schedules, webhooks, the dashboards
and the run queue all stay in one process, which keeps a message having one
definition and a cron tick happening once. A worker executes node bodies.
Running two engines against one data directory is not supported. Distribute
work with workers.
Only one request per profile is outstanding at a time. Jobs that fail to attach
within `provision_timeout_s` are cancelled, as are pending requests when the
engine shuts down. `max_idle_s` releases idle workers. Either setting accepts
`0` to disable its limit. Provisioning status appears under
`GET /api/v1/workers/resources`.
## See also
- [Runs: pipelines that finish](../concepts/runs.md#running-a-node-somewhere-else)
- [Writing node code](nodes.md)
- [Getting started: data science](../getting-started/data-science.md)
- [Runs](../concepts/runs.md#running-a-node-somewhere-else): remote execution.
- [Writing node code](nodes.md): node functions and artifact helpers.
- [Getting started: data science](../getting-started/data-science.md): resource declarations.