app/docs/code/workers.md

# Remote workers

A worker runs node code on another machine: a GPU host, a Raspberry Pi or a
batch allocation. It connects outward to the engine over an authenticated
WebSocket. Keep one engine; add workers where the hardware lives.

## Create a worker token

On the engine, use a superuser access token. In these examples:

- `API_URL` is the engine's API base, including `/api/v1`, such as
  `https://fluksio.example.com/api/v1` or `http://localhost:8080/api/v1` for a Docker container.
- `TOKEN` is your superuser bearer token. The login endpoint
  `POST /api/v1/login/access-token` accepts form fields `username` and `password`.

Mint a token with the labels this machine should carry:

```sh
curl -X POST "$API_URL/workers/tokens" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}'
```

The response contains `token`, `jti` and the expiry. Save the token as
`FLUKSIO_WORKER_TOKEN` on the worker host. It is returned once and is valid
for 365 days. The signed name and labels control which nodes and secrets the
engine sends to that worker.

## Install and attach

Use Python 3.12 or later on the worker host:

```sh
pip install fluksio-worker
export FLUKSIO_WORKER_TOKEN='<token from the engine>'

fluksio-worker \
  --url wss://fluksio.example.com/api/v1/workers/attach \
  --labels gpu,cuda12 \
  --python /opt/torch-venv/bin/python
```

Use your engine's hostname. `--python` selects the environment containing the
node's dependencies, such as PyTorch or a device SDK. If you install the worker
in that environment, you can omit the option. An engine installation also
provides the same agent as `fluksio worker`.

| Option | Default | Purpose |
|---|---|---|
| `--url` | required | engine's `/api/v1/workers/attach` WebSocket |
| `--token` | `$FLUKSIO_WORKER_TOKEN` | credential; prefer the environment variable to keep it out of process arguments |
| `--name` | hostname | suggested name; the engine uses the token's name |
| `--labels` | token's labels | comma-separated; when supplied, must match the token |
| `--python` | worker's interpreter | Python used for node code |
| `--parallel` | `1` | concurrent calls |
| `--artifact-url` | derived from `--url` | artifact endpoint, if hosted separately |
| `--service-host` | address the engine sees | host in a service's url, when the engine reaches this machine elsewhere, such as through a tunnel |
| `--cpus` | detected | advertised CPU count |
| `--gpus` | detected | advertised GPU count; explicit `0` means none |
| `--ram-mb` | detected | advertised memory in MB |
| `--max-idle` | `0` | exit after this many idle seconds; `0` disables |
| `--insecure` | off | allow unencrypted connections to a non-loopback host |

For a trusted LAN stack, the URL is
`ws://<host-ip>:8080/api/v1/workers/attach` and requires `--insecure`.
Outside loopback, the worker otherwise requires encrypted URLs (`wss://` and
`https://`): node secrets and the worker token travel over the connection.

Node code receives a filtered host environment. Credential-like variables are
removed; pass credentials through [Secrets](../interface/operations.md#secrets).

## Send a node to it

Set these fields on a node through `PUT /api/v1/flows/{name}`:

```json
{
  "id": "train",
  "worker": "gpu",
  "worker_policy": "require",
  "timeout": 7200
}
```

This is a node fragment; keep the node's other fields and the rest of the flow
when updating it. `worker` and `worker_policy` are not in the node panel yet.

| `worker_policy` | If no matching worker is attached |
|---|---|
| `require` (default) | queues until a matching worker is available |
| `prefer` | falls back to the engine |

Source code travels with each call and compiles on the worker, so remote-only
dependencies do not need to be installed on the engine. Calls to `emit`,
`save_artifact` and `load_artifact` work there too.

Cancelling a run stops its active node processes. A disconnected worker fails
in-flight calls. While executing, the agent sends heartbeats every ten seconds;
90 seconds without contact fails the call. Heartbeats do not reset the node's
own silence timeout.

Every call runs in a process group of its own. The agent kills the group when
the engine ends the call, when the connection drops and when the agent stops,
so nothing a node started keeps running. If the agent itself is killed, its
service manager has to clean up: systemd's default `KillMode=control-group`
does.

## Services

A [Service node](../reference/node-types.md#service) runs a server on a worker
for as long as its flow runs, such as a speech model that should stay loaded.
It is one long call that holds none of the worker's `--parallel` slots and no
resources, so it does not block calls or hold a GPU against a training run.
The engine checks the server's health over HTTP and restarts it whenever it
stops, and its url names the address the engine sees the worker connect from.
Set `--service-host` on the worker where that address is wrong.

A service's `weights` setting holds an artifact digest. The worker fetches it
as node code fetches an artifact and passes the local path as `{weights}`.
The hourly artifact sweep keeps any digest a node setting names in a published
or draft flow, so the weights stay after the run that produced them is deleted.

## Resources and artifacts

Workers report CPU, GPU and memory inventory, and scheduling checks both labels
and available resources. CPU and memory detection uses Slurm allocations when
present, otherwise host figures. GPU detection uses `SLURM_GPUS_ON_NODE`,
`FLUKSIO_WORKER_GPUS` or `SLURM_JOB_GPUS` before counting NVIDIA device nodes.
Explicit flags override detection.

Artifacts are identified by content hash. Remote workers fetch and upload over
HTTP beside the WebSocket endpoint; node code uses the same artifact helpers
as on the engine. Downloads are cached by digest:

| Variable | Default |
|---|---|
| `FLUKSIO_ARTIFACT_CACHE` | `fluksio-artifacts` in the temporary directory |
| `FLUKSIO_ARTIFACT_CACHE_BYTES` | `1073741824` (1 GiB) |

Oldest files are removed when the cache exceeds its limit. Raise it if workers
repeatedly use large inputs.

The **Workers** screen shows attached hosts, available resources and queued
nodes. The API exposes the same information:

```sh
curl "$API_URL/workers" -H "Authorization: Bearer $TOKEN"
curl "$API_URL/workers/resources" -H "Authorization: Bearer $TOKEN"
```

## Revoke a token

List tokens and revoke one by its `jti`:

```sh
curl "$API_URL/workers/tokens" -H "Authorization: Bearer $TOKEN"
curl -X POST "$API_URL/workers/tokens/$JTI/revoke" \
  -H "Authorization: Bearer $TOKEN"
```

Revocation also disconnects attached workers and blocks artifact access.
Replacing the OAuth signing key revokes all worker tokens. Older tokens without
signed labels and a `jti` need to be replaced.

## Upgrading workers

Engine and worker must speak the same protocol. A worker on another protocol is
refused and exits, so a release that bumps the protocol needs
`pip install -U fluksio-worker` on every worker host. Upgrade the engine first,
then the workers.

The **Workers** screen shows each worker's package version and protocol. A
worker refused for an older protocol stays listed as **Needs upgrade** until it
attaches again, or until the engine restarts.

## Slurm provisioning

The engine can submit a Slurm job that starts a worker. Put this configuration
in `$DATA_DIR/provisioners.json`, replacing the example addresses and paths:

```json
[{
  "type": "slurm",
  "name": "hpc",
  "login": "me@login.cluster",
  "ssh_key": "/secrets/hpc_ed25519",
  "engine_url": "wss://fluksio.example.com/api/v1/workers/attach",
  "max_idle_s": 300,
  "provision_timeout_s": 900,
  "profiles": [{
    "name": "gpu-small",
    "cpus": 8,
    "gpus": 1,
    "ram_mb": 65536,
    "labels": ["gpu"],
    "sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"],
    "prerun": ["module load cuda/12", "source ~/venvs/fluksio/bin/activate"]
  }]
}]
```

The engine uses system SSH to submit `sbatch`. Prepare the environment named
by `prerun` with `fluksio-worker` and your node dependencies; the generated job
does not install packages. Compute nodes must be able to connect to the
engine. Each job receives its own worker token, valid for 24 hours.

Only one request per profile is outstanding at a time. Jobs that fail to attach
within `provision_timeout_s` are cancelled, as are pending requests when the
engine shuts down. `max_idle_s` releases idle workers. Either setting accepts
`0` to disable its limit. Provisioning status appears under
`GET /api/v1/workers/resources`.

## See also

- [Runs](../concepts/runs.md#running-a-node-somewhere-else): remote execution.
- [Writing node code](nodes.md): node functions and artifact helpers.
- [Getting started: data science](../getting-started/data-science.md): resource declarations.