# Remote workers A worker runs node code on another machine: a GPU host, a Raspberry Pi or a batch allocation. It connects outward to the engine over an authenticated WebSocket. Keep one engine; add workers where the hardware lives. ## Create a worker token On the engine, use a superuser access token. In these examples: - `API_URL` is the engine's API base, including `/api/v1`, such as `https://fluksio.example.com/api/v1` or `http://localhost:8080/api/v1` for a Docker container. - `TOKEN` is your superuser bearer token. The login endpoint `POST /api/v1/login/access-token` accepts form fields `username` and `password`. Mint a token with the labels this machine should carry: ```sh curl -X POST "$API_URL/workers/tokens" \ -H "Authorization: Bearer $TOKEN" \ -H 'Content-Type: application/json' \ -d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}' ``` The response contains `token`, `jti` and the expiry. Save the token as `FLUKSIO_WORKER_TOKEN` on the worker host. It is returned once and is valid for 365 days. The signed name and labels control which nodes and secrets the engine sends to that worker. ## Install and attach Use Python 3.12 or later on the worker host: ```sh pip install fluksio-worker export FLUKSIO_WORKER_TOKEN='' fluksio-worker \ --url wss://fluksio.example.com/api/v1/workers/attach \ --labels gpu,cuda12 \ --python /opt/torch-venv/bin/python ``` Use your engine's hostname. `--python` selects the environment containing the node's dependencies, such as PyTorch or a device SDK. If you install the worker in that environment, you can omit the option. An engine installation also provides the same agent as `fluksio worker`. | Option | Default | Purpose | |---|---|---| | `--url` | required | engine's `/api/v1/workers/attach` WebSocket | | `--token` | `$FLUKSIO_WORKER_TOKEN` | credential; prefer the environment variable to keep it out of process arguments | | `--name` | hostname | suggested name; the engine uses the token's name | | `--labels` | token's labels | comma-separated; when supplied, must match the token | | `--python` | worker's interpreter | Python used for node code | | `--parallel` | `1` | concurrent calls | | `--artifact-url` | derived from `--url` | artifact endpoint, if hosted separately | | `--cpus` | detected | advertised CPU count | | `--gpus` | detected | advertised GPU count; explicit `0` means none | | `--ram-mb` | detected | advertised memory in MB | | `--max-idle` | `0` | exit after this many idle seconds; `0` disables | | `--insecure` | off | allow unencrypted connections to a non-loopback host | For a trusted LAN stack, the URL is `ws://:8080/api/v1/workers/attach` and requires `--insecure`. Outside loopback, the worker otherwise requires encrypted URLs (`wss://` and `https://`): node secrets and the worker token travel over the connection. Node code receives a filtered host environment. Credential-like variables are removed; pass credentials through [Secrets](../interface/operations.md#secrets). ## Send a node to it Set these fields on a node through `PUT /api/v1/flows/{name}`: ```json { "id": "train", "device": "gpu", "device_policy": "require", "timeout": 7200 } ``` This is a node fragment; keep the node's other fields and the rest of the flow when updating it. `device` and `device_policy` are not in the node panel yet. | `device_policy` | If no matching worker is attached | |---|---| | `require` (default) | queues until a matching worker is available | | `prefer` | falls back to the engine | Source code travels with each call and compiles on the worker, so remote-only dependencies do not need to be installed on the engine. Calls to `emit`, `save_artifact` and `load_artifact` work there too. Cancelling a run stops its active node processes. A disconnected worker fails in-flight calls. While executing, the agent sends heartbeats every ten seconds; 90 seconds without contact fails the call. Heartbeats do not reset the node's own silence timeout. ## Resources and artifacts Workers report CPU, GPU and memory inventory, and scheduling checks both labels and available resources. CPU and memory detection uses Slurm allocations when present, otherwise host figures. GPU detection uses `SLURM_GPUS_ON_NODE`, `FLUKSIO_WORKER_GPUS` or `SLURM_JOB_GPUS` before counting NVIDIA device nodes. Explicit flags override detection. Artifacts are identified by content hash. Remote workers fetch and upload over HTTP beside the WebSocket endpoint; node code uses the same artifact helpers as on the engine. Downloads are cached by digest: | Variable | Default | |---|---| | `FLUKSIO_ARTIFACT_CACHE` | `fluksio-artifacts` in the temporary directory | | `FLUKSIO_ARTIFACT_CACHE_BYTES` | `1073741824` (1 GiB) | Oldest files are removed when the cache exceeds its limit. Raise it if workers repeatedly use large inputs. The **Workers** screen shows attached hosts, available resources and queued nodes. The API exposes the same information: ```sh curl "$API_URL/workers" -H "Authorization: Bearer $TOKEN" curl "$API_URL/workers/resources" -H "Authorization: Bearer $TOKEN" ``` ## Revoke a token List tokens and revoke one by its `jti`: ```sh curl "$API_URL/workers/tokens" -H "Authorization: Bearer $TOKEN" curl -X POST "$API_URL/workers/tokens/$JTI/revoke" \ -H "Authorization: Bearer $TOKEN" ``` Revocation also disconnects attached workers and blocks artifact access. Replacing the OAuth signing key revokes all worker tokens. Older tokens without signed labels and a `jti` need to be replaced. ## Upgrading workers Engine and worker must speak the same protocol. A worker on another protocol is refused and exits, so a release that bumps the protocol needs `pip install -U fluksio-worker` on every worker host. Upgrade the engine first, then the workers. The **Workers** screen shows each worker's package version and protocol. A worker refused for an older protocol stays listed as **Needs upgrade** until it attaches again, or until the engine restarts. ## Slurm provisioning The engine can submit a Slurm job that starts a worker. Put this configuration in `$DATA_DIR/provisioners.json`, replacing the example addresses and paths: ```json [{ "type": "slurm", "name": "hpc", "login": "me@login.cluster", "ssh_key": "/secrets/hpc_ed25519", "engine_url": "wss://fluksio.example.com/api/v1/workers/attach", "max_idle_s": 300, "provision_timeout_s": 900, "profiles": [{ "name": "gpu-small", "cpus": 8, "gpus": 1, "ram_mb": 65536, "labels": ["gpu"], "sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"], "prerun": ["module load cuda/12", "source ~/venvs/fluksio/bin/activate"] }] }] ``` The engine uses system SSH to submit `sbatch`. Prepare the environment named by `prerun` with `fluksio-worker` and your node dependencies; the generated job does not install packages. Compute nodes must be able to connect to the engine. Each job receives its own worker token, valid for 24 hours. Only one request per profile is outstanding at a time. Jobs that fail to attach within `provision_timeout_s` are cancelled, as are pending requests when the engine shuts down. `max_idle_s` releases idle workers. Either setting accepts `0` to disable its limit. Provisioning status appears under `GET /api/v1/workers/resources`. ## See also - [Runs](../concepts/runs.md#running-a-node-somewhere-else): remote execution. - [Writing node code](nodes.md): node functions and artifact helpers. - [Getting started: data science](../getting-started/data-science.md): resource declarations.