Files

T

stroblme 53916d869b Clients take a flow's id or key: one resolver in the SDK, id first, explicit id:/key:, a warning for reads and a refusal for changes when a name means two flows, deleted flows found by their last key; sync creates by POST, retries a create race through the drift checks and reports the id; MCP gets create_flow and resolves every flow argument with the caller's token; displays show flow_name; an engine from before ids is named as such

2026-11-02 19:39:14 +01:00

8.7 KiB

Raw Blame History

Remote workers

A worker runs node code on another machine: a GPU host, a Raspberry Pi or a batch allocation. It connects outward to the engine over an authenticated WebSocket. Keep one engine; add workers where the hardware lives.

Create a worker token

On the engine, use a superuser access token. In these examples:

API_URL is the engine's API base, including /api/v1, such as https://fluksio.example.com/api/v1 or http://localhost:8080/api/v1 for a Docker container.
TOKEN is your superuser bearer token. The login endpoint POST /api/v1/login/access-token accepts form fields username and password.

Mint a token with the labels this machine should carry:

curl -X POST "$API_URL/workers/tokens" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"name": "gpu-dev", "labels": ["gpu", "cuda12"]}'

The response contains token, jti and the expiry. Save the token as FLUKSIO_WORKER_TOKEN on the worker host. It is returned once and is valid for 365 days. The signed name and labels control which nodes and secrets the engine sends to that worker.

Install and attach

Use Python 3.12 or later on the worker host:

pip install fluksio-worker
export FLUKSIO_WORKER_TOKEN='<token from the engine>'

fluksio-worker \
  --url wss://fluksio.example.com/api/v1/workers/attach \
  --labels gpu,cuda12 \
  --python /path/to/node-venv/bin/python

Replace the address and interpreter path with your own. --python selects the environment containing the node's dependencies, such as PyTorch or a device SDK. If you install the worker in that environment, you can omit the option. An engine installation also provides the same agent as fluksio worker.

Option	Default	Purpose
`--url`	required	engine's `/api/v1/workers/attach` WebSocket
`--token`	`$FLUKSIO_WORKER_TOKEN`	credential; prefer the environment variable to keep it out of process arguments
`--name`	hostname	suggested name; the engine uses the token's name
`--labels`	token's labels	comma-separated; when supplied, must match the token
`--python`	worker's interpreter	Python used for node code
`--parallel`	`1`	concurrent calls
`--artifact-url`	derived from `--url`	artifact endpoint, if hosted separately
`--service-host`	address the engine sees	reachable hostname or IP for services, when the connection address is unsuitable
`--cpus`	detected	advertised CPU count
`--gpus`	detected	advertised GPU count; explicit `0` means none
`--ram-mb`	detected	advertised memory in MB
`--max-idle`	`0`	exit after this many idle seconds; `0` disables
`--insecure`	off	allow unencrypted connections to a non-loopback host

For a trusted LAN stack, the URL is ws://<host-ip>:8080/api/v1/workers/attach and requires --insecure. Outside loopback, the worker otherwise requires encrypted URLs (wss:// and https://): node secrets and the worker token travel over the connection.

Node code receives a filtered host environment. Credential-like variables are removed; pass credentials through Secrets.

Send a node to it

Set these fields on a node through PUT /api/v1/flows/{id}:

{
  "id": "train",
  "worker": "gpu",
  "worker_policy": "require",
  "timeout": 7200
}

This is a node fragment; keep the node's other fields and the rest of the flow when updating it. worker and worker_policy are not in the node panel yet.

`worker_policy`	If no matching worker is attached
`require` (default)	queues until a matching worker is available
`prefer`	falls back to the engine

Source code travels with each call and compiles on the worker, so remote-only dependencies do not need to be installed on the engine. Calls to emit, save_artifact and load_artifact work there too.

Cancelling a run stops its active node processes. A disconnected worker fails in-flight calls. While executing, the agent sends heartbeats every ten seconds; 90 seconds without contact fails the call. Heartbeats do not reset the node's own silence timeout.

Every call runs in a process group of its own. The agent kills the group when the engine ends the call, when the connection drops and when the agent stops, so nothing a node started keeps running. If the agent itself is killed, its service manager has to clean up: systemd's default KillMode=control-group does.

Services

A Service node runs a server on a worker for as long as its flow runs, such as a speech model that should stay loaded. It is one long call that holds none of the worker's --parallel slots and no resources, so it does not block calls or hold a GPU against a training run. The engine checks the server's health over HTTP and restarts it whenever it stops, and its url names the address the engine sees the worker connect from. Set --service-host on the worker where that address is wrong.

A service's weights setting holds an artifact digest. The worker fetches it as node code fetches an artifact and passes the local path as {weights}. The hourly artifact sweep keeps any digest a node setting names in a published or draft flow, so the weights stay after the run that produced them is deleted.

Resources and artifacts

Workers report CPU, GPU and memory inventory, and scheduling checks both labels and available resources. CPU and memory detection uses Slurm allocations when present, otherwise host figures. GPU detection uses SLURM_GPUS_ON_NODE, FLUKSIO_WORKER_GPUS or SLURM_JOB_GPUS before counting NVIDIA device nodes. Explicit flags override detection.

Artifacts are identified by content hash. Remote workers fetch and upload over HTTP beside the WebSocket endpoint; node code uses the same artifact helpers as on the engine. Downloads are cached by digest:

Variable	Default
`FLUKSIO_ARTIFACT_CACHE`	`fluksio-artifacts` in the temporary directory
`FLUKSIO_ARTIFACT_CACHE_BYTES`	`1073741824` (1 GiB)

Oldest files are removed when the cache exceeds its limit. Raise it if workers repeatedly use large inputs.

The Workers screen shows attached hosts, available resources and queued nodes. The API exposes the same information:

curl "$API_URL/workers" -H "Authorization: Bearer $TOKEN"
curl "$API_URL/workers/resources" -H "Authorization: Bearer $TOKEN"

Revoke a token

List tokens and revoke one by its jti:

curl "$API_URL/workers/tokens" -H "Authorization: Bearer $TOKEN"
curl -X POST "$API_URL/workers/tokens/$JTI/revoke" \
  -H "Authorization: Bearer $TOKEN"

Revocation also disconnects attached workers and blocks artifact access. Replacing the OAuth signing key revokes all worker tokens. Older tokens without signed labels and a jti need to be replaced.

Upgrading workers

Engine and worker must speak the same protocol. A worker on another protocol is refused and exits, so a release that bumps the protocol needs pip install -U fluksio-worker on every worker host. Upgrade the engine first, then the workers.

The Workers screen shows each worker's package version and protocol. A worker refused for an older protocol stays listed as Needs upgrade until it attaches again, or until the engine restarts.

Slurm provisioning

The engine can submit a Slurm job that starts a worker. Put this configuration in $DATA_DIR/provisioners.json, replacing the example addresses and paths:

[{
  "type": "slurm",
  "name": "hpc",
  "login": "user@login.cluster.example.com",
  "ssh_key": "/secrets/hpc_ed25519",
  "engine_url": "wss://fluksio.example.com/api/v1/workers/attach",
  "max_idle_s": 300,
  "provision_timeout_s": 900,
  "profiles": [{
    "name": "gpu-small",
    "cpus": 8,
    "gpus": 1,
    "ram_mb": 65536,
    "labels": ["gpu"],
    "sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"],
    "prerun": ["module load cuda", "source /path/to/node-venv/bin/activate"]
  }]
}]

The engine uses system SSH to submit sbatch. Prepare the environment named by prerun with fluksio-worker and your node dependencies; the generated job does not install packages. Compute nodes must be able to connect to the engine. Each job receives its own worker token, valid for 24 hours.

Only one request per profile is outstanding at a time. Jobs that fail to attach within provision_timeout_s are cancelled, as are pending requests when the engine shuts down. max_idle_s releases idle workers. Either setting accepts 0 to disable its limit. Provisioning status appears under GET /api/v1/workers/resources.

8.7 KiB Raw Blame History