app/docs/code/workers.md

# Remote workers

The engine runs where the automations are. The GPU is somewhere else, the
Raspberry Pi with the relays is in a shed, and neither of them is on the same
network as the other.

A **worker** is a process that runs the code of nodes marked for it. It dials
*out* to the engine over one authenticated websocket, so nothing on that
machine has to be reachable — and nothing has to expose the engine's state
backend across hosts, which it never should.

## Install and attach

```sh
pip install fluksio-worker

fluksio-worker \
  --url wss://api.example.com/api/v1/workers/attach \
  --token "$FLUKSIO_WORKER_TOKEN" \
  --labels gpu,cuda12 \
  --python /opt/torch-venv/bin/python
```

`fluksio-worker` is its own distribution — the agent, the node runner, and
`websockets`. Nothing of the engine, so a GPU box does not install a database
driver in order to run a training step. An engine host already has it, and
`fluksio worker …` is the same program.

| Option | Default | What it does |
|---|---|---|
| `--url` | **required** | `wss://…/api/v1/workers/attach` |
| `--token` | `$FLUKSIO_WORKER_TOKEN` | the credential, minted on the engine |
| `--name` | this host's name | how it shows up in the worker list |
| `--labels` | none | comma-separated; what a node's `device` matches |
| `--python` | this interpreter | the interpreter node code runs on |
| `--parallel` | `1` | how many node calls it will take at once |
| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |

`--python` is the important one. It is how this machine keeps its own wheels —
the CUDA build, the vendor SDK, the thing that will not install anywhere else —
without the engine ever installing them or knowing about them.

## Mint the token

On the engine, as a superuser:

```sh
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"name": "gpu-dev"}'
```

Shown once, valid for a year — a worker is a machine somebody sets up and
leaves running. It is signed with the same keypair agent tokens use, so
rotating that key revokes every worker along with them.

??? note "A host where pip is not an option"

    The two files work copied into one directory and run with `python agent.py
    …`. The engine serves the runner itself at `GET /api/v1/workers/runtime` —
    it is the same module its own local workers run, deliberately standard
    library only.

## Send a node to it

A node declares the label of the machine it needs:

```json
{
  "id": "train",
  "device": "gpu",
  "device_policy": "require",
  "timeout": 7200
}
```

| `device_policy` | Behaviour when nothing carrying the label is attached |
|---|---|
| `require` (default) | the run stays `queued` and says what it is waiting for |
| `prefer` | it runs on the engine instead |

`prefer` is what makes a flow work before the GPU box exists. `require` is what
you want once it does.

!!! note "Set from the API"

    `device` and `device_policy` are not yet fields in the node panel. Set them
    with `PUT /flows/{name}`.

## What follows from this

- **The node's source travels with every call.** Nothing has to be deployed to
  the worker, and changing a node's code takes effect on the next execution.
- **A node bound to a device is compiled on that machine.** A node importing
  `torch` is correct on the GPU box and a missing module on the engine, so
  checking it here would fail something that is fine.
- **`import fluksio` inside a node is the worker's own reporter.** `emit`,
  `save_artifact`, `load_artifact` — installed before your code runs, so an
  installed `fluksio` package on that box never shadows it.
- **Cancelling a run kills what it is executing**, there or here, and leaves
  other runs of the same node alone.
- **If the worker disappears mid-call**, the run fails in seconds with
  `worker went away mid-call` rather than waiting out its timeout.
- **A worker sends a heartbeat every ten seconds while it executes**, so a long
  node is distinguishable from a dead socket. Ninety seconds of nothing at all —
  not even a heartbeat — fails the call as gone. A heartbeat says the *agent* is
  alive and nothing about the node, so it never satisfies a node's own timeout:
  one set to thirty seconds fires after thirty seconds of the node reporting
  nothing, wherever it runs.

## Artifacts across machines

An artifact reference names content by its hash, not a location, so it stays
valid wherever the store is reachable from. A worker that shares the engine's
filesystem writes to it directly; one that does not fetches and uploads over
HTTP, using the artifact endpoint beside the socket it already has. Either way
your node code is the same two calls.

## Seeing what is attached

```sh
curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq
```

Name, labels, how many calls it will take at once, how many are in flight, when
it attached, when it was last seen, its Python version, and a digest of its
environment.

## What a worker is not

It is not a second engine. Subscriptions, schedules, webhooks, the dashboards
and the run queue all stay in one process — that is what keeps a message having
one definition and a cron tick happening once. A worker executes node bodies.

Running two engines against one data directory is not supported. Distribute
work with workers.

## See also

- [Runs: pipelines that finish](../concepts/runs.md#running-a-node-somewhere-else)
- [Writing node code](nodes.md)
- [Getting started: data science](../getting-started/data-science.md)