Let a worker say what machine it is

A worker reported its labels and nothing about the machine behind them, so the
engine could route a node to a GPU box but not tell whether that box had a GPU
free. Inventory — cores, GPUs, memory — now arrives with the hello frame, and
the run frame carries back what the engine allocated for that call.

Which is protocol 2 on both ends. GPUs are never probed: asking a vendor tool
would make the one dependency two, so a GPU is what the batch job says it was
given or what --gpus says. A worker that reports nothing still attaches and is
scheduled by its label alone.

Two things a job scheduler needs: --max-idle stops a worker started for one job
rather than letting it hold its allocation to the walltime, and a refusal is now
fatal instead of a reconnect loop that reads as a hang in a job's log.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
This commit is contained in:
2026-08-27 08:29:35 +02:00
co-authored by Claude Opus 5
parent 0ffcabfdb9
commit 1a9753fa9d
7 changed files with 359 additions and 36 deletions
+38 -2
View File
@@ -35,11 +35,37 @@ driver in order to run a training step. An engine host already has it, and
| `--python` | this interpreter | the interpreter node code runs on |
| `--parallel` | `1` | how many node calls it will take at once |
| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |
| `--cpus` | what the job or the machine has | cores to advertise |
| `--gpus` | what the job says, else none | GPUs to advertise; never probed |
| `--ram-mb` | what the job or the machine has | memory to advertise, in MB |
| `--max-idle` | never | stop after this many seconds with nothing running |
`--python` is the important one. It is how this machine keeps its own wheels —
the CUDA build, the vendor SDK, the thing that will not install anywhere else —
without the engine ever installing them or knowing about them.
### What it says it has
A worker reports its inventory when it attaches — cores, GPUs and memory — and
the engine schedules against it: a node asking for two cores and a GPU goes to
a machine that has them free, not merely to one carrying the right label.
Cores and memory are read off the machine, or off the batch job that started
this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **GPUs are never
probed.** Asking a vendor tool would make the one dependency two, so a GPU is
something the job says it was given (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`,
or `FLUKSIO_WORKER_GPUS`) or something you say with `--gpus`. A worker that
reports nothing still attaches and is scheduled by its label alone, as every
worker was before any of them reported anything.
The engine tells each call what it may use — thread caps, and the devices it
may see. The worker starts a process per call, so it applies them at the only
moment a numerical library still reads them: before the import.
`--max-idle` is for a worker something else started for one job — a batch
scheduler, say. It exits when nothing has run for that long, so the allocation
goes back rather than idling until its walltime.
## Mint the token
On the engine, as a superuser:
@@ -122,8 +148,18 @@ curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq
```
Name, labels, how many calls it will take at once, how many are in flight, when
it attached, when it was last seen, its Python version, and a digest of its
environment.
it attached, when it was last seen, its Python version, a digest of its
environment, and what it says it has: cores, GPUs and memory.
## Upgrading
The engine and the worker speak a version-matched protocol, and a worker
announcing anything else is refused rather than half-understood. Protocol 2 —
the one that carries inventory — is `fluksio-worker` 0.2.0. An older agent is
told so on the socket and stops, rather than retrying against an engine that
will never accept it; `pip install -U fluksio-worker` on that host is the whole
upgrade. Nothing changed in the runner served at `GET /api/v1/workers/runtime`,
so a host that copies its two files copies the same one as before.
## What a worker is not