Let a worker say what machine it is
A worker reported its labels and nothing about the machine behind them, so the engine could route a node to a GPU box but not tell whether that box had a GPU free. Inventory — cores, GPUs, memory — now arrives with the hello frame, and the run frame carries back what the engine allocated for that call. Which is protocol 2 on both ends. GPUs are never probed: asking a vendor tool would make the one dependency two, so a GPU is what the batch job says it was given or what --gpus says. A worker that reports nothing still attaches and is scheduled by its label alone. Two things a job scheduler needs: --max-idle stops a worker started for one job rather than letting it hold its allocation to the walltime, and a refusal is now fatal instead of a reconnect loop that reads as a hang in a job's log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
This commit is contained in:
+38
-2
@@ -35,11 +35,37 @@ driver in order to run a training step. An engine host already has it, and
|
||||
| `--python` | this interpreter | the interpreter node code runs on |
|
||||
| `--parallel` | `1` | how many node calls it will take at once |
|
||||
| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |
|
||||
| `--cpus` | what the job or the machine has | cores to advertise |
|
||||
| `--gpus` | what the job says, else none | GPUs to advertise; never probed |
|
||||
| `--ram-mb` | what the job or the machine has | memory to advertise, in MB |
|
||||
| `--max-idle` | never | stop after this many seconds with nothing running |
|
||||
|
||||
`--python` is the important one. It is how this machine keeps its own wheels —
|
||||
the CUDA build, the vendor SDK, the thing that will not install anywhere else —
|
||||
without the engine ever installing them or knowing about them.
|
||||
|
||||
### What it says it has
|
||||
|
||||
A worker reports its inventory when it attaches — cores, GPUs and memory — and
|
||||
the engine schedules against it: a node asking for two cores and a GPU goes to
|
||||
a machine that has them free, not merely to one carrying the right label.
|
||||
|
||||
Cores and memory are read off the machine, or off the batch job that started
|
||||
this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **GPUs are never
|
||||
probed.** Asking a vendor tool would make the one dependency two, so a GPU is
|
||||
something the job says it was given (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`,
|
||||
or `FLUKSIO_WORKER_GPUS`) or something you say with `--gpus`. A worker that
|
||||
reports nothing still attaches and is scheduled by its label alone, as every
|
||||
worker was before any of them reported anything.
|
||||
|
||||
The engine tells each call what it may use — thread caps, and the devices it
|
||||
may see. The worker starts a process per call, so it applies them at the only
|
||||
moment a numerical library still reads them: before the import.
|
||||
|
||||
`--max-idle` is for a worker something else started for one job — a batch
|
||||
scheduler, say. It exits when nothing has run for that long, so the allocation
|
||||
goes back rather than idling until its walltime.
|
||||
|
||||
## Mint the token
|
||||
|
||||
On the engine, as a superuser:
|
||||
@@ -122,8 +148,18 @@ curl -s $FLUKSIO/workers -H "Authorization: Bearer $TOKEN" | jq
|
||||
```
|
||||
|
||||
Name, labels, how many calls it will take at once, how many are in flight, when
|
||||
it attached, when it was last seen, its Python version, and a digest of its
|
||||
environment.
|
||||
it attached, when it was last seen, its Python version, a digest of its
|
||||
environment, and what it says it has: cores, GPUs and memory.
|
||||
|
||||
## Upgrading
|
||||
|
||||
The engine and the worker speak a version-matched protocol, and a worker
|
||||
announcing anything else is refused rather than half-understood. Protocol 2 —
|
||||
the one that carries inventory — is `fluksio-worker` 0.2.0. An older agent is
|
||||
told so on the socket and stops, rather than retrying against an engine that
|
||||
will never accept it; `pip install -U fluksio-worker` on that host is the whole
|
||||
upgrade. Nothing changed in the runner served at `GET /api/v1/workers/runtime`,
|
||||
so a host that copies its two files copies the same one as before.
|
||||
|
||||
## What a worker is not
|
||||
|
||||
|
||||
Reference in New Issue
Block a user