Ask a cluster for a machine when nothing here will do

Slurm is not a machine that attaches and stays; it is a queue somebody else
owns. So nothing here submits a node to it. It submits a job whose payload is an
ordinary worker dialling back in, and everything downstream — the protocol, the
artifacts, cancellation, the books — already worked and did not have to learn
what Slurm is.

The alternative, which Covalent takes, is to stage a serialized call and a
runner onto the login node, poll squeue and copy the result back: a second way
of running a node beside the one that exists. The cost of not doing that is one
assumption, that a compute node can open a connection outward. Where that is
false, _payload is the single method a staged variant would replace.

Clusters are configured in provisioners.json beside the alerts, since this is
infrastructure an operator writes rather than anything a flow says. The script
is generated with the system ssh and no new dependency, and prerun owns the
environment — deliberately no pip install, because what is on a cluster is
somebody's decision.

One outstanding request per profile, cancelled if it never attaches and on the
way out. Nothing autoscales.

The run gate needed the same hook: a run held before it starts never reaches the
placer's own wait, so it would have queued forever on a machine nothing had
asked for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW

docs/code/workers.md

@@ -161,6 +161,57 @@ will never accept it; `pip install -U fluksio-worker` on that host is the whole
 upgrade. Nothing changed in the runner served at `GET /api/v1/workers/runtime`,
 so a host that copies its two files copies the same one as before.
 
+## Machines from a batch scheduler
+
+A cluster is not a machine that attaches and stays — it is a queue somebody else
+owns. So Fluksio does not submit *nodes* to Slurm. It submits a job whose payload
+is an ordinary worker dialling back in, and from there everything works the way
+it already does: the same protocol, the same artifacts, the same cancellation.
+
+Write the clusters into `provisioners.json` beside the flows:
+
+```json
+[{
+  "type": "slurm",
+  "name": "hpc",
+  "login": "me@login.cluster",
+  "ssh_key": "/secrets/hpc_ed25519",
+  "engine_url": "wss://api.example.com/api/v1/workers/attach",
+  "max_idle_s": 300,
+  "provision_timeout_s": 900,
+  "profiles": [{
+    "name": "gpu-small",
+    "cpus": 8, "gpus": 1, "ram_mb": 65536,
+    "labels": ["gpu"],
+    "sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"],
+    "prerun": ["module load cuda/12", "source ~/venvs/flux/bin/activate"]
+  }]
+}]
+```
+
+A **profile** is what the scheduler is asked for, where a flavor is what a node
+asks for. They are separate on purpose, and agree when you set them up to.
+
+When a node needs a machine nothing attached can give, and a profile would fit,
+the engine `sbatch`es one over ssh — the system `ssh`, so nothing new is
+installed — and the run waits meanwhile, saying so. `prerun` owns the
+environment: a `module load`, a venv with `fluksio-worker` already in it. There
+is deliberately no `pip install` in the generated script, because what is
+installed on a cluster is somebody's decision and not this program's.
+
+One outstanding request per profile, however often it is asked for. A job that
+never attaches within `provision_timeout_s` is `scancel`led, as is anything
+outstanding when the engine stops. `--max-idle` is what ends the job at the
+other end, so an allocation goes back rather than idling to its walltime.
+
+`GET /workers/resources` reports what is outstanding and what last went wrong.
+
+!!! note "It needs a route out"
+
+    A compute node must be able to open a connection to the engine. That is
+    true of most clusters and false of air-gapped ones; there is no staging
+    path today.
+
 ## What a worker is not
 
 It is not a second engine. Subscriptions, schedules, webhooks, the dashboards