Ask a cluster for a machine when nothing here will do

Slurm is not a machine that attaches and stays; it is a queue somebody else
owns. So nothing here submits a node to it. It submits a job whose payload is an
ordinary worker dialling back in, and everything downstream — the protocol, the
artifacts, cancellation, the books — already worked and did not have to learn
what Slurm is.

The alternative, which Covalent takes, is to stage a serialized call and a
runner onto the login node, poll squeue and copy the result back: a second way
of running a node beside the one that exists. The cost of not doing that is one
assumption, that a compute node can open a connection outward. Where that is
false, _payload is the single method a staged variant would replace.

Clusters are configured in provisioners.json beside the alerts, since this is
infrastructure an operator writes rather than anything a flow says. The script
is generated with the system ssh and no new dependency, and prerun owns the
environment — deliberately no pip install, because what is on a cluster is
somebody's decision.

One outstanding request per profile, cancelled if it never attaches and on the
way out. Nothing autoscales.

The run gate needed the same hook: a run held before it starts never reaches the
placer's own wait, so it would have queued forever on a machine nothing had
asked for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW

backend/fluksio/main.py

@@ -31,6 +31,7 @@ from fluksio.flow.nodes.http import close_shared_client
 from fluksio.flow.pipeline import ValueSource
 from fluksio.flow.placement import Placer
 from fluksio.flow.plugins import load_plugins
+from fluksio.flow.provision import load_provisioners
 from fluksio.flow.queue import MemoryWorkQueue, RedisWorkQueue, WorkQueue
 from fluksio.flow.remote import RemoteWorkerHub
 from fluksio.flow.resources import ResourceAccountant, fair_share_env
@@ -147,6 +148,8 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
     app.state.resources = accountant
     # Every machine a node could run on: this one, and whatever attaches.
     placer = Placer(local=accountant, events=event_bus)
+    # Where more machines can be asked for when nothing attached will do.
+    placer.provisioners = load_provisioners(settings.PROVISIONERS_FILE, event_bus)
     app.state.placer = placer
     pool = PythonWorkerPool(
         python=modules.venv_python(),
@@ -251,6 +254,10 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
         await run_in_threadpool(run_service.stop)
         await controller.stop()
         pool.stop()
+        # A machine asked for and not yet arrived would hold an allocation
+        # nobody is going to use.
+        for provisioner in placer.provisioners:
+            await run_in_threadpool(provisioner.shutdown)
         close_shared_client()
         if settings.MCP_ENABLED:
             from fluksio.mcp.http import aclose