Ask a cluster for a machine when nothing here will do

Slurm is not a machine that attaches and stays; it is a queue somebody else
owns. So nothing here submits a node to it. It submits a job whose payload is an
ordinary worker dialling back in, and everything downstream — the protocol, the
artifacts, cancellation, the books — already worked and did not have to learn
what Slurm is.

The alternative, which Covalent takes, is to stage a serialized call and a
runner onto the login node, poll squeue and copy the result back: a second way
of running a node beside the one that exists. The cost of not doing that is one
assumption, that a compute node can open a connection outward. Where that is
false, _payload is the single method a staged variant would replace.

Clusters are configured in provisioners.json beside the alerts, since this is
infrastructure an operator writes rather than anything a flow says. The script
is generated with the system ssh and no new dependency, and prerun owns the
environment — deliberately no pip install, because what is on a cluster is
somebody's decision.

One outstanding request per profile, cancelled if it never attaches and on the
way out. Nothing autoscales.

The run gate needed the same hook: a run held before it starts never reaches the
placer's own wait, so it would have queued forever on a machine nothing had
asked for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW

backend/fluksio/flow/placement.py

@@ -297,6 +297,22 @@ class Placer:
                 provisioner.provision(cpus, gpus, ram_mb, device)
                 return
 
+    def provision_for(self, needs: dict[str, Any] | None) -> None:
+        """Ask for a machine a queued run is waiting on.
+
+        A run held before it starts never reaches the wait above, so this is
+        where the asking happens for it — otherwise it would sit waiting for a
+        machine that nothing ever requested.
+        """
+        if not needs:
+            return
+        self._provision(
+            int(needs.get("cpus") or 1),
+            int(needs.get("gpus") or 0),
+            int(needs.get("ram_mb") or 0),
+            needs.get("device") or None,
+        )
+
     def _announce(self, node: str, run: str, since: float, reason: str) -> int:
         """Say a node is queued, not stuck.