Let a node say how much of the machine it takes

Five concurrent training nodes, each sizing its thread pool to every core,
left the engine's own event loop unscheduled: the API stopped answering
within 10 s and every client died. The same shape on a GPU deadlocked a run
for 21 minutes at 0% utilisation with nothing failing and nothing to read --
it just sat in `running`.

@node(resources={"cpus": 2}) is the declaration. The engine holds that much
for the length of the execution, so more of them than the machine has room
for wait their turn rather than oversubscribing it, and a `gpus` node holds
its card exclusively. FLOW_CPUS defaults to every core but two, and those two
are what keeps the engine answering.

Because a thread cap is read when the process imports the library, a warm
worker cannot be told a different one -- so an environment gets a pool of its
own and nodes deriving the same one share it, rather than paying a cold start
per call on exactly the nodes whose imports are slowest. XLA_FLAGS is never
derived: it is a composed, version-dependent string, so it travels in
resources.env where it is visible.

A node that declares nothing is not accounted for and behaves as it always
did -- it just gets FLOW_CPUS/FLOW_MAX_WORKERS as a thread cap, which is the
half of this that fixes the reported incident without anybody declaring
anything. An operator who set OMP_NUM_THREADS themselves still wins.

Resources are claimed strictly before a worker slot, so the two blocking
waits cannot deadlock. A node queued for them publishes node_queued and shows
on GET /workers/resources, because waiting and hanging looked identical.

Accounted, not enforced: no cgroups, no rlimits. Scheduling across machines,
flavours and enforcement are the next steps.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

backend/fluksio/flow/controller.py

@@ -62,6 +62,7 @@ from fluksio.flow.pipeline import (
     ValueSource,
 )
 from fluksio.flow.remote import RemoteWorkerHub
+from fluksio.flow.resources import ResourceAccountant, derive_env
 from fluksio.flow.schemas import (
     BrainEdge,
     BrainGraph,
@@ -71,6 +72,7 @@ from fluksio.flow.schemas import (
     NodeDef,
     NodeStatusPublic,
     NodeTypeInfo,
+    Resources,
 )
 from fluksio.flow.secrets import SecretNotFound, resolve_params
 from fluksio.flow.state import MemoryState, StateBackend
@@ -341,6 +343,7 @@ class FlowController:
         alerts: AlertManager | None = None,
         workers: PythonWorkerPool | None = None,
         remote: RemoteWorkerHub | None = None,
+        resources: ResourceAccountant | None = None,
     ) -> None:
         self.store = store
         # Without a pool, python nodes are compiled and run in this process —
@@ -348,6 +351,9 @@ class FlowController:
         self.workers = workers
         # Workers on other hosts. A node without a device never touches it.
         self.remote = remote
+        # What the machine has, for the nodes that say what they need. Without
+        # one, a declaration is recorded and nothing is held against it.
+        self.resources = resources
         self.state = state if state is not None else MemoryState()
         self.events = events
         self.max_workers = max_workers
@@ -790,6 +796,56 @@ class FlowController:
     # Building
     # -------------------------------------------------------------------------
 
+    def _allocated(
+        self,
+        wanted: Resources,
+        owner: str,
+        local: str,
+        code: str,
+        *,
+        node_id: str,
+        timeout: float,
+        run_id: str,
+        on_event: Callable[[dict[str, Any]], None],
+    ) -> Callable[..., Any]:
+        """A call that holds its share of the machine while it runs.
+
+        The order is load-bearing: the resources are claimed first, and only
+        then is a worker slot taken. The other way round, a node holding a slot
+        could sit waiting for cores that a node holding the cores cannot get a
+        slot to release.
+
+        The worker comes from the pool whose environment this allocation
+        derives, so what the node is told about its share is what the library
+        inside it reads at import — the only moment those variables are read.
+        """
+        if self.workers is None or self.resources is None:
+            return self.workers.proxy(  # type: ignore[union-attr]
+                owner,
+                local,
+                code,
+                node_id=node_id,
+                timeout=timeout,
+                run_id=run_id,
+                on_event=on_event,
+            )
+        accountant, pool = self.resources, self.workers
+
+        def call(**kwargs: Any) -> Any:
+            with accountant.claim(wanted, node=node_id, run=run_id) as allocation:
+                return pool.for_env(derive_env(wanted, allocation)).run(
+                    owner,
+                    local,
+                    code,
+                    kwargs,
+                    node_id,
+                    timeout,
+                    run_id=run_id,
+                    on_event=on_event,
+                )
+
+        return call
+
     def _build_flows(
         self,
         flows: list[tuple[FlowDef, bool]],
@@ -894,6 +950,17 @@ class FlowController:
                         run_id=run.run_id if run else "",
                         on_event=emissions.handle,
                     )
+                    if node_def.resources is not None and not node_def.device:
+                        function = self._allocated(
+                            node_def.resources,
+                            owner,
+                            local,
+                            code,
+                            node_id=node_id,
+                            timeout=timeout,
+                            run_id=run.run_id if run else "",
+                            on_event=emissions.handle,
+                        )
                     if node_def.device and self.remote is not None:
                         # A node with a device runs on a worker carrying that
                         # label. Which worker is decided per call, so one that