Let a node say how much of the machine it takes

Five concurrent training nodes, each sizing its thread pool to every core,
left the engine's own event loop unscheduled: the API stopped answering
within 10 s and every client died. The same shape on a GPU deadlocked a run
for 21 minutes at 0% utilisation with nothing failing and nothing to read --
it just sat in `running`.

@node(resources={"cpus": 2}) is the declaration. The engine holds that much
for the length of the execution, so more of them than the machine has room
for wait their turn rather than oversubscribing it, and a `gpus` node holds
its card exclusively. FLOW_CPUS defaults to every core but two, and those two
are what keeps the engine answering.

Because a thread cap is read when the process imports the library, a warm
worker cannot be told a different one -- so an environment gets a pool of its
own and nodes deriving the same one share it, rather than paying a cold start
per call on exactly the nodes whose imports are slowest. XLA_FLAGS is never
derived: it is a composed, version-dependent string, so it travels in
resources.env where it is visible.

A node that declares nothing is not accounted for and behaves as it always
did -- it just gets FLOW_CPUS/FLOW_MAX_WORKERS as a thread cap, which is the
half of this that fixes the reported incident without anybody declaring
anything. An operator who set OMP_NUM_THREADS themselves still wins.

Resources are claimed strictly before a worker slot, so the two blocking
waits cannot deadlock. A node queued for them publishes node_queued and shows
on GET /workers/resources, because waiting and hanging looked identical.

Accounted, not enforced: no cgroups, no rlimits. Scheduling across machines,
flavours and enforcement are the next steps.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

backend/fluksio/main.py

@@ -31,6 +31,7 @@ from fluksio.flow.pipeline import ValueSource
 from fluksio.flow.plugins import load_plugins
 from fluksio.flow.queue import MemoryWorkQueue, RedisWorkQueue, WorkQueue
 from fluksio.flow.remote import RemoteWorkerHub
+from fluksio.flow.resources import ResourceAccountant, fair_share_env
 from fluksio.flow.runs import RUN_STATE_TTL, RunService
 from fluksio.flow.secrets import init_secrets
 from fluksio.flow.state import MemoryState, RedisState, StateBackend
@@ -116,13 +117,24 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
     # not something anyone wrote, so it has no business in the git repository.
     artifacts = ArtifactStore(settings.FLOWS_DIR.parent / "artifacts")
     app.state.artifact_store = artifacts
+    accountant = ResourceAccountant(
+        cpus=settings.FLOW_CPUS, gpus=settings.FLOW_GPUS, events=event_bus
+    )
+    app.state.resources = accountant
     pool = PythonWorkerPool(
         python=modules.venv_python(),
         size=settings.FLOW_MAX_WORKERS,
         events=event_bus,
         # A worker in this container writes to the store directly; a remote one
         # is given a URL instead. Node code calls the same two functions.
-        env={ARTIFACT_DIR_ENV: str(artifacts.root)},
+        env={
+            ARTIFACT_DIR_ENV: str(artifacts.root),
+            # Every slot can be busy at once, so a worker left to size its own
+            # thread pool to the machine means as many processes as there are
+            # slots, each believing it has the whole of it. A node that says
+            # what it needs overrides this; one that says nothing gets a share.
+            **fair_share_env(accountant.cpus, settings.FLOW_MAX_WORKERS),
+        },
     )
     pool.start()
     app.state.worker_pool = pool
@@ -138,6 +150,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
         alerts=alerts,
         workers=pool,
         remote=worker_hub,
+        resources=accountant,
     )
     app.state.flow_controller = controller
     # A "dashboard" alert channel puts its alert into the graph. Bound here