Let a node say how much of the machine it takes

Five concurrent training nodes, each sizing its thread pool to every core,
left the engine's own event loop unscheduled: the API stopped answering
within 10 s and every client died. The same shape on a GPU deadlocked a run
for 21 minutes at 0% utilisation with nothing failing and nothing to read --
it just sat in `running`.

@node(resources={"cpus": 2}) is the declaration. The engine holds that much
for the length of the execution, so more of them than the machine has room
for wait their turn rather than oversubscribing it, and a `gpus` node holds
its card exclusively. FLOW_CPUS defaults to every core but two, and those two
are what keeps the engine answering.

Because a thread cap is read when the process imports the library, a warm
worker cannot be told a different one -- so an environment gets a pool of its
own and nodes deriving the same one share it, rather than paying a cold start
per call on exactly the nodes whose imports are slowest. XLA_FLAGS is never
derived: it is a composed, version-dependent string, so it travels in
resources.env where it is visible.

A node that declares nothing is not accounted for and behaves as it always
did -- it just gets FLOW_CPUS/FLOW_MAX_WORKERS as a thread cap, which is the
half of this that fixes the reported incident without anybody declaring
anything. An operator who set OMP_NUM_THREADS themselves still wins.

Resources are claimed strictly before a worker slot, so the two blocking
waits cannot deadlock. A node queued for them publishes node_queued and shows
on GET /workers/resources, because waiting and hanging looked identical.

Accounted, not enforced: no cgroups, no rlimits. Scheduling across machines,
flavours and enforcement are the next steps.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

backend/fluksio/sdk/__init__.py

@@ -178,6 +178,7 @@ class NodeSpec:
         device: str | None,
         device_policy: str,
         cache: bool = True,
+        resources: dict[str, Any] | None = None,
     ) -> None:
         self.fn = fn
         self.id = id
@@ -190,6 +191,7 @@ class NodeSpec:
         self.device = device
         self.device_policy = device_policy
         self.cache = cache
+        self.resources = resources
 
     def rebind(
         self, *, id: str = "", wire: dict[str, str] | None = None, **settings: Any
@@ -221,6 +223,7 @@ class NodeSpec:
             device=self.device,
             device_policy=self.device_policy,
             cache=self.cache,
+            resources=self.resources,
         )
 
 
@@ -252,6 +255,7 @@ def node(
     device: str | None = None,
     device_policy: str = "require",
     cache: bool = True,
+    resources: dict[str, Any] | None = None,
 ) -> Callable[[F], F]:
     """Mark a function as a node, declaring its ports.
 
@@ -271,9 +275,29 @@ def node(
     ``timeout`` is seconds of silence — a yield or an emit resets it — after
     which the node is stopped. Left out it inherits the engine's default, which
     is no limit; set one where silence means stuck rather than working.
+
+    ``resources`` says how much of the machine one execution takes, for a node
+    heavy enough that the answer is not "a share of it"::
+
+        @node(..., resources={"cpus": 4})
+        @node(..., resources={"gpus": 1,
+                              "env": {"XLA_PYTHON_CLIENT_MEM_FRACTION": "0.9"}})
+
+    The engine holds that much while the node runs, so the rest of them wait
+    their turn rather than oversubscribing the box, and the worker is started
+    with thread limits matching what it was given — which is the only way a
+    library that sizes itself to every visible core is told otherwise. ``env``
+    is for the tuning the engine must not invent, such as ``XLA_FLAGS``.
+    Declaring nothing is the default and changes nothing.
     """
     if device_policy not in ("require", "prefer"):
         raise SyncError("device_policy is 'require' or 'prefer'")
+    if resources is not None:
+        unknown = sorted(set(resources) - {"cpus", "gpus", "env"})
+        if unknown:
+            raise SyncError(
+                f"resources={{'{unknown[0]}': ...}} is not one of cpus, gpus, env"
+            )
     if timeout is not None and timeout < 0:
         raise SyncError(
             f"node timeout must be 0 or more (0 disables it), got {timeout}"
@@ -294,6 +318,7 @@ def node(
             device=device,
             device_policy=device_policy,
             cache=cache,
+            resources=dict(resources) if resources is not None else None,
         )
         _check_signature(spec)
         fn.__fluksio__ = spec  # type: ignore[attr-defined]
@@ -499,6 +524,7 @@ def _check(spec: NodeSpec, mode: str) -> dict[str, Any]:
         "device": spec.device,
         "device_policy": spec.device_policy,
         "cache": spec.cache,
+        "resources": spec.resources,
     }