Let a node say how much of the machine it takes

Five concurrent training nodes, each sizing its thread pool to every core,
left the engine's own event loop unscheduled: the API stopped answering
within 10 s and every client died. The same shape on a GPU deadlocked a run
for 21 minutes at 0% utilisation with nothing failing and nothing to read --
it just sat in `running`.

@node(resources={"cpus": 2}) is the declaration. The engine holds that much
for the length of the execution, so more of them than the machine has room
for wait their turn rather than oversubscribing it, and a `gpus` node holds
its card exclusively. FLOW_CPUS defaults to every core but two, and those two
are what keeps the engine answering.

Because a thread cap is read when the process imports the library, a warm
worker cannot be told a different one -- so an environment gets a pool of its
own and nodes deriving the same one share it, rather than paying a cold start
per call on exactly the nodes whose imports are slowest. XLA_FLAGS is never
derived: it is a composed, version-dependent string, so it travels in
resources.env where it is visible.

A node that declares nothing is not accounted for and behaves as it always
did -- it just gets FLOW_CPUS/FLOW_MAX_WORKERS as a thread cap, which is the
half of this that fixes the reported incident without anybody declaring
anything. An operator who set OMP_NUM_THREADS themselves still wins.

Resources are claimed strictly before a worker slot, so the two blocking
waits cannot deadlock. A node queued for them publishes node_queued and shows
on GET /workers/resources, because waiting and hanging looked identical.

Accounted, not enforced: no cgroups, no rlimits. Scheduling across machines,
flavours and enforcement are the next steps.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

backend/tests/flow/test_workers.py

@@ -468,3 +468,83 @@ def test_an_emission_on_an_undeclared_port_fails_the_call(pool):
 
     # The worker was retired rather than left mid-generator, so the slot works.
     assert run(pool, "def process():\n    return {'out': 4}\n") == {"out": 4}
+
+
+# -----------------------------------------------------------------------------
+# Pools per environment
+#
+# A thread cap is read when the process imports the library, so a warm worker
+# cannot be told a different one. A node that declared resources therefore gets
+# a pool started with its own environment, and nodes deriving the same one
+# share it rather than paying for a cold start each.
+# -----------------------------------------------------------------------------
+
+
+def test_a_declared_environment_gets_a_pool_of_its_own(pool):
+    child = pool.for_env({"OMP_NUM_THREADS": "2"})
+
+    assert child is not pool
+    assert child.env["OMP_NUM_THREADS"] == "2"
+    # The same environment is the same pool: warm workers are the point.
+    assert pool.for_env({"OMP_NUM_THREADS": "2"}) is child
+    assert pool.for_env({"OMP_NUM_THREADS": "3"}) is not child
+    # Nothing to derive means the shared pool, with no second process anywhere.
+    assert pool.for_env({}) is pool
+
+
+def test_a_child_worker_is_started_with_what_it_was_given(pool):
+    child = pool.for_env({"OMP_NUM_THREADS": "2"})
+
+    seen = child.run(
+        "demo",
+        "env",
+        "import os\n\n\ndef process():\n"
+        "    return {'threads': os.environ.get('OMP_NUM_THREADS', '')}\n",
+        {},
+        "demo.env",
+        timeout=10,
+    )
+    assert seen == {"threads": "2"}
+
+
+def test_retiring_workers_reaches_the_children(pool):
+    child = pool.for_env({"OMP_NUM_THREADS": "2"})
+    before = child._generation
+
+    pool.respawn_all()
+
+    assert child._generation > before
+
+
+def test_cancelling_reaches_a_node_running_in_a_child(pool):
+    child = pool.for_env({"OMP_NUM_THREADS": "2"})
+    started = threading.Event()
+    failed: list[Exception] = []
+
+    def call() -> None:
+        try:
+            child.run(
+                "demo",
+                "slow",
+                "import time\n\n\ndef process():\n"
+                "    print('up', flush=True)\n"
+                "    time.sleep(30)\n"
+                "    return {'out': 1}\n",
+                {},
+                "demo.slow",
+                timeout=0,
+                on_event=lambda _event: started.set(),
+            )
+        except Exception as exc:  # noqa: BLE001 — the point is that it stopped
+            failed.append(exc)
+
+    thread = threading.Thread(target=call)
+    thread.start()
+    # The node has no events, so wait for the process rather than for a frame.
+    deadline = time.monotonic() + 10
+    while time.monotonic() < deadline and not child._running:
+        time.sleep(0.01)
+
+    assert pool.cancel("demo.slow"), "the parent must find a child's node"
+    thread.join(timeout=10)
+    assert failed