Let a node say how much of the machine it takes

Five concurrent training nodes, each sizing its thread pool to every core,
left the engine's own event loop unscheduled: the API stopped answering
within 10 s and every client died. The same shape on a GPU deadlocked a run
for 21 minutes at 0% utilisation with nothing failing and nothing to read --
it just sat in `running`.

@node(resources={"cpus": 2}) is the declaration. The engine holds that much
for the length of the execution, so more of them than the machine has room
for wait their turn rather than oversubscribing it, and a `gpus` node holds
its card exclusively. FLOW_CPUS defaults to every core but two, and those two
are what keeps the engine answering.

Because a thread cap is read when the process imports the library, a warm
worker cannot be told a different one -- so an environment gets a pool of its
own and nodes deriving the same one share it, rather than paying a cold start
per call on exactly the nodes whose imports are slowest. XLA_FLAGS is never
derived: it is a composed, version-dependent string, so it travels in
resources.env where it is visible.

A node that declares nothing is not accounted for and behaves as it always
did -- it just gets FLOW_CPUS/FLOW_MAX_WORKERS as a thread cap, which is the
half of this that fixes the reported incident without anybody declaring
anything. An operator who set OMP_NUM_THREADS themselves still wins.

Resources are claimed strictly before a worker slot, so the two blocking
waits cannot deadlock. A node queued for them publishes node_queued and shows
on GET /workers/resources, because waiting and hanging looked identical.

Accounted, not enforced: no cgroups, no rlimits. Scheduling across machines,
flavours and enforcement are the next steps.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/getting-started/data-science.md

@@ -262,6 +262,54 @@ stops your own code from running.
     )])
     ```
 
+### Declaring what a node needs
+
+A training node is not like the rest of a flow. NumPy, JAX and PyTorch each
+size their thread pool to every core they can see, so a few of them at once
+oversubscribe the machine badly enough to starve the engine itself — the API
+stops answering and every client waiting on a run dies with it. On a GPU the
+same shape deadlocks: two processes each preallocating most of the card sit at
+zero utilisation with nothing failing and nothing to read.
+
+Say what one execution takes, and the engine holds it:
+
+```python
+@node(..., resources={"cpus": 2, "env": {"XLA_FLAGS": "--xla_cpu_multi_thread_eigen=false"}})
+def fit(dataset, lr, epochs=25):
+    ...
+
+@node(..., resources={"gpus": 1, "env": {"XLA_PYTHON_CLIENT_MEM_FRACTION": "0.9"}})
+def finetune(checkpoint):
+    ...
+```
+
+Two things follow. The node waits its turn rather than starting alongside more
+of itself than the machine has room for — the same waiting it already does for
+a worker. And the worker it runs in is *started* with thread limits matching
+what it was given, because that is the only moment a numerical library reads
+them; a GPU node is told which card is its through `CUDA_VISIBLE_DEVICES`, and
+nothing else is given that card while it runs.
+
+`env` is for the tuning the engine must not invent. `XLA_FLAGS` is one composed
+string whose contents depend on the version you have installed, so writing it
+for you would silently replace whatever you had put there.
+
+Declaring nothing is the default and is right for most nodes — a poll, a
+threshold, a message on its way somewhere. Those share the engine's worker pool
+and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops
+several of them at once from each sizing to the whole box.
+
+Ask what is free, and what is waiting for it, at
+`GET /api/v1/workers/resources` — a node queued for cores looks exactly like a
+node that has hung unless you can see the queue.
+
+!!! note "Accounted, not enforced"
+
+    Nothing stops a node that ignores its declaration; the numbers are
+    bookkeeping plus the environment its libraries read. Scheduling across
+    several machines, named hardware flavours and real enforcement are the
+    next steps, not this one.
+
 ## Say which nodes make a flow
 
 Membership is a list, not a directory layout: the functions can live wherever

docs/reference/configuration.md

@@ -125,8 +125,18 @@ warning into a refusal to start.
 | `FLOW_MAX_WORKERS` | `4` | node-code subprocesses run in parallel |
 | `FLOW_MAX_CASCADES` | `4` | cascades in flight at once; throughput is this over the mean cascade time, so raise it where nodes wait on a network rather than a CPU |
 | `FLOW_NODE_TIMEOUT` | `0` | seconds a node may be silent, unless it sets its own; 0 is no limit |
+| `FLOW_CPUS` | `0` | cores nodes that declare `resources` may be given; 0 works it out as every core but two, which are what keeps the engine answering while the machine is busy |
+| `FLOW_GPUS` | `0` | GPUs on this machine, each held by one node at a time. Not detected — say how many there are |
 | `OBS_RETENTION_DAYS` | `30` | how long metrics, events and run records are kept |
 
+A node that declares nothing is not accounted against `FLOW_CPUS`; it runs on
+the shared pool and is given `FLOW_CPUS / FLOW_MAX_WORKERS` as a thread cap, so
+several at once cannot each size themselves to the whole machine. Setting
+`OMP_NUM_THREADS` (or any of its siblings) on the engine yourself overrides
+that default. What is free, and which nodes are queued for it, is
+`GET /api/v1/workers/resources`. See
+[declaring resources](../getting-started/data-science.md#declaring-what-a-node-needs).
+
 ## Agents
 
 | Variable | Default | Notes |