Let a node say how much of the machine it takes

Five concurrent training nodes, each sizing its thread pool to every core,
left the engine's own event loop unscheduled: the API stopped answering
within 10 s and every client died. The same shape on a GPU deadlocked a run
for 21 minutes at 0% utilisation with nothing failing and nothing to read --
it just sat in `running`.

@node(resources={"cpus": 2}) is the declaration. The engine holds that much
for the length of the execution, so more of them than the machine has room
for wait their turn rather than oversubscribing it, and a `gpus` node holds
its card exclusively. FLOW_CPUS defaults to every core but two, and those two
are what keeps the engine answering.

Because a thread cap is read when the process imports the library, a warm
worker cannot be told a different one -- so an environment gets a pool of its
own and nodes deriving the same one share it, rather than paying a cold start
per call on exactly the nodes whose imports are slowest. XLA_FLAGS is never
derived: it is a composed, version-dependent string, so it travels in
resources.env where it is visible.

A node that declares nothing is not accounted for and behaves as it always
did -- it just gets FLOW_CPUS/FLOW_MAX_WORKERS as a thread cap, which is the
half of this that fixes the reported incident without anybody declaring
anything. An operator who set OMP_NUM_THREADS themselves still wins.

Resources are claimed strictly before a worker slot, so the two blocking
waits cannot deadlock. A node queued for them publishes node_queued and shows
on GET /workers/resources, because waiting and hanging looked identical.

Accounted, not enforced: no cgroups, no rlimits. Scheduling across machines,
flavours and enforcement are the next steps.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-26 21:36:59 +02:00
co-authored by Claude Opus 5
parent 4a38c6ed31
commit 608d30d884
13 changed files with 868 additions and 10 deletions
+56 -1
View File
@@ -9,7 +9,7 @@ from __future__ import annotations
import re
from typing import Any, Literal
from pydantic import BaseModel, Field, field_validator
from pydantic import BaseModel, ConfigDict, Field, field_validator
from fluksio.flow.messages import MessageSpec
@@ -24,6 +24,53 @@ def _validate_name(value: str) -> str:
return value
class Resources(BaseModel):
"""What one execution of a node needs to have to itself.
Declaring nothing is the default and means what it always did: the node
runs on the shared worker pool and nothing is accounted for it. That is
right for the kind of node most flows are made of — a poll, a threshold, a
message on its way somewhere.
It is wrong for the other kind. A numerical library sizes its thread pool
to every core it can see, so a handful of them at once oversubscribe the
machine badly enough to starve the engine's own event loop, and a GPU
library that preallocates most of the card deadlocks when a second one
arrives. Both are a node saying how much of the machine it takes, which is
what this is.
"""
model_config = ConfigDict(extra="forbid")
cpus: int = Field(
default=1,
ge=1,
description=(
"Cores held for the whole execution. Also what the thread-pool "
"variables are set to, so a library sizing itself to the machine "
"sizes itself to this instead."
),
)
gpus: int = Field(
default=0,
ge=0,
description=(
"Whole devices held for the whole execution, named to the node "
"through CUDA_VISIBLE_DEVICES. Nothing else is given them while it "
"runs, which is what keeps two preallocating processes apart."
),
)
env: dict[str, str] = Field(
default_factory=dict,
description=(
"Extra environment for the worker this node runs in, applied over "
"what the allocation derives. Where a library's own tuning goes — "
"XLA_FLAGS, XLA_PYTHON_CLIENT_MEM_FRACTION — since those are "
"composed strings the engine must not invent."
),
)
class NodeDef(BaseModel):
"""A node as stored: identity, configuration and ports.
@@ -78,6 +125,14 @@ class NodeDef(BaseModel):
"function whose answer can change on its own."
),
)
resources: Resources | None = Field(
default=None,
description=(
"What one execution of this node holds while it runs. Absent — the "
"default — means it is not accounted for and shares the engine's "
"workers, which is right for everything that is not compute-heavy."
),
)
@field_validator("id")
@classmethod