Schedule a node across every machine, not just this one
The engine answered "where does this node run" twice, in two ways that could not see each other: a device sent it to a worker carrying that label, and resources were counted against the engine's own cores. Declaring both meant the second answer won and nothing was counted at all — which the data-science getting-started page and the worked example both do. One question now, in flow/placement.py: of every machine attached, which could grant what this node asked for, and which of those has it free. The books move onto each machine — one accountant per worker, built from the inventory it reported — and the waiting moves above them, where one condition variable can be woken by a release anywhere or by a worker attaching. Locks go one way: placer, then a machine's books, never back. So a node asking for a card now finds the box that has one, rather than being clamped down to none and run here. When nothing can grant the ask at all it is still cut down and run — a flow written on a cluster has to work on a laptop — but the ceiling is one real machine now, since taking the largest of each dimension separately can describe a machine nobody has. Two things fixed on the way. A device on a connector node held every batch run of its flow forever, waiting for a worker that could never run an entry point. And `prefer` falling back to the engine skipped the books, so the fallback held nothing. The bench flow's node has taken a `params` argument that with_settings has not forwarded for some time, so the benchmark could not run at all: 62 ms median submit-to-result with this, against the 61 ms on record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
This commit is contained in:
Claude Opus 5
@@ -8,16 +8,18 @@ seconds. On a GPU the same shape is worse: three processes each preallocating
|
||||
most of the card deadlock at zero utilisation, with nothing failing and nothing
|
||||
to read.
|
||||
|
||||
:class:`ResourceAccountant` is the first half: a node that declares what it
|
||||
needs waits until that much is free, the way it already waits for a worker
|
||||
slot. :func:`derive_env` is the second: what it got is handed to the library as
|
||||
the environment it reads at import, because that is the only moment those knobs
|
||||
are read.
|
||||
:class:`ResourceAccountant` is the first half: the books for one machine, which
|
||||
hand out what is free and take it back. :func:`derive_env` is the second: what
|
||||
a node got is handed to the library as the environment it reads at import,
|
||||
because that is the only moment those knobs are read.
|
||||
|
||||
One accountant is one machine. Which machine a node goes to, and the waiting
|
||||
when none of them has room, is :mod:`fluksio.flow.placement` — the books here
|
||||
never block, so a caller holding several of them can ask each in turn.
|
||||
|
||||
Deliberately cooperative — nothing here is enforced with cgroups or rlimits, so
|
||||
a node that ignores its share is only accounted for, not stopped. That is the
|
||||
same trust the worker pool already extends to node code, and the enforcement
|
||||
half belongs with the scheduler work this is the first step of.
|
||||
same trust the worker pool already extends to node code.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -25,12 +27,9 @@ from __future__ import annotations
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
|
||||
from fluksio.flow.events import EventBus
|
||||
from fluksio.flow.schemas import Resources
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -63,146 +62,92 @@ class Allocation:
|
||||
|
||||
cpus: int = 1
|
||||
gpus: tuple[int, ...] = ()
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Waiting:
|
||||
node: str
|
||||
since: float
|
||||
reason: str
|
||||
ram_mb: int = 0
|
||||
|
||||
|
||||
class ResourceAccountant:
|
||||
"""What is free on this machine, and who is waiting for it.
|
||||
"""The books for one machine: what it holds, and what is free of it.
|
||||
|
||||
A plain condition variable over two counters. Blocking is the whole
|
||||
mechanism: the caller is a node thread that would otherwise be executing,
|
||||
and making it wait is the same backpressure that the worker pool's slot
|
||||
queue already applies — one reason the two must always be taken in the same
|
||||
order, resources first, so neither can be held while waiting for the other.
|
||||
Nothing here blocks. A node that has to wait waits in the placer, which
|
||||
holds one of these per machine and tries each in turn — a lock per machine
|
||||
could not answer "is there room anywhere", and a node waiting inside one
|
||||
machine's books could not be woken by another machine attaching.
|
||||
|
||||
``on_release`` is how it tells the placer something came free. It is called
|
||||
outside this object's lock, deliberately: the placer takes its own lock
|
||||
there, and the two are only ever taken in that one order.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, cpus: int = 0, gpus: int = 0, events: EventBus | None = None
|
||||
self,
|
||||
cpus: int = 0,
|
||||
gpus: int = 0,
|
||||
ram_mb: int | None = None,
|
||||
on_release: Callable[[], None] | None = None,
|
||||
) -> None:
|
||||
self.cpus = cpus or machine_cpus()
|
||||
self.gpus = max(0, gpus)
|
||||
self.events = events
|
||||
#: None is memory this machine did not report, which is allowed: it is
|
||||
#: then not accounted rather than accounted as zero, since zero would
|
||||
#: refuse every node that asks for any.
|
||||
self.ram_mb = ram_mb
|
||||
self.on_release = on_release
|
||||
self._free_cpus = self.cpus
|
||||
self._free_gpus = list(range(self.gpus))
|
||||
self._waiting: dict[int, _Waiting] = {}
|
||||
self._ticket = 0
|
||||
self._condition = threading.Condition()
|
||||
self._free_ram = ram_mb or 0
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# -- what a caller asks for ------------------------------------------------
|
||||
def fits(self, cpus: int, gpus: int, ram_mb: int = 0) -> bool:
|
||||
"""Whether this machine could ever grant that much, busy or not.
|
||||
|
||||
def _clamp(self, wanted: Resources, node: str) -> tuple[int, int]:
|
||||
"""What can be granted here, which may be less than what was asked.
|
||||
|
||||
A flow written against a sixty-four core box should still run on a
|
||||
laptop. Waiting forever for cores that do not exist is not a smaller
|
||||
machine, it is a hung run.
|
||||
The question a placer asks before it decides to wait: waiting for a
|
||||
card that is merely busy is queueing, and waiting for one the machine
|
||||
does not have is a hung run.
|
||||
"""
|
||||
cpus = min(wanted.cpus, self.cpus)
|
||||
gpus = min(wanted.gpus, self.gpus)
|
||||
if cpus != wanted.cpus or gpus != wanted.gpus:
|
||||
logger.warning(
|
||||
"%s asked for %d cpu(s) and %d gpu(s); this engine has %d and %d",
|
||||
node or "a node",
|
||||
wanted.cpus,
|
||||
wanted.gpus,
|
||||
self.cpus,
|
||||
self.gpus,
|
||||
)
|
||||
return cpus, gpus
|
||||
if cpus > self.cpus or gpus > self.gpus:
|
||||
return False
|
||||
return self.ram_mb is None or ram_mb <= self.ram_mb
|
||||
|
||||
@contextmanager
|
||||
def claim(
|
||||
self, wanted: Resources, node: str = "", run: str = ""
|
||||
) -> Iterator[Allocation]:
|
||||
"""Hold this node's share for as long as it runs."""
|
||||
cpus, gpus = self._clamp(wanted, node)
|
||||
allocation = self._take(cpus, gpus, node, run)
|
||||
try:
|
||||
yield allocation
|
||||
finally:
|
||||
self._give_back(allocation)
|
||||
|
||||
def _take(self, cpus: int, gpus: int, node: str, run: str) -> Allocation:
|
||||
waited_from = time.monotonic()
|
||||
ticket = 0
|
||||
with self._condition:
|
||||
while self._free_cpus < cpus or len(self._free_gpus) < gpus:
|
||||
if not ticket:
|
||||
reason = (
|
||||
f"waiting for {cpus} cpu(s) ({self._free_cpus} free)"
|
||||
if self._free_cpus < cpus
|
||||
else f"waiting for {gpus} gpu(s) ({len(self._free_gpus)} free)"
|
||||
)
|
||||
ticket = self._announce(node, run, waited_from, reason)
|
||||
self._condition.wait()
|
||||
def try_take(self, cpus: int, gpus: int, ram_mb: int = 0) -> Allocation | None:
|
||||
"""Take this much if it is free right now, or answer that it is not."""
|
||||
with self._lock:
|
||||
if self._free_cpus < cpus or len(self._free_gpus) < gpus:
|
||||
return None
|
||||
if self.ram_mb is not None and self._free_ram < ram_mb:
|
||||
return None
|
||||
self._free_cpus -= cpus
|
||||
taken = tuple(self._free_gpus.pop(0) for _ in range(gpus))
|
||||
if ticket:
|
||||
self._waiting.pop(ticket, None)
|
||||
if ticket:
|
||||
logger.info(
|
||||
"%s waited %.1fs for %d cpu(s) and %d gpu(s)",
|
||||
node or "a node",
|
||||
time.monotonic() - waited_from,
|
||||
cpus,
|
||||
gpus,
|
||||
)
|
||||
return Allocation(cpus=cpus, gpus=taken)
|
||||
if self.ram_mb is not None:
|
||||
self._free_ram -= ram_mb
|
||||
return Allocation(cpus=cpus, gpus=taken, ram_mb=ram_mb)
|
||||
|
||||
def _give_back(self, allocation: Allocation) -> None:
|
||||
with self._condition:
|
||||
def give_back(self, allocation: Allocation) -> None:
|
||||
"""Return what an execution held, however it ended."""
|
||||
with self._lock:
|
||||
self._free_cpus += allocation.cpus
|
||||
self._free_gpus.extend(allocation.gpus)
|
||||
self._free_gpus.sort()
|
||||
self._condition.notify_all()
|
||||
|
||||
# -- what it looks like from outside ---------------------------------------
|
||||
|
||||
def _announce(self, node: str, run: str, since: float, reason: str) -> int:
|
||||
"""Say a node is queued, not stuck.
|
||||
|
||||
The failure this exists for looked identical to a hang: a run sitting
|
||||
at `running` for twenty minutes with no error and no output. A node
|
||||
waiting its turn has to say so somewhere a person will look.
|
||||
"""
|
||||
self._ticket += 1
|
||||
ticket = self._ticket
|
||||
self._waiting[ticket] = _Waiting(node=node, since=since, reason=reason)
|
||||
logger.info("%s is %s", node or "a node", reason)
|
||||
if self.events is not None:
|
||||
self.events.publish(
|
||||
{
|
||||
"type": "node_queued",
|
||||
"flow": node.split(".", 1)[0] if node else "",
|
||||
"node": node,
|
||||
"run": run,
|
||||
"detail": reason,
|
||||
"ts": time.time(),
|
||||
}
|
||||
)
|
||||
return ticket
|
||||
if self.ram_mb is not None:
|
||||
self._free_ram += allocation.ram_mb
|
||||
if self.on_release is not None:
|
||||
self.on_release()
|
||||
|
||||
def snapshot(self) -> dict[str, object]:
|
||||
"""What is free and who is waiting, for the workers screen."""
|
||||
with self._condition:
|
||||
free_cpus, free_gpus = self._free_cpus, len(self._free_gpus)
|
||||
"""What this machine holds and what is free of it."""
|
||||
with self._lock:
|
||||
free_cpus, free_gpus, free_ram = (
|
||||
self._free_cpus,
|
||||
len(self._free_gpus),
|
||||
self._free_ram,
|
||||
)
|
||||
return {
|
||||
"cpus": {"total": self.cpus, "free": free_cpus},
|
||||
"gpus": {"total": self.gpus, "free": free_gpus},
|
||||
"waiting": [
|
||||
{
|
||||
"node": entry.node,
|
||||
"reason": entry.reason,
|
||||
"seconds": round(time.monotonic() - entry.since, 1),
|
||||
}
|
||||
for entry in list(self._waiting.values())
|
||||
],
|
||||
"ram_mb": (
|
||||
None
|
||||
if self.ram_mb is None
|
||||
else {"total": self.ram_mb, "free": free_ram}
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user