Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Schedule a node across every machine, not just this one

Browse Source 
The engine answered "where does this node run" twice, in two ways that could
not see each other: a device sent it to a worker carrying that label, and
resources were counted against the engine's own cores. Declaring both meant the
second answer won and nothing was counted at all — which the data-science
getting-started page and the worked example both do.

One question now, in flow/placement.py: of every machine attached, which could
grant what this node asked for, and which of those has it free. The books move
onto each machine — one accountant per worker, built from the inventory it
reported — and the waiting moves above them, where one condition variable can
be woken by a release anywhere or by a worker attaching. Locks go one way:
placer, then a machine's books, never back.

So a node asking for a card now finds the box that has one, rather than being
clamped down to none and run here. When nothing can grant the ask at all it is
still cut down and run — a flow written on a cluster has to work on a laptop —
but the ceiling is one real machine now, since taking the largest of each
dimension separately can describe a machine nobody has.

Two things fixed on the way. A device on a connector node held every batch run
of its flow forever, waiting for a worker that could never run an entry point.
And `prefer` falling back to the engine skipped the books, so the fallback held
nothing.

The bench flow's node has taken a `params` argument that with_settings has not
forwarded for some time, so the benchmark could not run at all: 62 ms median
submit-to-result with this, against the 61 ms on record.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
This commit is contained in:
stroblme
2026-08-27 08:49:36 +02:00
co-authored by Claude Opus 5
parent 1a9753fa9d
commit 6ff56533f5
14 changed files with 1214 additions and 326 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/fluksio/flow/runs.py
+56 -8
View File
@@ -160,6 +160,17 @@ def _has_no_body(
return not store.has_node_source(flow, node.id, draft=draft)
def _runs_on_a_worker(node: NodeDef) -> bool:
"""Whether this node's body could go to a worker at all.
Only a node whose source travels can: a connector is an entry point loaded
in this process, so a device on one is a field nothing reads. Holding a run
for a worker that could never take that node is a run that never starts.
"""
node_type = NODE_TYPES.get(node.type)
return node_type is not None and node_type.has_source
def required_labels(flow: FlowDef) -> list[str]:
"""Worker labels this flow cannot run without.
@@ -171,11 +182,38 @@ def required_labels(flow: FlowDef) -> list[str]:
{
node.device
for node in flow.nodes
if node.device and node.device.strip() and node.device_policy == "require"
if node.device
and node.device.strip()
and node.device_policy == "require"
and _runs_on_a_worker(node)
}
)
def required_resources(flow: FlowDef) -> dict[str, Any] | None:
"""The largest single thing this flow needs a machine to have.
Dimension by dimension rather than per node, which is deliberately blunt:
it answers "could this run start at all", not "in what order". A node that
only *prefers* its device is left out — it runs here when nothing carries
the label, so it is never what a run is waiting for.
"""
cpus, gpus, ram = 0, 0, 0
device = ""
for node in flow.nodes:
wanted = node.resources
if wanted is None or not _runs_on_a_worker(node):
continue
cpus = max(cpus, wanted.cpus)
gpus = max(gpus, wanted.gpus)
ram = max(ram, wanted.ram or 0)
if node.device and node.device_policy == "require":
device = node.device
if not (cpus or gpus or ram):
return None
return {"cpus": cpus, "gpus": gpus, "ram_mb": ram, "device": device}
#: What a run's output is called from outside it: ``@run:<id>.<output>``.
RUN_REF_PREFIX = "@run:"
@@ -759,6 +797,7 @@ class RunService:
no_cache=no_cache,
status="queued",
labels=required_labels(flow),
needs=required_resources(flow),
created_at=datetime.now(UTC),
actor=actor,
idempotency_key=idempotency_key,
@@ -854,7 +893,7 @@ class RunService:
if not item.run_id:
self.queue.ack(item)
continue
missing = self._missing_labels(item.run_id)
missing = self._missing_requirements(item.run_id)
if missing:
# Left in the queue rather than failed: submitting a run
# before turning the GPU box on is a normal way to work, and
@@ -870,21 +909,30 @@ class RunService:
except RuntimeError:
logger.warning("Run %s not started: shutting down", item.run_id)
def _missing_labels(self, run_id: str) -> list[str]:
"""Worker labels this run needs that nothing attached carries."""
def _missing_requirements(self, run_id: str) -> list[str]:
"""What this run needs that nothing attached can give it yet."""
with Session(db_engine) as session:
run = session.get(Run, run_id)
needed = list(run.labels) if run else []
if not needed:
return []
needs = dict(run.needs) if run and run.needs else None
hub = self.controller.remote
available = hub.labels() | {w.name for w in hub.workers()} if hub else set()
# A node that only prefers its label runs locally instead, so it is not
# a reason to hold the run back; that is decided per node at call time.
return sorted(set(needed) - available)
missing = [f"labelled {label}" for label in sorted(set(needed) - available)]
placer = self.controller.placer
if placer is not None:
# Only a wait something is going to end — a machine that could take
# this is being started for it. An ask nothing can grant is cut down
# to what is here and runs, so holding the run would be holding it
# for something that is not coming.
short = placer.satisfiable(needs)
if short:
missing.append(f"with {short}")
return missing
def _waiting(self, run_id: str, missing: list[str]) -> None:
reason = f"Waiting for a worker labelled {', '.join(missing)}"
reason = f"Waiting for a worker {' and '.join(missing)}"
try:
with Session(db_engine) as session:
session.exec(