Schedule a node across every machine, not just this one

The engine answered "where does this node run" twice, in two ways that could
not see each other: a device sent it to a worker carrying that label, and
resources were counted against the engine's own cores. Declaring both meant the
second answer won and nothing was counted at all — which the data-science
getting-started page and the worked example both do.

One question now, in flow/placement.py: of every machine attached, which could
grant what this node asked for, and which of those has it free. The books move
onto each machine — one accountant per worker, built from the inventory it
reported — and the waiting moves above them, where one condition variable can
be woken by a release anywhere or by a worker attaching. Locks go one way:
placer, then a machine's books, never back.

So a node asking for a card now finds the box that has one, rather than being
clamped down to none and run here. When nothing can grant the ask at all it is
still cut down and run — a flow written on a cluster has to work on a laptop —
but the ceiling is one real machine now, since taking the largest of each
dimension separately can describe a machine nobody has.

Two things fixed on the way. A device on a connector node held every batch run
of its flow forever, waiting for a worker that could never run an entry point.
And `prefer` falling back to the engine skipped the books, so the fallback held
nothing.

The bench flow's node has taken a `params` argument that with_settings has not
forwarded for some time, so the benchmark could not run at all: 62 ms median
submit-to-result with this, against the 61 ms on record.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
This commit is contained in:
2026-08-27 08:49:36 +02:00
co-authored by Claude Opus 5
parent 1a9753fa9d
commit 6ff56533f5
14 changed files with 1214 additions and 326 deletions
+10 -4
View File
@@ -29,6 +29,7 @@ from fluksio.flow.executor import ExecutionService
from fluksio.flow.metrics import MetricsCollector
from fluksio.flow.nodes.http import close_shared_client
from fluksio.flow.pipeline import ValueSource
from fluksio.flow.placement import Placer
from fluksio.flow.plugins import load_plugins
from fluksio.flow.queue import MemoryWorkQueue, RedisWorkQueue, WorkQueue
from fluksio.flow.remote import RemoteWorkerHub
@@ -142,10 +143,11 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
# not something anyone wrote, so it has no business in the git repository.
artifacts = ArtifactStore(settings.FLOWS_DIR.parent / "artifacts")
app.state.artifact_store = artifacts
accountant = ResourceAccountant(
cpus=settings.FLOW_CPUS, gpus=settings.FLOW_GPUS, events=event_bus
)
accountant = ResourceAccountant(cpus=settings.FLOW_CPUS, gpus=settings.FLOW_GPUS)
app.state.resources = accountant
# Every machine a node could run on: this one, and whatever attaches.
placer = Placer(local=accountant, events=event_bus)
app.state.placer = placer
pool = PythonWorkerPool(
python=modules.venv_python(),
size=settings.FLOW_MAX_WORKERS,
@@ -163,7 +165,10 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
)
pool.start()
app.state.worker_pool = pool
worker_hub = RemoteWorkerHub()
# Assigned rather than passed both ways: the hub tells the placer when a
# machine comes or goes, and the placer needs the hub to know what is there.
worker_hub = RemoteWorkerHub(on_change=placer.wake)
placer.hub = worker_hub
app.state.worker_hub = worker_hub
controller = FlowController(
store=store,
@@ -176,6 +181,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
workers=pool,
remote=worker_hub,
resources=accountant,
placer=placer,
)
app.state.flow_controller = controller
# A "dashboard" alert channel puts its alert into the graph. Bound here