Say at startup when a flow wants a card, and record one seed rather than two
Docs / docs (push) Successful in 22s
Playwright Tests / test-playwright (1, 2) (push) Failing after 2m46s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m49s
pre-commit / pre-commit (push) Failing after 1m57s
Test Backend / test-backend (push) Failing after 2m28s
Compose Smoke Test / test-compose (push) Successful in 35s
Playwright Tests / merge-reports (push) Successful in 1m19s
Docs / docs (push) Successful in 22s
Playwright Tests / test-playwright (1, 2) (push) Failing after 2m46s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m49s
pre-commit / pre-commit (push) Failing after 1m57s
Test Backend / test-backend (push) Failing after 2m28s
Compose Smoke Test / test-compose (push) Successful in 35s
Playwright Tests / merge-reports (push) Successful in 1m19s
Three things the first pass left.
`serve` now names the flows asking for a GPU when the engine has none
declared. The placer already warned, but into the log, where a fresh install
that forgot `--gpus` does not read it — and the cost of missing it is GPU
nodes running concurrently, which is what the declaration exists to prevent.
The seed was the one field an export still had to coalesce: `--seed 1`
filled the run-level column and left `param.seed` blank, while a declared
seed filled the parameter and left the column blank. It is resolved like
every other input now, and the column carries the seed the run actually used
however it arrived — including when a parameter outranks the run's own,
where the column used to report the one that lost.
And the docs say plainly that declaring the card is what buys the worker
retirement: a node that imports jax without `resources={"gpus": 1}` never
gets CUDA_VISIBLE_DEVICES, so nothing marks its worker as one holding a card.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc
This commit is contained in:
@@ -82,6 +82,39 @@ def _mention_other_installation(data_dir: Path) -> None:
|
||||
_say(" `fluksio serve --global` runs that one instead.")
|
||||
|
||||
|
||||
def _mention_undeclared_cards() -> None:
|
||||
"""Say when a stored flow asks for a card this engine does not have.
|
||||
|
||||
Cards are declared rather than detected, so an engine told nothing has
|
||||
none — and a node asking for one is clamped to zero and runs beside every
|
||||
other, which on a GPU is the deadlock the declaration exists to prevent.
|
||||
The placer says so once it happens, into the log; this says it while
|
||||
somebody is still reading the terminal.
|
||||
"""
|
||||
from fluksio.core.config import settings
|
||||
|
||||
if settings.FLOW_GPUS:
|
||||
return
|
||||
from fluksio.flow.runs import required_resources
|
||||
from fluksio.flow.store import FlowStore
|
||||
|
||||
store = FlowStore(settings.FLOWS_DIR)
|
||||
asking = []
|
||||
for name in store.list_flows():
|
||||
try:
|
||||
needs = required_resources(store.read_flow(name))
|
||||
except Exception:
|
||||
# A flow that will not parse is the engine's to complain about.
|
||||
continue
|
||||
if (needs or {}).get("gpus"):
|
||||
asking.append(name)
|
||||
if not asking:
|
||||
return
|
||||
named = ", ".join(sorted(asking)[:3]) + (" …" if len(asking) > 3 else "")
|
||||
_say(f" Cards 0 declared, but {named} asks for one.")
|
||||
_say(" Nothing detects them: `--gpus N` says how many are here.")
|
||||
|
||||
|
||||
def _warn_if_networked(path: Path) -> None:
|
||||
"""A cluster's $HOME is often NFS, and SQLite's WAL does not work there."""
|
||||
try:
|
||||
@@ -454,6 +487,7 @@ def cmd_serve(args: argparse.Namespace) -> int:
|
||||
_say(" fluksio enroll <code>")
|
||||
_say(f" Signed in as {admin_email}")
|
||||
_say(f" token in {token_path}")
|
||||
_mention_undeclared_cards()
|
||||
_mention_other_installation(data_dir)
|
||||
|
||||
# One process: it holds the flow engine, and a second worker would be a
|
||||
|
||||
@@ -849,17 +849,23 @@ class RunService:
|
||||
# What the run actually starts from, not only what was passed: an input
|
||||
# left out takes its declared value, and a row that records `{}` cannot
|
||||
# say which. Folded literally — an initial is a value from the
|
||||
# definition, never a reference to resolve. The run-level seed still
|
||||
# wins over a declared one, the way `seed_values` has it.
|
||||
params = {
|
||||
**{
|
||||
declared.spec.name: declared.initial
|
||||
for declared in flow.inputs
|
||||
if declared.initial is not None
|
||||
and not (declared.spec.name == "seed" and seed is not None)
|
||||
},
|
||||
**params,
|
||||
# definition, never a reference to resolve.
|
||||
declared = {
|
||||
one.spec.name: one.initial for one in flow.inputs if one.initial is not None
|
||||
}
|
||||
# The run's own seed fills an input of that name, outranking what the
|
||||
# flow declares and outranked by one passed as a parameter — the order
|
||||
# `seed_values` applies, moved to where the record is written.
|
||||
if seed is not None and any(one.spec.name == "seed" for one in flow.inputs):
|
||||
declared["seed"] = seed
|
||||
params = {**declared, **params}
|
||||
# And back the other way, so the run-level column holds the seed the
|
||||
# run actually used however it arrived. Otherwise `--seed 1` fills one
|
||||
# column and a declared seed the other, and that is the single field an
|
||||
# export still has to coalesce.
|
||||
resolved_seed = params.get("seed")
|
||||
if isinstance(resolved_seed, int) and not isinstance(resolved_seed, bool):
|
||||
seed = resolved_seed
|
||||
# Checked here rather than in the driver: a caller who mistyped a
|
||||
# parameter should be told now, not by a run that fails in a minute.
|
||||
seed_values(flow, params, seed)
|
||||
|
||||
Reference in New Issue
Block a user