Schedule a node across every machine, not just this one
The engine answered "where does this node run" twice, in two ways that could not see each other: a device sent it to a worker carrying that label, and resources were counted against the engine's own cores. Declaring both meant the second answer won and nothing was counted at all — which the data-science getting-started page and the worked example both do. One question now, in flow/placement.py: of every machine attached, which could grant what this node asked for, and which of those has it free. The books move onto each machine — one accountant per worker, built from the inventory it reported — and the waiting moves above them, where one condition variable can be woken by a release anywhere or by a worker attaching. Locks go one way: placer, then a machine's books, never back. So a node asking for a card now finds the box that has one, rather than being clamped down to none and run here. When nothing can grant the ask at all it is still cut down and run — a flow written on a cluster has to work on a laptop — but the ceiling is one real machine now, since taking the largest of each dimension separately can describe a machine nobody has. Two things fixed on the way. A device on a connector node held every batch run of its flow forever, waiting for a worker that could never run an entry point. And `prefer` falling back to the engine skipped the books, so the fallback held nothing. The bench flow's node has taken a `params` argument that with_settings has not forwarded for some time, so the benchmark could not run at all: 62 ms median submit-to-result with this, against the 61 ms on record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
This commit is contained in:
@@ -90,18 +90,54 @@ def read_workers(request: Request) -> Any:
|
||||
]
|
||||
|
||||
|
||||
@router.get("/resources", dependencies=[Depends(get_current_user)])
|
||||
class ResourceLevel(BaseModel):
|
||||
total: int
|
||||
free: int
|
||||
|
||||
|
||||
class WaitingNode(BaseModel):
|
||||
node: str
|
||||
reason: str
|
||||
seconds: float
|
||||
|
||||
|
||||
class TargetResources(BaseModel):
|
||||
"""One machine: this engine, or a worker attached to it."""
|
||||
|
||||
target: str
|
||||
cpus: ResourceLevel
|
||||
gpus: ResourceLevel
|
||||
#: Absent where the machine did not say how much memory it has.
|
||||
ram_mb: ResourceLevel | None = None
|
||||
labels: list[str] = Field(default_factory=list)
|
||||
in_flight: int = 0
|
||||
|
||||
|
||||
class ResourcesSnapshot(BaseModel):
|
||||
#: This engine's own figures, kept where they have always been.
|
||||
cpus: ResourceLevel
|
||||
gpus: ResourceLevel
|
||||
waiting: list[WaitingNode] = Field(default_factory=list)
|
||||
targets: list[TargetResources] = Field(default_factory=list)
|
||||
provisioners: list[dict[str, Any]] = Field(default_factory=list)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/resources",
|
||||
response_model=ResourcesSnapshot,
|
||||
dependencies=[Depends(get_current_user)],
|
||||
)
|
||||
def read_resources(request: Request) -> Any:
|
||||
"""What this machine has free, and which nodes are queued for it.
|
||||
"""Every machine, what is free of it, and which nodes are queued.
|
||||
|
||||
A node waiting its turn looks exactly like a node that has hung — the run
|
||||
sits at `running` and says nothing — so what is waiting, and for what, has
|
||||
to be readable somewhere.
|
||||
"""
|
||||
accountant = getattr(request.app.state, "resources", None)
|
||||
if accountant is None:
|
||||
placer = getattr(request.app.state, "placer", None)
|
||||
if placer is None:
|
||||
raise HTTPException(status_code=503, detail="Resources are not accounted here")
|
||||
return accountant.snapshot()
|
||||
return placer.snapshot()
|
||||
|
||||
|
||||
@router.post(
|
||||
|
||||
Reference in New Issue
Block a user