Hold a node down for a minute before calling it down

A connector that misses one poll turned the health badge amber and the node
on the canvas red until it recovered. `LoadedNode.health_since` marks when the
status last actually changed — a rotating error detail under an unchanged
status does not move it, or a connector retrying with a different errno each
poll would never debounce — and `unhealthy_nodes` applies a 60 s floor to it.

The floor is on the surfacing, not on the transition: the stored health is the
truth the moment a node reports it, so a flow reading node health is never
told a stale story. Only going down is held back; recovery clears at once.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CL9zvnnvcp1mvA8o7impxk
This commit is contained in:
2026-09-07 07:54:10 +02:00
co-authored by Claude Opus 5
parent 39c231d6f7
commit 7b67eb5a8e
3 changed files with 105 additions and 4 deletions
+39 -2
View File
@@ -113,6 +113,14 @@ REBUILD_WAIT = 15.0
# the whole-pipeline rebuild and would let a wedge sit unreported.
FLOW_REBUILD_WAIT = 5.0
# How long a node has to have been down before it is shown as down. A device
# that misses one poll and comes back is a flake, not a fault, and without this
# it turned the health badge amber and the node on the canvas red for a couple
# of seconds. Only going *down* is held back: recovery clears at once, since
# calling a working thing broken is the worse of the two mistakes to leave on
# screen. The stored health is never delayed — see `unhealthy_nodes`.
HEALTH_FLOOR = 60.0
class RebuildBusy(RuntimeError):
"""A rebuild could not start because the one before it has not finished.
@@ -142,6 +150,12 @@ class LoadedNode:
error: str | None = None
health: Health = "ok"
health_detail: str | None = None
#: When `health` last became what it is, for the floor `unhealthy_nodes`
#: applies. A detail rotating under an unchanged status does not move it.
#: Zero reads as "for as long as anyone knows", so a node that starts out
#: down is shown straight away rather than waiting out a floor from a
#: transition that never happened.
health_since: float = 0.0
#: The last time this node raised while running, and what it said. Kept
#: after it has run again — a failure nobody saw is the one worth keeping —
#: so only an acknowledgement clears it, not a good run and not a rebuild.
@@ -1240,6 +1254,12 @@ class FlowController:
entry = self.loaded.get(node.id)
if entry is None or (entry.health == status and entry.health_detail == detail):
return
if entry.health != status:
# Only a real transition restarts the clock. A connector retrying
# with a different errno each poll rotates the detail while staying
# down, and moving the mark for that would keep the node forever
# under `HEALTH_FLOOR` — debouncing nothing at all.
entry.health_since = time.time()
entry.health = cast(Health, status)
entry.health_detail = detail
self._publish(
@@ -1252,6 +1272,24 @@ class FlowController:
}
)
def unhealthy_nodes(self, flow: str | None = None) -> list[LoadedNode]:
"""Nodes that have been down long enough to be worth telling anyone.
The floor is on the surfacing, not on the transition: `entry.health` is
the truth the moment the node reports it, so a flow reading a node's
health is never told a stale story — only the screens wait. And only
for the way down; a node reading "ok" leaves this list immediately,
whenever it recovered.
"""
now = time.time()
return [
entry
for entry in self.loaded.values()
if entry.health == "down"
and now - entry.health_since >= HEALTH_FLOOR
and (flow is None or entry.flow == flow)
]
def _health_issues(self, flow: str | None = None) -> list[ValidationIssue]:
"""Nodes that are running but not working, as issues on their flow.
@@ -1269,8 +1307,7 @@ class FlowController:
flow=entry.flow,
node=entry.id,
)
for entry in self.loaded.values()
if entry.health == "down" and (flow is None or entry.flow == flow)
for entry in self.unhealthy_nodes(flow)
]
def flow_issues(self, flow: str) -> list[ValidationIssue]: