Health summary: a node is failing after two failures in a row; one is a warning, a good run ends both

The summary read degraded for 15 minutes after any single node_error, even
once the node had run well again and including one bad input from a visitor.
The controller now counts a node's failures since it last ran well and notes
when it recovered; both carry across rebuilds and an acknowledgement resets
the count. Two in a row inside the 15 minute window is "failing" (a problem),
one is listed under warnings, and nodes.failing keeps meaning "failing now".
This commit is contained in:
2026-10-28 12:42:35 +01:00
parent 2dc20911a3
commit 00842fa175
5 changed files with 161 additions and 14 deletions
+5
View File
@@ -70,6 +70,11 @@ After you have used batch runs, it also shows running and queued runs.
A degraded instance lists the problems that need attention. Warnings, such as
a device that is offline, are listed in amber and leave the status as it is.
A node that raises is a warning after its first failure and a problem when it
fails twice in a row with no good run between. A good run ends either, and so
does acknowledging the failure on the canvas. Without one, a failure counts for
15 minutes after the last time it happened.
### Execution time per node type
One bar per node type at its average time per run, slowest first, on a log