Keep the engine's own history, and a screen that reads it

A second bus subscriber folds executions, errors, timings and queue lag
into per-minute rollups, keeps failures with their traceback and an audit
trail of who published what, and records one row per cascade — manual runs
and previews included, under an id of their own that writes no idempotency
markers. Read back through /observability/*, which always answers 200 so a
degraded engine still renders its own health screen.

Also fixes two things found on the way: node-health alerts read `status`
where the engine publishes `health`, so a device dropping never alerted
anyone, and the Redis queue reported `parked: 0` whatever was held.
This commit is contained in:
2026-08-16 22:29:32 +02:00
parent e05e17943a
commit 07de4c4d9a
30 changed files with 2610 additions and 22 deletions
+2 -2
View File
@@ -102,7 +102,7 @@ def describe(event: dict[str, Any]) -> Alert | None:
node=node,
)
if kind == "node_health":
if event.get("status") != "down":
if event.get("health") != "down":
return None
return Alert(
title=f"{node or 'A node'} lost its connection",
@@ -253,7 +253,7 @@ class AlertManager:
if muted_until is not None:
if now < muted_until:
# Still flapping — push the window out and stay quiet.
if event.get("status") == "down":
if event.get("health") == "down":
self._flapping[key] = now + FLAP_WINDOW_S
return True
del self._flapping[key]