Keep the engine's own history, and a screen that reads it

A second bus subscriber folds executions, errors, timings and queue lag
into per-minute rollups, keeps failures with their traceback and an audit
trail of who published what, and records one row per cascade — manual runs
and previews included, under an id of their own that writes no idempotency
markers. Read back through /observability/*, which always answers 200 so a
degraded engine still renders its own health screen.

Also fixes two things found on the way: node-health alerts read `status`
where the engine publishes `health`, so a device dropping never alerted
anyone, and the Redis queue reported `parked: 0` whatever was held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MeiWk3Yq12n2pTvnQWYvt
This commit is contained in:
2026-08-16 22:29:32 +02:00
co-authored by Claude Fable 5
parent f300c43f3a
commit af3ba51571
30 changed files with 2610 additions and 22 deletions
+9 -3
View File
@@ -122,11 +122,11 @@ def test_a_flapping_connection_goes_quiet():
async def scenario():
for _ in range(FLAP_THRESHOLD * 2 + 6):
await alerts.handle(
{"type": "node_health", "node": "heating.pump", "status": "down"}
{"type": "node_health", "node": "heating.pump", "health": "down"}
)
clock.advance(5)
await alerts.handle(
{"type": "node_health", "node": "heating.pump", "status": "ok"}
{"type": "node_health", "node": "heating.pump", "health": "ok"}
)
clock.advance(5)
@@ -185,7 +185,13 @@ def test_alerting_can_be_switched_off():
"The work queue is unreachable",
),
({"type": "engine_degraded", "reason": "lag"}, "The engine is struggling"),
({"type": "node_health", "status": "ok"}, None),
({"type": "node_health", "health": "ok"}, None),
# The engine publishes `health`, not `status`: reading the wrong key
# meant a device dropping never alerted anyone.
(
{"type": "node_health", "node": "heating.pump", "health": "down"},
"heating.pump lost its connection",
),
],
)
def test_every_alerting_event_reads_as_a_sentence(event, expected):