Keep the engine's own history, and a screen that reads it
A second bus subscriber folds executions, errors, timings and queue lag into per-minute rollups, keeps failures with their traceback and an audit trail of who published what, and records one row per cascade — manual runs and previews included, under an id of their own that writes no idempotency markers. Read back through /observability/*, which always answers 200 so a degraded engine still renders its own health screen. Also fixes two things found on the way: node-health alerts read `status` where the engine publishes `health`, so a device dropping never alerted anyone, and the Redis queue reported `parked: 0` whatever was held.
This commit is contained in:
@@ -102,7 +102,7 @@ def describe(event: dict[str, Any]) -> Alert | None:
|
||||
node=node,
|
||||
)
|
||||
if kind == "node_health":
|
||||
if event.get("status") != "down":
|
||||
if event.get("health") != "down":
|
||||
return None
|
||||
return Alert(
|
||||
title=f"{node or 'A node'} lost its connection",
|
||||
@@ -253,7 +253,7 @@ class AlertManager:
|
||||
if muted_until is not None:
|
||||
if now < muted_until:
|
||||
# Still flapping — push the window out and stay quiet.
|
||||
if event.get("status") == "down":
|
||||
if event.get("health") == "down":
|
||||
self._flapping[key] = now + FLAP_WINDOW_S
|
||||
return True
|
||||
del self._flapping[key]
|
||||
|
||||
Reference in New Issue
Block a user