Keep the engine's own history, and a screen that reads it

A second bus subscriber folds executions, errors, timings and queue lag
into per-minute rollups, keeps failures with their traceback and an audit
trail of who published what, and records one row per cascade — manual runs
and previews included, under an id of their own that writes no idempotency
markers. Read back through /observability/*, which always answers 200 so a
degraded engine still renders its own health screen.

Also fixes two things found on the way: node-health alerts read `status`
where the engine publishes `health`, so a device dropping never alerted
anyone, and the Redis queue reported `parked: 0` whatever was held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MeiWk3Yq12n2pTvnQWYvt
This commit is contained in:
2026-08-16 22:29:32 +02:00
co-authored by Claude Fable 5
parent f300c43f3a
commit af3ba51571
30 changed files with 2610 additions and 22 deletions
+42
View File
@@ -265,3 +265,45 @@ async def apply_modules(requirements: str) -> Any:
manifest that does not resolve changes nothing.
"""
return await _call("POST", "/modules/apply", json={"requirements": requirements})
# -----------------------------------------------------------------------------
# Observability
# -----------------------------------------------------------------------------
@mcp.tool()
async def get_health() -> Any:
"""How the engine is doing: flows, nodes, queue, loop lag and recent failures."""
return await _call("GET", "/observability/summary")
@mcp.tool()
async def get_metrics(
flow: str | None = None, node: str | None = None, hours: int = 24
) -> Any:
"""Executions, errors and timings per minute over the last few hours."""
params: dict[str, Any] = {"hours": hours}
if flow:
params["flow"] = flow
if node:
params["node"] = node
return await _call("GET", "/observability/timeseries", params=params)
@mcp.tool()
async def list_failures(flow: str | None = None, limit: int = 50) -> Any:
"""Recent failures with their tracebacks, newest first."""
params: dict[str, Any] = {"kind": "failure", "limit": limit}
if flow:
params["flow"] = flow
return await _call("GET", "/observability/events", params=params)
@mcp.tool()
async def list_runs(flow: str | None = None, limit: int = 50) -> Any:
"""Recent cascades: what triggered them, how long they took, how they ended."""
params: dict[str, Any] = {"limit": limit}
if flow:
params["flow"] = flow
return await _call("GET", "/observability/runs", params=params)