Keep the engine's own history, and a screen that reads it

A second bus subscriber folds executions, errors, timings and queue lag
into per-minute rollups, keeps failures with their traceback and an audit
trail of who published what, and records one row per cascade — manual runs
and previews included, under an id of their own that writes no idempotency
markers. Read back through /observability/*, which always answers 200 so a
degraded engine still renders its own health screen.

Also fixes two things found on the way: node-health alerts read `status`
where the engine publishes `health`, so a device dropping never alerted
anyone, and the Redis queue reported `parked: 0` whatever was held.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MeiWk3Yq12n2pTvnQWYvt
This commit is contained in:
2026-08-16 22:29:32 +02:00
co-authored by Claude Fable 5
parent f300c43f3a
commit af3ba51571
30 changed files with 2610 additions and 22 deletions
+9 -1
View File
@@ -29,7 +29,6 @@ is what M4 still waits on, together with porting the flows.
### Bugs found while building the screens
- BUG/API: `POST /alerts/test/{channel}` always answers 200. `AlertManager.send()` catches and logs every delivery failure, so the alerts screen's Test button cannot tell a working channel from a broken one — the one thing it exists for. Let `send()` raise or return a result on the test path.
- BUG/API: `RedisWorkQueue.stats()` hard-codes `"parked": 0`, so held or stranded work never shows on the health endpoint. This is why a rebuild stranding parked items went unnoticed until it was looked for.
- BUG/FLOW: deleting a flow leaves its `pipeline:{flow}.*` Redis keys behind, and renaming one does not migrate them — the live instance carries `pipeline:__history__:dashboar.test` beside the correct `dashboard.test`. One cleanup on the delete/rename path covers both.
- CHORE/API: revoking an OAuth client does not invalidate access tokens already issued; they are stateless JWTs valid up to `MCP_TOKEN_EXPIRE_MINUTES`. Immediate revocation means `app/mcp/http.py` checking the client row still exists.
- CHORE/FLOW: `Pipeline.trigger`'s docstring says a paused flow still publishes so the value shows on the canvas. True only without a queue; with one the item parks before `apply_outputs` and nothing shows. Docstring and behaviour disagree.
@@ -44,6 +43,15 @@ is what M4 still waits on, together with porting the flows.
- FEAT/API: `POST /modules/apply` rebuilds the whole pipeline so a node that could not import its package stops being red. That resubscribes every MQTT node in the deployment; a targeted rebuild of the flows that actually failed to load would be gentler.
- CHORE/FLOW: a node's return value now round-trips through JSON, so tuples arrive downstream as lists and anything non-JSON is an explicit error. That is the message contract, but flows written before this may notice.
### Engine history
- CHORE/FLOW: a rate-limit flush gets no run record — it is the tail of the run that scheduled it, and there is no id linking the two. A flush that fails therefore shows as a failure with no run beside it.
- CHORE/FLOW: `Pipeline.flush` releasing a held value runs its cascade without a run id, so those executions land in the minute rollups but in no run. Threading the scheduling run's id through the queue item would close it.
- CHORE/API: the metrics collector is a bus subscriber, so a storm that overflows the bus queue undercounts. The events dropped are the same ones the websocket drops; exact accounting would need the collector to be fed from the engine rather than the bus.
- CHORE/API: `/observability/summary` reports the work queue's `depth` as the Redis stream length, which is the journal size (capped at `STREAM_MAXLEN`) rather than a backlog. The health screen shows `pending` instead; the field name still invites the wrong reading.
- FEAT/UI: the health screen's window is fixed at 24 hours and the charts fold minute buckets in Python. A range picker (and `date_bin()` behind it) is the next step if anyone wants a week.
- CHORE/FLOW: run records for a deleted flow stay until the retention window passes, so a flow that no longer exists keeps appearing in the history. Deliberate — it is a record of what ran — but `forget_flow` could offer to clear it.
### Dashboard follow-ups
- BUG/UI: ensure dashboard wallpanel (read-only) links hot reload automatically on dashboard changes