Tell someone when the engine breaks

Everything that goes wrong already travelled the event bus, but the only
subscriber was the editor's websocket — so a flow quarantined at three in
the morning was invisible until someone opened the browser.

An alert manager now watches the same bus and forwards failures to ntfy,
email or a webhook. Most of what it does is decline to send: the same
node failing every second is one alert with a count of what followed, a
connection flapping up and down is muted until it settles, and nothing
gets past ten notifications an hour. Verified against a live instance —
six identical failures produced one alert carrying the real traceback
message.

Channels and rules are configured through the API, with a test send so a
channel can be proven before anything depends on it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
This commit is contained in:
root
2026-08-16 08:01:50 +02:00
co-authored by Claude Fable 5
parent 04329149b3
commit dbdbcc1091
12 changed files with 824 additions and 3 deletions
+5 -1
View File
@@ -71,7 +71,11 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend M
- [x] Per-input/-output discretization interval setting: a port publishes, or
wakes its node, at most every n seconds. State keeps the latest value, so
only the delivery is skipped
- [ ] Alert / notification handler
- [x] Alert / notification handler: engine failures — a node raising, a connection
dropping, a flow quarantined, the queue gone — reach ntfy, email or a webhook.
Mostly it declines to send: the same fault repeating is one alert with a count,
a flapping connection is muted, and there is a ceiling per hour. Configured
through the API at `/alerts/config`, with a test send per channel
- [x] Deep health check (`GET /utils/health/`): reports event-loop lag and state-backend
reachability and fails the container healthcheck, so a wedged engine is restarted
rather than counted as up. One engine per deployment — the API image runs a single