Tell someone when the engine breaks
Everything that goes wrong already travelled the event bus, but the only subscriber was the editor's websocket — so a flow quarantined at three in the morning was invisible until someone opened the browser. An alert manager now watches the same bus and forwards failures to ntfy, email or a webhook. Most of what it does is decline to send: the same node failing every second is one alert with a count of what followed, a connection flapping up and down is muted until it settles, and nothing gets past ten notifications an hour. Verified against a live instance — six identical failures produced one alert carrying the real traceback message. Channels and rules are configured through the API, with a test send so a channel can be proven before anything depends on it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
This commit is contained in:
+5
-1
@@ -71,7 +71,11 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend – M
|
||||
- [x] Per-input/-output discretization interval setting: a port publishes, or
|
||||
wakes its node, at most every n seconds. State keeps the latest value, so
|
||||
only the delivery is skipped
|
||||
- [ ] Alert / notification handler
|
||||
- [x] Alert / notification handler: engine failures — a node raising, a connection
|
||||
dropping, a flow quarantined, the queue gone — reach ntfy, email or a webhook.
|
||||
Mostly it declines to send: the same fault repeating is one alert with a count,
|
||||
a flapping connection is muted, and there is a ceiling per hour. Configured
|
||||
through the API at `/alerts/config`, with a test send per channel
|
||||
- [x] Deep health check (`GET /utils/health/`): reports event-loop lag and state-backend
|
||||
reachability and fails the container healthcheck, so a wedged engine is restarted
|
||||
rather than counted as up. One engine per deployment — the API image runs a single
|
||||
|
||||
Reference in New Issue
Block a user