Backend: real alert test results, state cleanup on delete/rename, node trigger errors, queue and collector fixes
`AlertManager.send` swallowed every delivery failure, so the alerts screen's Test button answered 200 whatever happened — the one thing it exists for. It takes `raise_on_error` now, which only the test route passes; the per-channel loop keeps the swallow, because one dead channel must not stop the others hearing about the same fault. A refused delivery answers 502 with whatever the sender said. Renaming a flow left its values under the old name for good: the delete path already swept them, the rename path never did. It calls the same `forget_flow`, which covers the messages and the `__ts__`/`__version__`/`__history__` bookkeeping keyed by message name. Cleanup, not migration — they repopulate under the new name on the next run. Triggering a node by hand ran `Node.__call__` with nothing catching it, so a node that raised produced a 500 and a stack trace in the server log, and nothing at all on the canvas. `Pipeline.publish_error` is the reporting half of `_execute_node` lifted out; both paths go through it, so a manual failure now reads the same on the canvas and in the metrics as a queued one. The route answers 400 with the node's error. `MemoryWorkQueue.stats()` counts claimed-but-unacknowledged work rather than reporting zero, so the health tile means something without Redis. The metrics collector's held tracebacks are capped at `DETAIL_CAP` and swept on the same `RUN_STALE_S` cutoff the open runs use, instead of one untruncated traceback per node kept for the life of the process — a traceback still survives the flush between the log and the failure it belongs to. `GET /observability/events` takes `since`/`until`, the window `/runs` already took, so a failures list can cover the span the charts beside it are drawn from.
This commit is contained in:
@@ -271,10 +271,21 @@ class AlertManager:
|
||||
# Delivery
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
async def send(self, channel: Channel, alert: Alert) -> None:
|
||||
async def send(
|
||||
self, channel: Channel, alert: Alert, raise_on_error: bool = False
|
||||
) -> None:
|
||||
"""Deliver one alert.
|
||||
|
||||
A failure is logged and swallowed, because one dead channel must not
|
||||
stop the others hearing about the same fault. The test button passes
|
||||
``raise_on_error``: telling a working channel from a broken one is the
|
||||
only thing it exists for.
|
||||
"""
|
||||
try:
|
||||
config = resolve_params(channel.config)
|
||||
except Exception as exc:
|
||||
if raise_on_error:
|
||||
raise
|
||||
logger.error("Channel '%s' has unusable settings: %s", channel.name, exc)
|
||||
return
|
||||
|
||||
@@ -286,6 +297,8 @@ class AlertManager:
|
||||
else:
|
||||
await self._send_email(config, alert)
|
||||
except Exception as exc:
|
||||
if raise_on_error:
|
||||
raise
|
||||
logger.error("Could not alert through '%s': %s", channel.name, exc)
|
||||
|
||||
async def _send_ntfy(self, config: dict[str, Any], alert: Alert) -> None:
|
||||
|
||||
Reference in New Issue
Block a user