Backend: real alert test results, state cleanup on delete/rename, node trigger errors, queue and collector fixes
`AlertManager.send` swallowed every delivery failure, so the alerts screen's Test button answered 200 whatever happened — the one thing it exists for. It takes `raise_on_error` now, which only the test route passes; the per-channel loop keeps the swallow, because one dead channel must not stop the others hearing about the same fault. A refused delivery answers 502 with whatever the sender said. Renaming a flow left its values under the old name for good: the delete path already swept them, the rename path never did. It calls the same `forget_flow`, which covers the messages and the `__ts__`/`__version__`/`__history__` bookkeeping keyed by message name. Cleanup, not migration — they repopulate under the new name on the next run. Triggering a node by hand ran `Node.__call__` with nothing catching it, so a node that raised produced a 500 and a stack trace in the server log, and nothing at all on the canvas. `Pipeline.publish_error` is the reporting half of `_execute_node` lifted out; both paths go through it, so a manual failure now reads the same on the canvas and in the metrics as a queued one. The route answers 400 with the node's error. `MemoryWorkQueue.stats()` counts claimed-but-unacknowledged work rather than reporting zero, so the health tile means something without Redis. The metrics collector's held tracebacks are capped at `DETAIL_CAP` and swept on the same `RUN_STALE_S` cutoff the open runs use, instead of one untruncated traceback per node kept for the life of the process — a traceback still survives the flush between the log and the failure it belongs to. `GET /observability/events` takes `since`/`until`, the window `/runs` already took, so a failures list can cover the span the charts beside it are drawn from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XC2jX6Hdj7pxGGKzBTrbqB
This commit is contained in:
@@ -11,7 +11,7 @@ from datetime import datetime, timezone
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from app.flow.events import EventBus
|
||||
from app.flow.metrics import MetricsCollector
|
||||
from app.flow.metrics import DETAIL_CAP, RUN_STALE_S, MetricsCollector
|
||||
from app.models import EngineEvent, FlowRun, MetricBucket
|
||||
|
||||
FLOW = "metrics-test"
|
||||
@@ -197,3 +197,30 @@ def test_a_run_that_did_not_fail_reads_ok(db: Session) -> None:
|
||||
|
||||
run = db.exec(select(FlowRun).where(FlowRun.id == "manual-abc")).one()
|
||||
assert (run.status, run.source) == ("ok", "manual")
|
||||
|
||||
|
||||
def test_a_traceback_no_failure_ever_claims_is_dropped() -> None:
|
||||
"""One entry per node that ever errored would else last the process out."""
|
||||
node = f"{FLOW}.stale"
|
||||
collector = MetricsCollector(EventBus())
|
||||
collector.handle(
|
||||
{
|
||||
"type": "node_log",
|
||||
"flow": FLOW,
|
||||
"node": node,
|
||||
"level": "error",
|
||||
"text": "x" * (DETAIL_CAP * 2),
|
||||
"ts": time.time() - RUN_STALE_S - 1,
|
||||
}
|
||||
)
|
||||
|
||||
# Untruncated, one of these holds a whole run of a chatty node.
|
||||
assert len(collector._tracebacks[(FLOW, node)][1]) == DETAIL_CAP
|
||||
|
||||
# Something to write, so the flush does not stop at its early return.
|
||||
collector.handle(
|
||||
{"type": "node_executed", "flow": FLOW, "node": node, "ts": time.time()}
|
||||
)
|
||||
asyncio.run(collector.flush())
|
||||
|
||||
assert collector._tracebacks == {}
|
||||
|
||||
Reference in New Issue
Block a user