stroblme
bfaaa7cd03
The alert manager kept its cooldowns and its ceiling per hour in memory, so a node that stayed down was announced again by every restart. Each delivered alert is now a row in the new alert_sent table (additive migration), loaded for the last seven days when the manager starts and pruned as new ones come in. A ledger that fails never stops an alert. The manager's clock is wall time for the same reason. A node_health "down" alert also waits 60 s (HEALTH_GRACE_S) in a pending task per node; a later event that is not "down" cancels it, as does stopping the manager, so a broker restarting with the stack alerts nobody. The cooldown used to be stamped before the ceiling per hour was checked, so an alert the ceiling held back still spent its cooldown. It is stamped only once the alert is sent now.
27 lines
821 B
Python
27 lines
821 B
Python
"""The alert ledger: what the manager remembers across a restart."""
|
|
|
|
import pytest
|
|
from sqlalchemy import delete
|
|
from sqlmodel import Session
|
|
|
|
from fluksio.core.db import engine
|
|
from fluksio.flow.alert_ledger import AlertLedger
|
|
from fluksio.models import AlertSent
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _empty(db: Session): # noqa: ARG001 - the schema has to exist first
|
|
with Session(engine) as session:
|
|
session.exec(delete(AlertSent))
|
|
session.commit()
|
|
yield
|
|
|
|
|
|
def test_a_sent_alert_comes_back_and_old_ones_are_pruned():
|
|
ledger = AlertLedger()
|
|
ledger.add("node_error:house.calc", 100.0, keep_since=0.0)
|
|
ledger.add("node_health:house.mqtt", 5000.0, keep_since=1000.0)
|
|
|
|
assert ledger.load(since=0.0) == [("node_health:house.mqtt", 5000.0)]
|
|
assert ledger.load(since=6000.0) == []
|