stroblme
bfaaa7cd03
The alert manager kept its cooldowns and its ceiling per hour in memory, so a node that stayed down was announced again by every restart. Each delivered alert is now a row in the new alert_sent table (additive migration), loaded for the last seven days when the manager starts and pruned as new ones come in. A ledger that fails never stops an alert. The manager's clock is wall time for the same reason. A node_health "down" alert also waits 60 s (HEALTH_GRACE_S) in a pending task per node; a later event that is not "down" cancels it, as does stopping the manager, so a broker restarting with the stack alerts nobody. The cooldown used to be stamped before the ceiling per hour was checked, so an alert the ceiling held back still spent its cooldown. It is stamped only once the alert is sent now.
35 lines
1.4 KiB
Python
35 lines
1.4 KiB
Python
"""The alerts the manager has sent, in the engine's database.
|
|
|
|
Cooldowns used to live in memory, so a node that stayed down was announced
|
|
again by every restart. One row per delivered alert is enough to rebuild both
|
|
the cooldowns and the ceiling per hour: the newest row of a key is when it was
|
|
last sent, and the rows of the last hour are what the ceiling counts. Rows older
|
|
than the retention are deleted as new ones come in; a cooldown longer than that
|
|
is not supported.
|
|
|
|
Blocking: the manager calls it through ``asyncio.to_thread``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from sqlalchemy import delete
|
|
from sqlmodel import Session, col, select
|
|
|
|
from fluksio.core.db import engine
|
|
from fluksio.models import AlertSent
|
|
|
|
|
|
class AlertLedger:
|
|
def load(self, since: float) -> list[tuple[str, float]]:
|
|
"""Every alert sent at or after *since*, as ``(key, ts)``."""
|
|
with Session(engine) as session:
|
|
rows = session.exec(select(AlertSent).where(AlertSent.ts >= since)).all()
|
|
return [(row.key, row.ts) for row in rows]
|
|
|
|
def add(self, key: str, ts: float, keep_since: float) -> None:
|
|
"""Record one alert, and drop what is older than *keep_since*."""
|
|
with Session(engine) as session:
|
|
session.add(AlertSent(key=key, ts=ts))
|
|
session.exec(delete(AlertSent).where(col(AlertSent.ts) < keep_since))
|
|
session.commit()
|