Alerts: cooldowns survive a restart, and a connection must stay down a minute before it alerts
The alert manager kept its cooldowns and its ceiling per hour in memory, so a node that stayed down was announced again by every restart. Each delivered alert is now a row in the new alert_sent table (additive migration), loaded for the last seven days when the manager starts and pruned as new ones come in. A ledger that fails never stops an alert. The manager's clock is wall time for the same reason. A node_health "down" alert also waits 60 s (HEALTH_GRACE_S) in a pending task per node; a later event that is not "down" cancels it, as does stopping the manager, so a broker restarting with the stack alerts nobody. The cooldown used to be stamped before the ceiling per hour was checked, so an alert the ceiling held back still spent its cooldown. It is stamped only once the alert is sent now.
This commit is contained in:
@@ -83,9 +83,13 @@ matches every supported alert event, including future additions.
|
||||
| `queue_unavailable` | the work queue becomes unreachable |
|
||||
| `run_finished` | a batch run ends in error |
|
||||
|
||||
Repeated faults are deduplicated with a 15-minute default cooldown.
|
||||
Flapping connections are suppressed, and the engine caps alert dispatches at
|
||||
ten per hour. A quiet channel may be cooling down; **Test** helps distinguish
|
||||
Repeated faults are deduplicated with a 15-minute default cooldown. The
|
||||
engine remembers what it sent in its database, so a restart does not start
|
||||
every cooldown over, and the ten-per-hour cap holds across one too. Cooldowns
|
||||
longer than seven days are not kept. A `node_health` alert waits until the
|
||||
connection has been down for a minute, so a broker that restarts with the
|
||||
stack or a device that reconnects alerts nobody. Flapping connections are
|
||||
suppressed. A quiet channel may be cooling down; **Test** helps distinguish
|
||||
that from a broken configuration.
|
||||
|
||||
## Admin: users and clients
|
||||
|
||||
Reference in New Issue
Block a user