Let a quarantine expire, and stop two tasks spending one budget

A house's inverter broker dropped at 04:27 and the power flow was quarantined
20 seconds later. Quarantine was terminal — the supervised task returned and
only a publish or an engine restart could bring it back — so five hours of
power and battery readings are missing, and what ended it was an unrelated
`git pull` restarting uvicorn.

Two changes, both in that path:

- the failure budget is per task, not per flow. `power` runs an MQTT subscriber
  and a Victron keepalive publisher against the same broker; they died together
  and spent one shared budget in 41s, giving up before the 60s backoff step was
  ever reached.
- quarantine is now a rest. The task sits out 5min, then 15, then an hour, and
  each time gets its budget back and tries again, so a broker that comes back
  is picked up without anyone watching. `quarantined` reads from whichever
  tasks are currently resting.

The alert for it says when it will try again.
This commit is contained in:
2026-08-28 10:35:13 +02:00
parent 617e77c76b
commit 94518ef2eb
3 changed files with 140 additions and 36 deletions
+3 -1
View File
@@ -114,10 +114,12 @@ def describe(event: dict[str, Any]) -> Alert | None:
node=node,
)
if kind == "flow_quarantined":
retry = event.get("retry_in_s")
again = f" Trying again in {round(float(retry) / 60)} min." if retry else ""
return Alert(
title=f"Flow '{flow}' was quarantined",
body=(
f"It kept crashing, so the engine stopped restarting it. {error}"
f"It kept crashing, so the engine stood it down. {error}{again}"
).strip(),
flow=flow,
)