Let a quarantine expire, and stop two tasks spending one budget
A house's inverter broker dropped at 04:27 and the power flow was quarantined 20 seconds later. Quarantine was terminal — the supervised task returned and only a publish or an engine restart could bring it back — so five hours of power and battery readings are missing, and what ended it was an unrelated `git pull` restarting uvicorn. Two changes, both in that path: - the failure budget is per task, not per flow. `power` runs an MQTT subscriber and a Victron keepalive publisher against the same broker; they died together and spent one shared budget in 41s, giving up before the 60s backoff step was ever reached. - quarantine is now a rest. The task sits out 5min, then 15, then an hour, and each time gets its budget back and tries again, so a broker that comes back is picked up without anyone watching. `quarantined` reads from whichever tasks are currently resting. The alert for it says when it will try again.
This commit is contained in:
@@ -114,10 +114,12 @@ def describe(event: dict[str, Any]) -> Alert | None:
|
||||
node=node,
|
||||
)
|
||||
if kind == "flow_quarantined":
|
||||
retry = event.get("retry_in_s")
|
||||
again = f" Trying again in {round(float(retry) / 60)} min." if retry else ""
|
||||
return Alert(
|
||||
title=f"Flow '{flow}' was quarantined",
|
||||
body=(
|
||||
f"It kept crashing, so the engine stopped restarting it. {error}"
|
||||
f"It kept crashing, so the engine stood it down. {error}{again}"
|
||||
).strip(),
|
||||
flow=flow,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user