Let a quarantine expire, and stop two tasks spending one budget
Docs / docs (push) Successful in 23s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m1s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m44s
pre-commit / pre-commit (push) Failing after 2m50s
Test Backend / test-backend (push) Successful in 2m39s
Compose Smoke Test / test-compose (push) Successful in 31s
Playwright Tests / merge-reports (push) Successful in 1m9s

A house's inverter broker dropped at 04:27 and the power flow was quarantined
20 seconds later. Quarantine was terminal — the supervised task returned and
only a publish or an engine restart could bring it back — so five hours of
power and battery readings are missing, and what ended it was an unrelated
`git pull` restarting uvicorn.

Two changes, both in that path:

- the failure budget is per task, not per flow. `power` runs an MQTT subscriber
  and a Victron keepalive publisher against the same broker; they died together
  and spent one shared budget in 41s, giving up before the 60s backoff step was
  ever reached.
- quarantine is now a rest. The task sits out 5min, then 15, then an hour, and
  each time gets its budget back and tries again, so a broker that comes back
  is picked up without anyone watching. `quarantined` reads from whichever
  tasks are currently resting.

The alert for it says when it will try again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C5H4uLCCpsbipL1R7WKCee
This commit is contained in:
2026-08-28 10:35:13 +02:00
co-authored by Claude Opus 5
parent e40bbb9245
commit 8f1e685526
3 changed files with 140 additions and 36 deletions
+3 -1
View File
@@ -114,10 +114,12 @@ def describe(event: dict[str, Any]) -> Alert | None:
node=node,
)
if kind == "flow_quarantined":
retry = event.get("retry_in_s")
again = f" Trying again in {round(float(retry) / 60)} min." if retry else ""
return Alert(
title=f"Flow '{flow}' was quarantined",
body=(
f"It kept crashing, so the engine stopped restarting it. {error}"
f"It kept crashing, so the engine stood it down. {error}{again}"
).strip(),
flow=flow,
)