Let a quarantine expire, and stop two tasks spending one budget
Docs / docs (push) Successful in 23s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m1s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m44s
pre-commit / pre-commit (push) Failing after 2m50s
Test Backend / test-backend (push) Successful in 2m39s
Compose Smoke Test / test-compose (push) Successful in 31s
Playwright Tests / merge-reports (push) Successful in 1m9s
Docs / docs (push) Successful in 23s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m1s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m44s
pre-commit / pre-commit (push) Failing after 2m50s
Test Backend / test-backend (push) Successful in 2m39s
Compose Smoke Test / test-compose (push) Successful in 31s
Playwright Tests / merge-reports (push) Successful in 1m9s
A house's inverter broker dropped at 04:27 and the power flow was quarantined 20 seconds later. Quarantine was terminal — the supervised task returned and only a publish or an engine restart could bring it back — so five hours of power and battery readings are missing, and what ended it was an unrelated `git pull` restarting uvicorn. Two changes, both in that path: - the failure budget is per task, not per flow. `power` runs an MQTT subscriber and a Victron keepalive publisher against the same broker; they died together and spent one shared budget in 41s, giving up before the 60s backoff step was ever reached. - quarantine is now a rest. The task sits out 5min, then 15, then an hour, and each time gets its budget back and tries again, so a broker that comes back is picked up without anyone watching. `quarantined` reads from whichever tasks are currently resting. The alert for it says when it will try again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5H4uLCCpsbipL1R7WKCee
This commit is contained in:
@@ -114,10 +114,12 @@ def describe(event: dict[str, Any]) -> Alert | None:
|
||||
node=node,
|
||||
)
|
||||
if kind == "flow_quarantined":
|
||||
retry = event.get("retry_in_s")
|
||||
again = f" Trying again in {round(float(retry) / 60)} min." if retry else ""
|
||||
return Alert(
|
||||
title=f"Flow '{flow}' was quarantined",
|
||||
body=(
|
||||
f"It kept crashing, so the engine stopped restarting it. {error}"
|
||||
f"It kept crashing, so the engine stood it down. {error}{again}"
|
||||
).strip(),
|
||||
flow=flow,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user