A house's inverter broker dropped at 04:27 and the power flow was quarantined
20 seconds later. Quarantine was terminal — the supervised task returned and
only a publish or an engine restart could bring it back — so five hours of
power and battery readings are missing, and what ended it was an unrelated
`git pull` restarting uvicorn.
Two changes, both in that path:
- the failure budget is per task, not per flow. `power` runs an MQTT subscriber
and a Victron keepalive publisher against the same broker; they died together
and spent one shared budget in 41s, giving up before the 60s backoff step was
ever reached.
- quarantine is now a rest. The task sits out 5min, then 15, then an hour, and
each time gets its budget back and tries again, so a broker that comes back
is picked up without anyone watching. `quarantined` reads from whichever
tasks are currently resting.
The alert for it says when it will try again.
A node's stop() and a supervised task's cancellation are both waited on
inside the rebuild lock, and neither had a deadline: an MQTT client whose
broker never acknowledges the disconnect leaves aiomqtt's __aexit__
waiting forever, so reload() never returned and every start, stop or
publish behind it hung until the container was restarted.
Each node now gets five seconds to close and is abandoned after that, and
cancel_all reports what is still running rather than waiting on it — it
also no longer swallows a cancellation aimed at the caller, which used to
make the lock holder unkillable. A rebuild asked for by a request gives up
on the lock after fifteen seconds with RebuildBusy, answered as a 503.
A wheel whose top-level module is `app` collides with anything else in a
user's venv, so the package that is about to be published takes the name
it is published under. Only the Python package moves; the repo, the
Docker WORKDIR and the compose project keep theirs.
stroblme