Commit Graph
5 Commits
Author SHA1 Message Date
stroblme f14f7bd413 Make narrowed backend test runs and the quarantine test order- and load-proof
The session db fixture now disposes the engine before deleting the file, since a
module under a no-op db (tests/sdk, tests/flow) may have connected first; the
sdk import tests forget the study packages they import; the supervision tests
keep the real quarantine length so it cannot expire mid-assertion.
2026-10-05 16:54:00 +02:00
stroblme 94518ef2eb Let a quarantine expire, and stop two tasks spending one budget
A house's inverter broker dropped at 04:27 and the power flow was quarantined
20 seconds later. Quarantine was terminal — the supervised task returned and
only a publish or an engine restart could bring it back — so five hours of
power and battery readings are missing, and what ended it was an unrelated
`git pull` restarting uvicorn.

Two changes, both in that path:

- the failure budget is per task, not per flow. `power` runs an MQTT subscriber
  and a Victron keepalive publisher against the same broker; they died together
  and spent one shared budget in 41s, giving up before the 60s backoff step was
  ever reached.
- quarantine is now a rest. The task sits out 5min, then 15, then an hour, and
  each time gets its budget back and tries again, so a broker that comes back
  is picked up without anyone watching. `quarantined` reads from whichever
  tasks are currently resting.

The alert for it says when it will try again.
2026-08-28 10:35:13 +02:00
stroblme 708318dc84 Let the supervisor stop one flow's tasks and clear its quarantine 2026-08-23 17:43:15 +02:00
stroblme 99534a82ee Rename the import package app to fluksio
A wheel whose top-level module is `app` collides with anything else in a
user's venv, so the package that is about to be published takes the name
it is published under. Only the Python package moves; the repo, the
Docker WORKDIR and the compose project keep theirs.
2026-08-21 21:48:05 +02:00
root 3a8edae928 Supervise the loops a flow starts, and give up loudly
A node's subscription, schedule or poll loop was a bare asyncio task: one
that raised outside its own retry handling was simply gone, and the node
went on being listed as running while nothing listened any more.

Those loops now run under a supervisor that restarts them with a growing
delay and quarantines a flow that burns through five restarts in five
minutes — a flow crash-looping every second is worse than one that is
visibly stopped, and the dashboard can now say which.

The MQTT subscription loses its private five-second reconnect in the
process: one backoff policy per socket, and it belongs to the supervisor.
2026-08-16 07:23:02 +02:00