Say how much work is waiting, not just how much is running

`RedisWorkQueue.stats` read XPENDING, which counts entries delivered to a
consumer and not yet acknowledged — work in progress. Entries sitting in the
stream undelivered were counted nowhere, so an engine hours behind reported
itself idle: on the house, `pending: 4` while the group's lag was 1554.

The group's own `lag` is the missing number. `backlog` now carries it on both
queues (`len(_items)` in memory), leads the health tile, and a sustained one
publishes `engine_degraded` from the timer thread — named with the flow most
of the waiting work belongs to, sampled from the undelivered tail, since that
is the actionable half. It is a summary problem rather than a /utils/health
503: a backlog should not restart the container.

Also drops the keyspace `scan_iter` `stats()` did per poll to count parked
items — it walked every state and idempotency key twice per ten seconds — for
a set the park/unpark path maintains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
This commit is contained in:
2026-08-26 09:43:23 +02:00
co-authored by Claude Opus 5
parent dd7db026e1
commit 5726c80948
6 changed files with 619 additions and 13 deletions
@@ -148,6 +148,8 @@ async def read_summary(request: Request, controller: FlowControllerDep) -> Any:
queue = await run_in_threadpool(controller.queue_stats)
if queue.get("oldest_pending_s", 0) > 120:
problems.append("queue stalled")
if queue.get("behind"):
problems.append(f"engine behind: {queue.get('backlog', 0)} items waiting")
if queue.get("error"):
problems.append("work queue unreachable")