Say how much work is waiting, not just how much is running
`RedisWorkQueue.stats` read XPENDING, which counts entries delivered to a consumer and not yet acknowledged — work in progress. Entries sitting in the stream undelivered were counted nowhere, so an engine hours behind reported itself idle: on the house, `pending: 4` while the group's lag was 1554. The group's own `lag` is the missing number. `backlog` now carries it on both queues (`len(_items)` in memory), leads the health tile, and a sustained one publishes `engine_degraded` from the timer thread — named with the flow most of the waiting work belongs to, sampled from the undelivered tail, since that is the actionable half. It is a summary problem rather than a /utils/health 503: a backlog should not restart the container. Also drops the keyspace `scan_iter` `stats()` did per poll to count parked items — it walked every state and idempotency key twice per ten seconds — for a set the park/unpark path maintains. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
This commit is contained in:
@@ -148,6 +148,8 @@ async def read_summary(request: Request, controller: FlowControllerDep) -> Any:
|
||||
queue = await run_in_threadpool(controller.queue_stats)
|
||||
if queue.get("oldest_pending_s", 0) > 120:
|
||||
problems.append("queue stalled")
|
||||
if queue.get("behind"):
|
||||
problems.append(f"engine behind: {queue.get('backlog', 0)} items waiting")
|
||||
if queue.get("error"):
|
||||
problems.append("work queue unreachable")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user