Say how much work is waiting, not just how much is running

`RedisWorkQueue.stats` read XPENDING, which counts entries delivered to a
consumer and not yet acknowledged — work in progress. Entries sitting in the
stream undelivered were counted nowhere, so an engine hours behind reported
itself idle: on the house, `pending: 4` while the group's lag was 1554.

The group's own `lag` is the missing number. `backlog` now carries it on both
queues (`len(_items)` in memory), leads the health tile, and a sustained one
publishes `engine_degraded` from the timer thread — named with the flow most
of the waiting work belongs to, sampled from the undelivered tail, since that
is the actionable half. It is a summary problem rather than a /utils/health
503: a backlog should not restart the container.

Also drops the keyspace `scan_iter` `stats()` did per poll to count parked
items — it walked every state and idempotency key twice per ten seconds — for
a set the park/unpark path maintains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
This commit is contained in:
2026-08-26 09:43:23 +02:00
co-authored by Claude Opus 5
parent dd7db026e1
commit 5726c80948
6 changed files with 619 additions and 13 deletions
@@ -144,10 +144,13 @@ export function HealthOverview({
: "nothing recorded"
}
/>
{/* Backlog leads: what is waiting is what says the engine is
behind. `pending` is work already running, which reads as idle
on an engine hours behind. */}
<Tile
label="Queue in flight"
value={String(queue.pending ?? 0)}
note={`${queue.delayed ?? 0} waiting · ${queue.parked ?? 0} parked`}
label="Queue backlog"
value={si(queue.backlog ?? 0)}
note={`${queue.pending ?? 0} in flight · ${queue.delayed ?? 0} delayed · ${queue.parked ?? 0} parked`}
/>
<Tile
label="Loop lag"