Cut the round trips a message costs the engine

Measured with `make bench-engine` against a real Redis: 103.6 -> 164.4
messages a second on a five-node chain (p50 latency 2125 -> 1171 ms) and
34.8 -> 63.2 on a fan-out of twenty. Against the memory backend, which is
what a pip install runs on, 262 -> 626.

The two that bought most of it:

- `StateBackend.record` puts a published value, its timestamp, its series
  and its version counter in one round trip. They were four calls building
  four pipelines, and a value crossing an edge pays them twice. A released
  rate-limit hold rides along instead of a DEL per port.
- the readiness check reads a node's inputs and hands them to the node,
  rather than reading the triggering ones to count them and having the node
  read the same keys again a moment later.

`apply_outputs` was a second copy of `_record_outputs` and is now the same
code plus the event that distinguishes it.

The rest, each small:

- `_derive` builds a node-by-id map and a `consumes` index, so dispatching
  an item and publishing a value stop scanning every node in the
  installation.
- `read_all` is memoised against the store revision — it sits on the
  publish path, so a dashboard slider was reading and validating every
  flow file per value. Same mechanism `_wiring` already uses.
- the `message_value` source block is built once per node instead of per
  emission.
- both timer threads ask the queue to promote only when something is
  actually due, which takes an idle engine from ~4 Redis round trips a
  second to one.
- the shared httpx client is bounded (32 connections, one retry); its
  default pool is 100 with no per-host cap, so one slow endpoint could
  take it and every other sender node with it.
- the MQTT and delay nodes no longer log a line per message at INFO.

Robustness, in the same pass:

- `MemoryWorkQueue._done` was a set nothing ever removed from — one entry
  per non-idempotent node per item, for the life of the process, in the
  default configuration. Capped, the way the Redis side expires its
  markers.
- a saturated engine can claim from the due lane past the cascade limit.
  The capacity gate sits in front of the claim, so the due lane's priority
  — decided inside it — did not apply while every slot was held: a motor's
  stop was not behind the long nodes, it was unread. Only after a slot has
  genuinely failed to free for half a second, and briefly, so the backlog
  is not starved in turn.
- `reclaim_stale` dispatches through that same gate. It could return sixty
  entries and push in-flight far past the limit the gate exists to hold.
- a flow's nodes are stopped together rather than one after another. Each
  gets `NODE_STOP_TIMEOUT`, so a flow whose broker was unreachable took
  five seconds per node — long enough to outlast `REBUILD_WAIT` and 503
  the deploy.
- the worker pool and the HTTP client are closed on a thread, not on the
  event loop, and a run closes the state backend it built (on Redis, a
  client and a connection pool per run).
- the five background tasks say something when they die. Each catches
  exceptions inside its loop, so one raised anywhere else left the engine
  serving with no metrics, no alerts or no artifact sweep, silently.

`tests/flow/test_round_trips.py` counts the state operations one message
costs — four, where it was about eleven — because none of the above would
fail a behavioural test if it were undone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6hPWS6YEbT1P8LxhhFb2T

backend/fluksio/flow/queue.py

@@ -17,7 +17,7 @@ import logging
 import threading
 import time
 from abc import ABC, abstractmethod
-from collections import deque
+from collections import OrderedDict, deque
 from collections.abc import Callable
 from dataclasses import dataclass, field
 from typing import Any, cast
@@ -41,6 +41,10 @@ DUE_PREFIX = "due:"
 # A consumer this far past its last read belongs to an engine that is gone. A
 # live one interacts every claim, so nothing in service comes close.
 STALE_CONSUMER_IDLE_MS = 3_600_000
+# How many "this side effect already happened" markers the memory queue keeps.
+# The Redis one expires each after an hour; this is the same idea sized by
+# count, since redelivery happens seconds after the claim and never later.
+DONE_MARKERS = 10_000
 
 
 @dataclass
@@ -154,8 +158,15 @@ class WorkQueue(ABC):
         """
 
     @abstractmethod
-    def claim(self, count: int, block_ms: int) -> list[WorkItem]:
-        """Take up to ``count`` items, waiting up to ``block_ms`` for one."""
+    def claim(
+        self, count: int, block_ms: int, due_only: bool = False
+    ) -> list[WorkItem]:
+        """Take up to ``count`` items, waiting up to ``block_ms`` for one.
+
+        ``due_only`` takes nothing but promoted timers. A saturated engine
+        uses it to let a motor's stop past the cascade limit while every slot
+        is held by a long node.
+        """
 
     @abstractmethod
     def ack(self, item: WorkItem) -> None:
@@ -240,9 +251,17 @@ class MemoryWorkQueue(WorkQueue):
 
     def __init__(self) -> None:
         self._items: deque[WorkItem] = deque()
+        # Promoted timers, kept apart from the backlog the way Redis keeps
+        # two streams — so they can be claimed on their own.
+        self._due: deque[WorkItem] = deque()
         self._delayed: list[tuple[float, int, WorkItem]] = []
         self._parked: dict[str, list[WorkItem]] = {}
-        self._done: set[tuple[str, str]] = set()
+        # Insertion-ordered and capped, because this grows one entry per
+        # non-idempotent node per item and nothing ever removed one — the
+        # Redis side expires its markers after an hour, this one leaked for
+        # the life of the process. Redelivery is what the marker guards, and
+        # that happens within seconds of the claim.
+        self._done: OrderedDict[tuple[str, str], None] = OrderedDict()
         # Claimed and not yet acknowledged, which is what Redis's `pending` is.
         self._in_flight = 0
         self._counter = 0
@@ -277,17 +296,25 @@ class MemoryWorkQueue(WorkQueue):
         with self._lock:
             return self._delayed[0][0] if self._delayed else None
 
-    def claim(self, count: int, block_ms: int) -> list[WorkItem]:
+    def claim(
+        self, count: int, block_ms: int, due_only: bool = False
+    ) -> list[WorkItem]:
         deadline = time.monotonic() + block_ms / 1000.0
         with self._wake:
-            while not self._items:
+            while not self._due and not (self._items and not due_only):
                 remaining = deadline - time.monotonic()
                 if remaining <= 0:
                     return []
                 self._wake.wait(remaining)
-            claimed = [
-                self._items.popleft() for _ in range(min(count, len(self._items)))
-            ]
+            # Due first: an item that has waited out a deadline is late by
+            # however long it queues here, while work merely enqueued is not
+            # waiting on a clock.
+            claimed = [self._due.popleft() for _ in range(min(count, len(self._due)))]
+            if not due_only:
+                claimed += [
+                    self._items.popleft()
+                    for _ in range(min(count - len(claimed), len(self._items)))
+                ]
             self._in_flight += len(claimed)
             return claimed
 
@@ -306,10 +333,7 @@ class MemoryWorkQueue(WorkQueue):
             while self._delayed and self._delayed[0][0] <= now:
                 due.append(heapq.heappop(self._delayed)[2])
             if due:
-                # In front of the backlog, in due order: an item that has
-                # waited out a deadline is late by however long it queues
-                # here, while work merely enqueued is not waiting on a clock.
-                self._items.extendleft(reversed(due))
+                self._due.extend(due)
                 self._wake.notify()
         return len(due)
 
@@ -335,12 +359,12 @@ class MemoryWorkQueue(WorkQueue):
 
     def backlog(self) -> int:
         with self._lock:
-            return len(self._items)
+            return len(self._items) + len(self._due)
 
     def stats(self) -> dict[str, Any]:
         with self._lock:
             return {
-                "backlog": len(self._items),
+                "backlog": len(self._items) + len(self._due),
                 "pending": self._in_flight,
                 "delayed": len(self._delayed),
                 "parked": sum(len(v) for v in self._parked.values()),
@@ -354,7 +378,9 @@ class MemoryWorkQueue(WorkQueue):
 
     def mark_done(self, entry_id: str, node: str) -> None:
         with self._lock:
-            self._done.add((entry_id, node))
+            self._done[(entry_id, node)] = None
+            while len(self._done) > DONE_MARKERS:
+                self._done.popitem(last=False)
 
     def was_done(self, entry_id: str, node: str) -> bool:
         with self._lock:
@@ -467,7 +493,9 @@ class RedisWorkQueue(WorkQueue):
             return self._due_stream, entry_id[len(DUE_PREFIX) :]
         return self._stream, entry_id
 
-    def claim(self, count: int, block_ms: int) -> list[WorkItem]:
+    def claim(
+        self, count: int, block_ms: int, due_only: bool = False
+    ) -> list[WorkItem]:
         """Take up to ``count`` items, due timers before anything queued.
 
         One read over both streams rather than a read each: the block has to
@@ -483,7 +511,11 @@ class RedisWorkQueue(WorkQueue):
             self._redis.xreadgroup(
                 GROUP,
                 self._consumer,
-                {self._due_stream: ">", self._stream: ">"},
+                (
+                    {self._due_stream: ">"}
+                    if due_only
+                    else {self._due_stream: ">", self._stream: ">"}
+                ),
                 count=count,
                 block=block_ms,
             ),