Stop paying five Redis round trips and a global lock per message
The engine was I/O-bound on its own state backend. `RedisState.lock()` is one key — `pipeline:_lock` — for the whole process, taken five times a message at two round trips each, and every cascade and every node read queued behind it. Inside it, reading a node's inputs was three round trips per input (an EXISTS for `in`, then EXISTS and GET for the value), writing was two updates that a single transaction already gives, and the version counters went one INCR at a time. Replaced with the atomic command that was always available: `get_present` is one MGET and tells a missing key from one holding null, so the lock it used to be read under bought nothing; value and timestamp land in one `update`, which is a MULTI/EXEC; `increment_multi` pipelines the counters. `values()` — what every websocket snapshot calls — is two reads whatever the message count instead of two per message. Beside that: every webhook did its blocking XADD on the asyncio event loop (MQTT already used `to_thread`); the per-execution `NodeOutcome` was built and validated even with no run watching; `_minute` built a tz-aware datetime per event on the loop thread to key a dict, and now keys on an int; `move_due` promoted delayed items one round trip each, every second; `FLOW_MAX_CASCADES` makes the in-flight ceiling a setting rather than a constant. `orjson` replaces stdlib json where a message pays for it — state, the journal, the engine side of the worker pipe. `fluksio-worker` stays dependency-free, and the run-cache digest stays on stdlib so no stored key is invalidated. A non-finite number now stores as `null` rather than the bare `NaN` that was never JSON. Measured with `scripts/bench_engine.py` against a real Redis, 200 messages: a five-node chain went from 43.9 to 103.1 msg/s with p50 latency 2110ms → 782ms and p95 3913ms → 1439ms; one source into twenty consumers went from 5.4 to 33.7 msg/s. In memory, twenty consumers went from 187 to 448 msg/s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
This commit is contained in:
@@ -67,8 +67,16 @@ RECORDED = {
|
||||
}
|
||||
|
||||
|
||||
def _minute(ts: float) -> datetime:
|
||||
return datetime.fromtimestamp(ts, UTC).replace(second=0, microsecond=0)
|
||||
def _minute(ts: float) -> int:
|
||||
"""The epoch second the minute containing ``ts`` starts at.
|
||||
|
||||
An integer rather than a datetime: this runs once per event on the API's
|
||||
event loop, and building a tz-aware datetime to key a dict with cost more
|
||||
than everything the collector does with the event afterwards. The row
|
||||
still wants one, so `_write` builds it — once per minute per node rather
|
||||
than once per event.
|
||||
"""
|
||||
return int(ts // 60) * 60
|
||||
|
||||
|
||||
def _detail(event: dict[str, Any]) -> str:
|
||||
@@ -84,7 +92,7 @@ class MetricsCollector:
|
||||
def __init__(self, events: EventBus, flush_s: float = FLUSH_INTERVAL_S) -> None:
|
||||
self._events = events
|
||||
self._flush_s = flush_s
|
||||
self._buckets: dict[tuple[str, str, datetime], dict[str, float]] = {}
|
||||
self._buckets: dict[tuple[str, str, int], dict[str, float]] = {}
|
||||
self._runs: dict[str, dict[str, Any]] = {}
|
||||
self._pending: list[EngineEvent] = []
|
||||
# The traceback arrives one event before the failure it belongs to,
|
||||
@@ -302,7 +310,7 @@ class MetricsCollector:
|
||||
|
||||
def _hold(
|
||||
self,
|
||||
buckets: dict[tuple[str, str, datetime], dict[str, float]],
|
||||
buckets: dict[tuple[str, str, int], dict[str, float]],
|
||||
pending: list[EngineEvent],
|
||||
) -> None:
|
||||
"""Take a batch the database refused back, rather than losing it.
|
||||
@@ -332,7 +340,7 @@ class MetricsCollector:
|
||||
|
||||
def _write(
|
||||
self,
|
||||
buckets: dict[tuple[str, str, datetime], dict[str, float]],
|
||||
buckets: dict[tuple[str, str, int], dict[str, float]],
|
||||
pending: list[EngineEvent],
|
||||
runs: list[dict[str, Any]],
|
||||
prune: bool,
|
||||
@@ -340,7 +348,10 @@ class MetricsCollector:
|
||||
with Session(engine) as session:
|
||||
for (flow, node, minute), agg in buckets.items():
|
||||
statement = insert(MetricBucket).values(
|
||||
flow=flow, node=node, bucket=minute, **agg
|
||||
flow=flow,
|
||||
node=node,
|
||||
bucket=datetime.fromtimestamp(minute, UTC),
|
||||
**agg,
|
||||
)
|
||||
# The same minute is written several times, so the counters add
|
||||
# and the maxima take whichever is larger. Columns are read by
|
||||
|
||||
Reference in New Issue
Block a user