stroblmeandClaude Opus 5
da528340a9
Measured with `make bench-engine` against a real Redis: 103.6 -> 164.4 messages a second on a five-node chain (p50 latency 2125 -> 1171 ms) and 34.8 -> 63.2 on a fan-out of twenty. Against the memory backend, which is what a pip install runs on, 262 -> 626. The two that bought most of it: - `StateBackend.record` puts a published value, its timestamp, its series and its version counter in one round trip. They were four calls building four pipelines, and a value crossing an edge pays them twice. A released rate-limit hold rides along instead of a DEL per port. - the readiness check reads a node's inputs and hands them to the node, rather than reading the triggering ones to count them and having the node read the same keys again a moment later. `apply_outputs` was a second copy of `_record_outputs` and is now the same code plus the event that distinguishes it. The rest, each small: - `_derive` builds a node-by-id map and a `consumes` index, so dispatching an item and publishing a value stop scanning every node in the installation. - `read_all` is memoised against the store revision — it sits on the publish path, so a dashboard slider was reading and validating every flow file per value. Same mechanism `_wiring` already uses. - the `message_value` source block is built once per node instead of per emission. - both timer threads ask the queue to promote only when something is actually due, which takes an idle engine from ~4 Redis round trips a second to one. - the shared httpx client is bounded (32 connections, one retry); its default pool is 100 with no per-host cap, so one slow endpoint could take it and every other sender node with it. - the MQTT and delay nodes no longer log a line per message at INFO. Robustness, in the same pass: - `MemoryWorkQueue._done` was a set nothing ever removed from — one entry per non-idempotent node per item, for the life of the process, in the default configuration. Capped, the way the Redis side expires its markers. - a saturated engine can claim from the due lane past the cascade limit. The capacity gate sits in front of the claim, so the due lane's priority — decided inside it — did not apply while every slot was held: a motor's stop was not behind the long nodes, it was unread. Only after a slot has genuinely failed to free for half a second, and briefly, so the backlog is not starved in turn. - `reclaim_stale` dispatches through that same gate. It could return sixty entries and push in-flight far past the limit the gate exists to hold. - a flow's nodes are stopped together rather than one after another. Each gets `NODE_STOP_TIMEOUT`, so a flow whose broker was unreachable took five seconds per node — long enough to outlast `REBUILD_WAIT` and 503 the deploy. - the worker pool and the HTTP client are closed on a thread, not on the event loop, and a run closes the state backend it built (on Redis, a client and a connection pool per run). - the five background tasks say something when they die. Each catches exceptions inside its loop, so one raised anywhere else left the engine serving with no metrics, no alerts or no artifact sweep, silently. `tests/flow/test_round_trips.py` counts the state operations one message costs — four, where it was about eleven — because none of the above would fail a behavioural test if it were undone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M6hPWS6YEbT1P8LxhhFb2T
127 lines
4.0 KiB
Python
127 lines
4.0 KiB
Python
"""What one message costs the state backend, in operations.
|
|
|
|
The engine's time goes on round trips, not on Python: a profile against a
|
|
real Redis put its own code at 8.5% of self-time and Redis I/O at 61%. So the
|
|
thing worth a regression test is the *count* — a change that quietly turns one
|
|
pipelined write back into four would not fail any behavioural test and would
|
|
cost a fifth of the throughput.
|
|
|
|
Counted against the memory backend, because the number is a property of the
|
|
call sites rather than of the transport.
|
|
"""
|
|
|
|
from collections import Counter
|
|
from typing import Any
|
|
|
|
from fluksio.flow.messages import DType, MessageSpec
|
|
from fluksio.flow.nodes import Node
|
|
from fluksio.flow.pipeline import Pipeline
|
|
from fluksio.flow.state import MemoryState
|
|
|
|
COUNTED = (
|
|
"get",
|
|
"set",
|
|
"get_multi",
|
|
"get_present",
|
|
"update",
|
|
"record",
|
|
"append_history",
|
|
"increment_multi",
|
|
"delete",
|
|
)
|
|
|
|
|
|
class CountingState(MemoryState):
|
|
"""A state backend that remembers how often it was asked for something."""
|
|
|
|
# MemoryState uses __slots__; this one needs an attribute of its own.
|
|
__slots__ = ("calls",)
|
|
|
|
def __init__(self) -> None:
|
|
super().__init__()
|
|
self.calls: Counter[str] = Counter()
|
|
|
|
def __getattribute__(self, name: str) -> Any:
|
|
if name in COUNTED:
|
|
object.__getattribute__(self, "calls")[name] += 1
|
|
return object.__getattribute__(self, name)
|
|
|
|
|
|
def spec(name: str) -> MessageSpec:
|
|
return MessageSpec(name=name, dtype=DType.FLOAT)
|
|
|
|
|
|
def make_node(node_id: str, flow: str, f, requires=(), provides=()) -> Node:
|
|
node = Node(f=f, requires=list(requires), provides=list(provides), name=node_id)
|
|
node.assign_flow(flow, node_id)
|
|
return node
|
|
|
|
|
|
def build() -> tuple[Pipeline, Node, CountingState]:
|
|
source = make_node(
|
|
"source", "chain", lambda params: {"a": 1.0}, provides=[spec("a")]
|
|
)
|
|
relay = make_node(
|
|
"relay",
|
|
"chain",
|
|
lambda a, params: {"b": a + 1},
|
|
requires=[spec("a")],
|
|
provides=[spec("b")],
|
|
)
|
|
sink = make_node(
|
|
"sink", "chain", lambda b, params: None, requires=[spec("b")]
|
|
)
|
|
state = CountingState()
|
|
pipeline = Pipeline([source, relay, sink], state=state)
|
|
return pipeline, source, state
|
|
|
|
|
|
def test_recording_a_value_is_one_write() -> None:
|
|
"""Value, timestamp, history and version counter go in one round trip.
|
|
|
|
They were four calls — `update`, `append_history`, `increment_multi` and a
|
|
`delete` per rate-limited port — and a value crossing an edge pays them
|
|
twice.
|
|
"""
|
|
pipeline, source, state = build()
|
|
state.calls.clear()
|
|
|
|
pipeline.apply_outputs(source, {"chain.a": 1.0})
|
|
|
|
assert state.calls["record"] == 1
|
|
for method in ("update", "append_history", "increment_multi"):
|
|
assert state.calls[method] == 0, f"{method} should be folded into record()"
|
|
|
|
|
|
def test_a_node_reads_its_inputs_once() -> None:
|
|
"""Readiness and execution share one read of the same keys.
|
|
|
|
The readiness check used to read the triggering inputs and throw the
|
|
values away, and the node then read the same keys again to run on them.
|
|
"""
|
|
pipeline, source, state = build()
|
|
state.calls.clear()
|
|
|
|
changed = pipeline.apply_outputs(source, {"chain.a": 1.0})
|
|
pipeline.run_downstream(source, changed=changed)
|
|
|
|
# Two nodes ran downstream (relay, sink); one bulk read each.
|
|
assert state.calls["get_present"] == 2
|
|
|
|
|
|
def test_a_hop_costs_a_bounded_number_of_state_operations() -> None:
|
|
"""The whole of one value crossing one edge, counted.
|
|
|
|
A ceiling rather than an exact number, so an unrelated change does not
|
|
fail it — but low enough that reintroducing a per-value read or a
|
|
split-up write does.
|
|
"""
|
|
pipeline, source, state = build()
|
|
state.calls.clear()
|
|
|
|
changed = pipeline.apply_outputs(source, {"chain.a": 1.0})
|
|
pipeline.run_downstream(source, changed=changed)
|
|
|
|
total = sum(state.calls[name] for name in COUNTED)
|
|
assert total <= 8, f"state operations per hop grew: {dict(state.calls)}"
|