Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Cut the round trips a message costs the engine

Browse Source 
Measured with `make bench-engine` against a real Redis: 103.6 -> 164.4
messages a second on a five-node chain (p50 latency 2125 -> 1171 ms) and
34.8 -> 63.2 on a fan-out of twenty. Against the memory backend, which is
what a pip install runs on, 262 -> 626.

The two that bought most of it:

- `StateBackend.record` puts a published value, its timestamp, its series
  and its version counter in one round trip. They were four calls building
  four pipelines, and a value crossing an edge pays them twice. A released
  rate-limit hold rides along instead of a DEL per port.
- the readiness check reads a node's inputs and hands them to the node,
  rather than reading the triggering ones to count them and having the node
  read the same keys again a moment later.

`apply_outputs` was a second copy of `_record_outputs` and is now the same
code plus the event that distinguishes it.

The rest, each small:

- `_derive` builds a node-by-id map and a `consumes` index, so dispatching
  an item and publishing a value stop scanning every node in the
  installation.
- `read_all` is memoised against the store revision — it sits on the
  publish path, so a dashboard slider was reading and validating every
  flow file per value. Same mechanism `_wiring` already uses.
- the `message_value` source block is built once per node instead of per
  emission.
- both timer threads ask the queue to promote only when something is
  actually due, which takes an idle engine from ~4 Redis round trips a
  second to one.
- the shared httpx client is bounded (32 connections, one retry); its
  default pool is 100 with no per-host cap, so one slow endpoint could
  take it and every other sender node with it.
- the MQTT and delay nodes no longer log a line per message at INFO.

Robustness, in the same pass:

- `MemoryWorkQueue._done` was a set nothing ever removed from — one entry
  per non-idempotent node per item, for the life of the process, in the
  default configuration. Capped, the way the Redis side expires its
  markers.
- a saturated engine can claim from the due lane past the cascade limit.
  The capacity gate sits in front of the claim, so the due lane's priority
  — decided inside it — did not apply while every slot was held: a motor's
  stop was not behind the long nodes, it was unread. Only after a slot has
  genuinely failed to free for half a second, and briefly, so the backlog
  is not starved in turn.
- `reclaim_stale` dispatches through that same gate. It could return sixty
  entries and push in-flight far past the limit the gate exists to hold.
- a flow's nodes are stopped together rather than one after another. Each
  gets `NODE_STOP_TIMEOUT`, so a flow whose broker was unreachable took
  five seconds per node — long enough to outlast `REBUILD_WAIT` and 503
  the deploy.
- the worker pool and the HTTP client are closed on a thread, not on the
  event loop, and a run closes the state backend it built (on Redis, a
  client and a connection pool per run).
- the five background tasks say something when they die. Each catches
  exceptions inside its loop, so one raised anywhere else left the engine
  serving with no metrics, no alerts or no artifact sweep, silently.

`tests/flow/test_round_trips.py` counts the state operations one message
costs — four, where it was about eleven — because none of the above would
fail a behavioural test if it were undone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6hPWS6YEbT1P8LxhhFb2T
This commit is contained in:
stroblme
2026-08-29 19:58:39 +02:00
co-authored by Claude Opus 5
parent 8dbec0b579
commit da528340a9
13 changed files with 639 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/fluksio/flow/pipeline.py
+130 -67
View File
@@ -23,6 +23,7 @@ from collections import deque
from collections.abc import Callable, Iterator
from concurrent.futures import Future, ThreadPoolExecutor, wait
from contextlib import contextmanager
from functools import lru_cache
from typing import Any, Literal, Protocol
from pydantic import BaseModel, computed_field
@@ -98,6 +99,17 @@ def node_source(node: Node) -> ValueSource:
return ValueSource(kind="node", id=node.id, label=node.local_id)
@lru_cache(maxsize=2048)
def _node_source_dump(node_id: str, local_id: str) -> dict[str, Any]:
"""The `source` block of a `message_value`, built once per node.
It is a pure function of the two names, and it was a pydantic model
constructed and dumped per emission — the same cost the live path already
refuses to pay for `NodeOutcome`.
"""
return ValueSource(kind="node", id=node_id, label=local_id).model_dump()
class NodeOutcome(BaseModel):
"""How one node execution went.
@@ -184,19 +196,33 @@ def run_cache_key(fingerprint: str, inputs: dict[str, Any], flow: str = "") -> s
def _derive(
nodes: list[Node],
) -> tuple[dict[str, list[Node]], dict[Node, frozenset[Node]]]:
) -> tuple[
dict[str, list[Node]],
dict[Node, frozenset[Node]],
dict[str, list[Node]],
dict[str, Node],
]:
"""Work out the wiring the node list implies: producers, then dependencies.
Done over the whole list rather than one flow's share of it, because a
flow is not a subgraph — its nodes can read and write messages another
flow owns — so there is no deriving one flow's edges on their own.
The two lookups come out of the same walk: who consumes a message, and
which node an id names. Both were linear scans over every node in the
installation, on the per-message path.
"""
# A message may have several producers; every one of them is upstream
# of the nodes consuming it.
produces: dict[str, list[Node]] = {}
consumes: dict[str, list[Node]] = {}
by_id: dict[str, Node] = {}
for node in nodes:
for msg in node.provides:
produces.setdefault(msg, []).append(node)
for msg in node.requires:
consumes.setdefault(msg, []).append(node)
by_id[node.id] = node
# A node never depends on itself: reading a message it also provides is
# how state is carried between runs, not a cycle. An input marked
@@ -211,7 +237,7 @@ def _derive(
)
for node in nodes
}
return produces, dependencies
return produces, dependencies, consumes, by_id
class Pipeline:
@@ -224,6 +250,8 @@ class Pipeline:
"_max_workers",
"produces",
"dependencies",
"consumes",
"_by_id",
"_edges",
"_execution_order",
"_downstream_cache",
@@ -298,7 +326,12 @@ class Pipeline:
# than the default puts its message in here. Swapped, never mutated.
self.history_limits: dict[str, int] = {}
self.produces, self.dependencies = _derive(self._nodes)
(
self.produces,
self.dependencies,
self.consumes,
self._by_id,
) = _derive(self._nodes)
self._edges: dict[Node, set[Node]] | None = None
self._execution_order: list[Node] | None = None
@@ -376,7 +409,7 @@ class Pipeline:
if not placed:
spliced.extend(nodes)
produces, dependencies = _derive(spliced)
produces, dependencies, consumes, by_id = _derive(spliced)
# Assigned only once everything above has succeeded, and rebound
# rather than mutated: a cascade already walking the graph holds
@@ -385,6 +418,8 @@ class Pipeline:
self._nodes = spliced
self.produces = produces
self.dependencies = dependencies
self.consumes = consumes
self._by_id = by_id
self._edges = None
self._execution_order = None
self._downstream_cache = {}
@@ -423,7 +458,7 @@ class Pipeline:
self._state.increment_multi([self._version_key(name) for name in seeded])
def get_node_by_id(self, nid: str) -> Node | None:
return next((n for n in self._nodes if n.id == nid), None)
return self._by_id.get(nid)
def flow_nodes(self, flow: str) -> set[Node]:
return {n for n in self._nodes if n.flow == flow}
@@ -634,16 +669,23 @@ class Pipeline:
due_at = min(due_at or window_ends, window_ends)
return passed, held, due_at, stamps.get(flush_key)
def _throttled(self, node: Node, result: dict[str, Any]) -> dict[str, Any]:
def _throttled(
self, node: Node, result: dict[str, Any]
) -> tuple[dict[str, Any], list[str]]:
"""Hold back the outputs whose port is not due to publish yet.
The value is not lost: it is kept and published when the window ends,
so a producer that goes quiet still delivers its last reading rather
than leaving the consumer on the one before it. Nothing declaring an
interval means nothing to look up.
Returns what passed and the held keys the publish makes stale — the
caller drops them in the same round trip it writes the values with,
rather than one DEL per port here.
"""
now = time.time()
passed, held, due_at, pending = self._window_split(node, result, now)
stale: list[str] = []
if self._queue is not None:
# Without a queue there is no timer to let the value out later, so
@@ -652,13 +694,13 @@ class Pipeline:
spec = node.provides.get(name)
if spec is not None and spec.interval > 0:
# A fresh publish makes anything held for that port stale.
self._state.delete(self._held_key(name))
stale.append(self._held_key(name))
if held:
self._state.update(
{self._held_key(name): value for name, value in held.items()}
)
self._schedule_flush(node, due_at, now, pending)
return passed
return passed, stale
def _schedule_flush(
self, node: Node, at: float, now: float, pending: float | None
@@ -791,25 +833,34 @@ class Pipeline:
self._state.update({self._delivered_key(node.id, name): now for name in due})
return True
def _is_node_ready(self, node: Node, state: StateBackend) -> bool:
# A non-triggering input is read if it happens to be there; waiting for
# it would make an accumulator's first run impossible, since it is what
# the node is about to write.
waited_on = [msg for msg, spec in node.requires.items() if spec.trigger]
def _is_node_ready(
self, node: Node, state: StateBackend
) -> tuple[bool, dict[str, Any]]:
"""Whether the node may run, and the inputs the answer was read from.
The read covers every input rather than only the triggering ones, so
the node itself does not have to read the same keys again a moment
later — one MGET per node per message instead of two. A non-triggering
input is read if it happens to be there; waiting for it would make an
accumulator's first run impossible, since it is what the node is about
to write.
"""
# One MGET rather than an EXISTS per input under the global state lock.
if len(state.get_present(waited_on)) != len(waited_on):
return False
values = state.get_present(list(node.requires))
waited_on = [msg for msg, spec in node.requires.items() if spec.trigger]
if any(msg not in values for msg in waited_on):
return False, values
if not self._input_is_due(node):
return False
return False, values
if not node.synchronous:
return True
return True, values
is_ready, current_versions = self._check_synchronous_ready(node)
if not is_ready:
return False
return self._try_acquire_synchronous_execution(node, current_versions)
return False, values
return self._try_acquire_synchronous_execution(node, current_versions), values
# -------------------------------------------------------------------------
# Execution
@@ -952,15 +1003,25 @@ class Pipeline:
state: StateBackend,
entry_id: str = "",
overrides: dict[str, Any] | None = None,
inputs: dict[str, Any] | None = None,
) -> dict[str, Any] | None:
"""Run one node and record its outputs. Never raises."""
"""Run one node and record its outputs. Never raises.
``inputs`` is what the readiness check already read — the same keys, a
moment earlier. Reading them again here was the second of two MGETs
per node per message; running on the values that made the node ready
is also what ``overrides`` already assumes.
"""
started = time.perf_counter()
collected = logs.Collector()
try:
# One MGET. The lock this used to be read under bought nothing a
# single bulk read does not, and it was the engine's one global
# mutex — every node of every cascade queued behind it.
inputs = state.get_present(list(node.requires))
if inputs is None:
inputs = state.get_present(list(node.requires))
else:
inputs = dict(inputs)
if overrides:
# The value this wave is delivering wins over whatever state
# holds by now. Not written back: the newest value is still the
@@ -999,11 +1060,12 @@ class Pipeline:
except Exception as exc:
logger.warning("Could not mark '%s' done: %s", node.id, exc)
stale: list[str] = []
if result:
result = self._throttled(node, result)
result, stale = self._throttled(node, result)
if result:
self._record_outputs(node, result, state)
self._record_outputs(node, result, state, stale)
duration_ms = round((time.perf_counter() - started) * 1000, 2)
self._publish(
@@ -1058,24 +1120,34 @@ class Pipeline:
return None
def _record_outputs(
self, node: Node, outputs: dict[str, Any], state: StateBackend
) -> None:
self,
node: Node,
outputs: dict[str, Any],
state: StateBackend,
drop: list[str] | None = None,
) -> float:
"""Put a node's outputs where everything downstream of them looks.
State, the timestamp beside it, the series, the version counter and the
event the canvas draws from. Shared by a node returning and a node
emitting mid-execution, because those are the same act: a value the
node produced, leaving through a port it declared.
The four writes go in one round trip: a pipeline is a transaction, so
they land together exactly as they did when they were four calls.
Returns the timestamp they were recorded under, which the caller
stamps its own events with so they agree.
"""
ts = time.time()
# Value and timestamp in one write, which is what the lock around two
# of them was for — a pipeline is a transaction, so they still land
# together and nobody waits on a mutex to do it.
state.update({**outputs, **{self._timestamp_key(name): ts for name in outputs}})
# Append-only, so it needs no lock of its own.
state.append_history(outputs, ts, self.history_limits)
self._increment_message_versions(outputs)
origin = node_source(node).model_dump()
state.record(
outputs,
{self._timestamp_key(name): ts for name in outputs},
[self._version_key(name) for name in outputs],
ts,
self.history_limits,
drop,
)
origin = _node_source_dump(node.id, node.local_id)
for name, value in outputs.items():
self._publish(
{
@@ -1087,6 +1159,7 @@ class Pipeline:
"source": origin,
}
)
return ts
def publish_emission(self, node: Node, outputs: dict[str, Any]) -> None:
"""Publish what a node produced while it is still running.
@@ -1103,10 +1176,10 @@ class Pipeline:
with no meaning.
"""
self._observe_emission(node, outputs)
passed = self._throttled(node, outputs)
passed, stale = self._throttled(node, outputs)
if not passed:
return
self._record_outputs(node, passed, self._state)
self._record_outputs(node, passed, self._state, stale)
if self._queue is not None:
# Journalled carrying the emitted values, as an ``emission`` item:
# the executor hands them to the nodes reading them instead of
@@ -1277,11 +1350,15 @@ class Pipeline:
complete(n)
progressed = True
continue
if check_ready and not self._is_node_ready(n, state):
if n.synchronous:
# Not ready now; a later trigger may make it ready.
skipped.add(n)
continue
ready_inputs: dict[str, Any] | None = None
if check_ready:
ready, ready_inputs = self._is_node_ready(n, state)
if not ready:
if n.synchronous:
# Not ready now; a later trigger may make it
# ready.
skipped.add(n)
continue
if replay and entry_id and self._already_done(entry_id, n):
# Its side effect happened on an earlier delivery; its
# outputs are still in state, so downstream carries on.
@@ -1290,7 +1367,7 @@ class Pipeline:
continue
submitted.add(n)
node_futures[n] = executor.submit(
self._execute_node, n, state, entry_id, overrides
self._execute_node, n, state, entry_id, overrides, ready_inputs
)
def drain(executor: ThreadPoolExecutor) -> None:
@@ -1345,31 +1422,16 @@ class Pipeline:
Returns the message names that actually reached state — post rate
limiting — which is what the cascade behind it has to walk from.
"""
stale: list[str] = []
if outputs:
# This is where a chatty subscriber gets thinned out, so a port set
# to publish every 60s does so whatever the broker sends.
outputs = self._throttled(node, outputs)
outputs, stale = self._throttled(node, outputs)
if not outputs:
return set()
state = self._state
ts = time.time()
state.update({**outputs, **{self._timestamp_key(name): ts for name in outputs}})
state.append_history(outputs, ts, self.history_limits)
self._increment_message_versions(outputs)
origin = node_source(node).model_dump()
for name, value in outputs.items():
self._publish(
{
"type": "message_value",
"flow": flow_of(name),
"name": name,
"value": value,
"ts": ts,
"source": origin,
}
)
ts = self._record_outputs(node, outputs, self._state, stale)
# An injecting node — an MQTT subscriber, a webhook — publishes
# without going through the executor, but it did emit.
self._publish(
@@ -1559,11 +1621,13 @@ class Pipeline:
origin = source or ValueSource(kind="api", label="API")
ts = time.time()
self._state.update(
{**values, **{self._timestamp_key(name): ts for name in values}}
self._state.record(
values,
{self._timestamp_key(name): ts for name in values},
[self._version_key(name) for name in values],
ts,
self.history_limits,
)
self._state.append_history(values, ts, self.history_limits)
self._increment_message_versions(values)
source_dump = origin.model_dump()
for name, value in values.items():
self._publish(
@@ -1579,10 +1643,9 @@ class Pipeline:
targets: set[Node] = set()
for name in values:
for consumer in self._nodes:
if name in consumer.requires:
targets.add(consumer)
targets.update(self._get_downstream(consumer))
for consumer in self.consumes.get(name, ()):
targets.add(consumer)
targets.update(self._get_downstream(consumer))
if targets:
self._execute_parallel(targets, self._state, check_ready=True)