Files
app/backend/fluksio/flow/nodes/delay.py
T
stroblmeandClaude Opus 5 da528340a9 Cut the round trips a message costs the engine
Measured with `make bench-engine` against a real Redis: 103.6 -> 164.4
messages a second on a five-node chain (p50 latency 2125 -> 1171 ms) and
34.8 -> 63.2 on a fan-out of twenty. Against the memory backend, which is
what a pip install runs on, 262 -> 626.

The two that bought most of it:

- `StateBackend.record` puts a published value, its timestamp, its series
  and its version counter in one round trip. They were four calls building
  four pipelines, and a value crossing an edge pays them twice. A released
  rate-limit hold rides along instead of a DEL per port.
- the readiness check reads a node's inputs and hands them to the node,
  rather than reading the triggering ones to count them and having the node
  read the same keys again a moment later.

`apply_outputs` was a second copy of `_record_outputs` and is now the same
code plus the event that distinguishes it.

The rest, each small:

- `_derive` builds a node-by-id map and a `consumes` index, so dispatching
  an item and publishing a value stop scanning every node in the
  installation.
- `read_all` is memoised against the store revision — it sits on the
  publish path, so a dashboard slider was reading and validating every
  flow file per value. Same mechanism `_wiring` already uses.
- the `message_value` source block is built once per node instead of per
  emission.
- both timer threads ask the queue to promote only when something is
  actually due, which takes an idle engine from ~4 Redis round trips a
  second to one.
- the shared httpx client is bounded (32 connections, one retry); its
  default pool is 100 with no per-host cap, so one slow endpoint could
  take it and every other sender node with it.
- the MQTT and delay nodes no longer log a line per message at INFO.

Robustness, in the same pass:

- `MemoryWorkQueue._done` was a set nothing ever removed from — one entry
  per non-idempotent node per item, for the life of the process, in the
  default configuration. Capped, the way the Redis side expires its
  markers.
- a saturated engine can claim from the due lane past the cascade limit.
  The capacity gate sits in front of the claim, so the due lane's priority
  — decided inside it — did not apply while every slot was held: a motor's
  stop was not behind the long nodes, it was unread. Only after a slot has
  genuinely failed to free for half a second, and briefly, so the backlog
  is not starved in turn.
- `reclaim_stale` dispatches through that same gate. It could return sixty
  entries and push in-flight far past the limit the gate exists to hold.
- a flow's nodes are stopped together rather than one after another. Each
  gets `NODE_STOP_TIMEOUT`, so a flow whose broker was unreachable took
  five seconds per node — long enough to outlast `REBUILD_WAIT` and 503
  the deploy.
- the worker pool and the HTTP client are closed on a thread, not on the
  event loop, and a run closes the state backend it built (on Redis, a
  client and a connection pool per run).
- the five background tasks say something when they die. Each catches
  exceptions inside its loop, so one raised anywhere else left the engine
  serving with no metrics, no alerts or no artifact sweep, silently.

`tests/flow/test_round_trips.py` counts the state operations one message
costs — four, where it was about eleven — because none of the above would
fail a behavioural test if it were undone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6hPWS6YEbT1P8LxhhFb2T
2026-08-29 19:58:39 +02:00

332 lines
11 KiB
Python

"""Timing nodes: fixed delay, rate limit, and the cron schedule."""
from __future__ import annotations
import asyncio
import logging
import time
from typing import TYPE_CHECKING, Any
from pydantic import BaseModel, ConfigDict
from fluksio.flow.nodes.base import Node
if TYPE_CHECKING:
from fastapi import FastAPI
logger = logging.getLogger(__name__)
class DelayNode(Node):
"""
A node that adds delay, rate-limiting, and/or cron-scheduled emissions.
This node can:
- Add a fixed delay before forwarding messages
- Rate-limit messages to a minimum interval between forwards
- Schedule emissions using crontab syntax (like a cron job)
The node passes through all input values to outputs with matching names.
If ``requires`` and ``provides`` have the same message names, the values are
forwarded directly. Otherwise, you can specify a ``mapping`` in params.
**Cron functionality:**
- Without ``requires``: Emits current timestamp at each cron tick
- With ``requires``: Emits last received input at each cron tick
**Execution order:** rate check → delay → forward
:param params: Parameters dict containing:
- ``delay`` (int): Fixed delay in seconds before forwarding (default: 0)
- ``interval`` (int): Minimum interval in seconds between forwards (default: 0)
- ``mapping`` (dict): Optional mapping from input names to output names
- ``cron`` (str): Crontab expression for scheduled emissions (optional).
Standard 5-field format: ``minute hour day-of-month month day-of-week``
:type params: dict
:param kwargs: Additional arguments passed to Node (requires, provides, name)
:example:
Simple passthrough with delay:
>>> delay_node = DelayNode(
... requires=[MessageSpec(name="temperature", dtype=DType.FLOAT)],
... provides=[MessageSpec(name="temperature", dtype=DType.FLOAT)],
... params={"delay": 1},
... )
Rate-limited forwarding:
>>> throttle_node = DelayNode(
... requires=[MessageSpec(name="sensor_data", dtype=DType.FLOAT)],
... provides=[MessageSpec(name="sensor_data", dtype=DType.FLOAT)],
... params={"interval": 5},
... )
Cron: emit timestamp every day at 08:30:
>>> alarm_node = DelayNode(
... provides=[MessageSpec(name="timestamp", dtype=DType.FLOAT)],
... params={"cron": "30 8 * * *"},
... )
Cron: emit stored input every 5 minutes:
>>> periodic_node = DelayNode(
... requires=[MessageSpec(name="value", dtype=DType.FLOAT)],
... provides=[MessageSpec(name="value", dtype=DType.FLOAT)],
... params={"cron": "*/5 * * * *"},
... )
Cron: emit on weekdays at 23:59:
>>> weekday_node = DelayNode(
... requires=[MessageSpec(name="daily_summary", dtype=DType.FLOAT)],
... provides=[MessageSpec(name="daily_summary", dtype=DType.FLOAT)],
... params={"cron": "59 23 * * 1-5"},
... )
"""
__slots__ = (
"delay",
"interval",
"ts",
"mapping",
"cron_expr",
"last_input",
"_cron_task",
"_stop_cron",
)
class Params(BaseModel):
model_config = ConfigDict(extra="allow")
# Seconds, fractional: a shutter's run-time is not a whole number.
delay: float = 0
interval: float = 0
# Input port → output port; ports are paired in order when empty.
mapping: dict[str, str] = {}
cron: str | None = None
def __init__(self, params: dict[str, Any] | None = None, **kwargs: Any) -> None:
cfg = self.Params.model_validate(params or {})
self.delay = cfg.delay
self.interval = cfg.interval
self.mapping = cfg.mapping
self.cron_expr = cfg.cron
self.ts = 0.0
self.last_input: dict[str, Any] = {}
self._cron_task: asyncio.Task[None] | None = None
self._stop_cron: asyncio.Event | None = None
super().__init__(f=self._f, params=params, **kwargs)
def _port_pairs(self) -> list[tuple[str, str]]:
"""Input port → output port pairs this node forwards along."""
if self.mapping:
return list(self.mapping.items())
return [
(i.port, o.port)
for i, o in zip(self.input_ports, self.output_ports, strict=False)
]
def _f(self, params: dict[str, Any], **kwargs: Any) -> dict[str, Any] | None:
"""Forward messages with optional delay, rate-limiting, and alarm."""
logger.debug("[%s] Received %s", self.name, kwargs)
# Store last input for cron use
if kwargs:
self.last_input = dict(kwargs)
# Apply rate limiting first
if self.interval > 0:
ts = time.time()
if ts <= self.ts + self.interval:
logger.debug("[%s] Stashing %s", self.name, kwargs)
return None
self.ts = ts
if not kwargs:
return None
output = {
out_port: kwargs[in_port]
for in_port, out_port in self._port_pairs()
if in_port in kwargs
}
if self.delay > 0:
# Hand the wait to the queue rather than sit on a worker thread:
# a handful of delay nodes would otherwise occupy the whole pool
# and the engine would stop dead until they woke up.
if self._pipeline is not None and self._pipeline.defer(
self, self._to_messages(output) or {}, self.delay
):
logger.debug(
"[%s] Sending %s in %ss", self.name, output, self.delay
)
return None
time.sleep(self.delay)
logger.debug("[%s] Sending %s", self.name, output)
return output
# -----------------------------------------------------------------
# Cron scheduler
# -----------------------------------------------------------------
async def start(self, app: FastAPI | None = None) -> None:
"""Only a node with a schedule has anything to run on its own."""
if self.cron_expr:
await self.start_cron()
async def stop(self, app: FastAPI | None = None) -> None:
if self.cron_expr:
await self.stop_cron()
async def start_cron(self) -> None:
"""
Start the cron scheduler.
Requires the ``croniter`` package. The scheduler runs in the background
and triggers emissions according to the ``cron`` expression in params.
:raises ValueError: If no ``cron`` expression is configured.
"""
if not self.cron_expr:
logger.warning("No cron expression configured for node '%s'", self.name)
return
if self._cron_task is not None:
logger.info("Cron already running for node '%s'", self.name)
return
self._stop_cron = asyncio.Event()
self._cron_task = self._run_supervised("cron", self._cron_loop)
logger.info("Started cron for node '%s': %s", self.name, self.cron_expr)
async def stop_cron(self) -> None:
"""Stop the cron scheduler."""
if self._stop_cron is None:
return
self._stop_cron.set()
if self._cron_task is not None:
await self._cancel_task(self._cron_task)
self._cron_task = None
self._stop_cron = None
logger.info("Stopped cron for node '%s'", self.name)
async def _cron_loop(self) -> None:
"""Background loop that sleeps until the next cron tick and triggers."""
from datetime import datetime
try:
from croniter import croniter # type: ignore[import-untyped]
except ImportError:
logger.error(
"croniter package is required for cron scheduling. "
"Install it with: pip install croniter"
)
return
if not croniter.is_valid(self.cron_expr):
logger.error(
"Invalid cron expression '%s' for node '%s'",
self.cron_expr,
self.name,
)
return
cron = croniter(self.cron_expr, datetime.now())
# Walrus so the body can use the event without re-reading a field
# stop_cron() may have cleared in the meantime.
while (stop := self._stop_cron) is not None and not stop.is_set():
try:
# Compute seconds until next tick
next_dt = cron.get_next(datetime)
now = datetime.now()
wait_seconds = max(0, (next_dt - now).total_seconds())
logger.info(
"Node '%s' cron: next tick at %s (in %.1fs)",
self.name,
next_dt.isoformat(),
wait_seconds,
)
# Sleep until next tick (wake up on stop signal)
try:
await asyncio.wait_for(stop.wait(), timeout=wait_seconds)
# If we get here, stop was requested
break
except TimeoutError:
# Timeout means it's time to fire
pass
logger.info(
"Cron triggered for node '%s' (%s)",
self.name,
self.cron_expr,
)
await self._trigger_cron()
except asyncio.CancelledError:
raise
except Exception:
# One bad tick should not cost the schedule; the supervisor
# picks it up from here, backoff included.
logger.error(
"Error in cron loop for node '%s'", self.name, exc_info=True
)
raise
async def _trigger_cron(self) -> None:
"""Emit data into the pipeline on a cron tick."""
if self._pipeline is None:
logger.warning(
"Node '%s' not bound to pipeline, cannot trigger cron",
self.name,
)
return
try:
output = {}
if not self.input_ports:
# Nothing to forward — emit the current time on every output.
output = {spec.port: time.time() for spec in self.output_ports}
logger.info("Cron emission for node '%s': timestamp", self.name)
elif self.last_input:
output = {
out_port: self.last_input[in_port]
for in_port, out_port in self._port_pairs()
if in_port in self.last_input
}
logger.info(
"Cron emission for node '%s': stored input %s",
self.name,
output,
)
else:
logger.warning(
"Cron for node '%s' triggered but no input stored yet",
self.name,
)
return
if output:
await asyncio.to_thread(
self._pipeline.trigger, self, self._to_messages(output)
)
except Exception as e:
logger.error(
"Error triggering cron for node '%s': %s",
self.name,
e,
exc_info=True,
)