Journal work before running it, so a crash stops losing messages
Execution was fire-and-forget: an MQTT message or webhook ran a cascade on a ThreadPoolExecutor built for that one wave, and an engine that died halfway through simply lost whatever was in flight. Concurrent triggers each built their own pool, so load meant unbounded threads. Every external trigger is now journaled to a Redis Streams queue before anything runs, and acknowledged only once its cascade finishes. A consumer thread drives cascades on one long-lived pool while node bodies run on another, so a cascade cannot starve the nodes it is waiting for. A reaper reclaims what a dead consumer never acknowledged — verified end to end: work journaled while the engine was stopped runs on restart, and work abandoned mid-cascade comes back as a second delivery. At-least-once needs a guard, so nodes that reach outside are marked non-idempotent and skipped on a redelivery they already completed. Without Redis the queue degrades to an in-memory one that does not pretend to be durable, and interactive callers still run inline. Also fixes two things this turned up: a delay node was sleeping on a worker thread, where a handful of them could occupy the whole pool, and webhooks 404'd whenever MCP was enabled because the app mounted at / answered first for every path.
This commit is contained in:
@@ -15,8 +15,10 @@ from app.core.config import settings
|
||||
from app.flow import logs
|
||||
from app.flow.controller import FlowController
|
||||
from app.flow.events import event_bus
|
||||
from app.flow.executor import ExecutionService
|
||||
from app.flow.nodes.http import close_shared_client
|
||||
from app.flow.plugins import load_plugins
|
||||
from app.flow.queue import MemoryWorkQueue, RedisWorkQueue, WorkQueue
|
||||
from app.flow.secrets import init_secrets
|
||||
from app.flow.state import MemoryState, RedisState, StateBackend
|
||||
from app.flow.store import FlowStore
|
||||
@@ -37,6 +39,13 @@ def _state_backend() -> StateBackend:
|
||||
return MemoryState()
|
||||
|
||||
|
||||
def _work_queue() -> WorkQueue:
|
||||
"""Redis makes queued work survive the process; memory does not pretend to."""
|
||||
if settings.REDIS_HOST:
|
||||
return RedisWorkQueue(host=settings.REDIS_HOST, port=settings.REDIS_PORT)
|
||||
return MemoryWorkQueue()
|
||||
|
||||
|
||||
def _mcp_sessions() -> AbstractAsyncContextManager[None]:
|
||||
"""The MCP session manager's run scope, or nothing when MCP is off."""
|
||||
if not settings.MCP_ENABLED:
|
||||
@@ -56,12 +65,18 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
# Connectors register their node types before any flow is built with them.
|
||||
load_plugins()
|
||||
|
||||
execution = ExecutionService(
|
||||
queue=_work_queue(),
|
||||
max_workers=settings.FLOW_MAX_WORKERS,
|
||||
events=event_bus,
|
||||
)
|
||||
controller = FlowController(
|
||||
store=FlowStore(settings.FLOWS_DIR),
|
||||
state=_state_backend(),
|
||||
events=event_bus,
|
||||
max_workers=settings.FLOW_MAX_WORKERS,
|
||||
fastapi_app=app,
|
||||
execution=execution,
|
||||
)
|
||||
app.state.flow_controller = controller
|
||||
watchdog = LoopWatchdog(event_bus)
|
||||
|
||||
Reference in New Issue
Block a user