Journal work before running it, so a crash stops losing messages

Execution was fire-and-forget: an MQTT message or webhook ran a cascade
on a ThreadPoolExecutor built for that one wave, and an engine that died
halfway through simply lost whatever was in flight. Concurrent triggers
each built their own pool, so load meant unbounded threads.

Every external trigger is now journaled to a Redis Streams queue before
anything runs, and acknowledged only once its cascade finishes. A
consumer thread drives cascades on one long-lived pool while node bodies
run on another, so a cascade cannot starve the nodes it is waiting for.
A reaper reclaims what a dead consumer never acknowledged — verified end
to end: work journaled while the engine was stopped runs on restart, and
work abandoned mid-cascade comes back as a second delivery.

At-least-once needs a guard, so nodes that reach outside are marked
non-idempotent and skipped on a redelivery they already completed.
Without Redis the queue degrades to an in-memory one that does not
pretend to be durable, and interactive callers still run inline.

Also fixes two things this turned up: a delay node was sleeping on a
worker thread, where a handful of them could occupy the whole pool, and
webhooks 404'd whenever MCP was enabled because the app mounted at /
answered first for every path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
This commit is contained in:
root
2026-08-16 07:57:07 +02:00
co-authored by Claude Fable 5
parent 2cc25970e3
commit 04329149b3
17 changed files with 1271 additions and 66 deletions
+6
View File
@@ -80,6 +80,12 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend M
restarted with growing delay when it dies, and a flow that spends its failure
budget is quarantined and surfaced rather than left crash-looping. The loops
themselves no longer carry private retry logic
- [x] Durable work queue: every external trigger is journaled to Redis Streams before
anything runs and acknowledged once its cascade finishes, so an engine that dies
mid-cascade picks the work up again instead of losing it. A reaper reclaims what
a dead consumer never acknowledged; nodes that reach outside are skipped on a
redelivery they already ran. Long-lived worker pools replace the per-wave
executors, and a delay now waits in the queue rather than on a worker thread
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
deployment on failure
- [ ] User management scoped per flow and per data set