Journal work before running it, so a crash stops losing messages

Execution was fire-and-forget: an MQTT message or webhook ran a cascade
on a ThreadPoolExecutor built for that one wave, and an engine that died
halfway through simply lost whatever was in flight. Concurrent triggers
each built their own pool, so load meant unbounded threads.

Every external trigger is now journaled to a Redis Streams queue before
anything runs, and acknowledged only once its cascade finishes. A
consumer thread drives cascades on one long-lived pool while node bodies
run on another, so a cascade cannot starve the nodes it is waiting for.
A reaper reclaims what a dead consumer never acknowledged — verified end
to end: work journaled while the engine was stopped runs on restart, and
work abandoned mid-cascade comes back as a second delivery.

At-least-once needs a guard, so nodes that reach outside are marked
non-idempotent and skipped on a redelivery they already completed.
Without Redis the queue degrades to an in-memory one that does not
pretend to be durable, and interactive callers still run inline.

Also fixes two things this turned up: a delay node was sleeping on a
worker thread, where a handful of them could occupy the whole pool, and
webhooks 404'd whenever MCP was enabled because the app mounted at /
answered first for every path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
This commit is contained in:
root
2026-08-16 07:57:07 +02:00
co-authored by Claude Fable 5
parent 2cc25970e3
commit 04329149b3
17 changed files with 1271 additions and 66 deletions
+3 -6
View File
@@ -11,10 +11,9 @@ Deferring because out of scope is fine, but don't mention deferring than.
- BUG/UI: enlarge the icon in the sidebar slightly
- BUG/UI: clicking outside the panel does not discard the flow edit panel
- BUG/UI: the graph showed in the node edit panel should also be shown for a specific edge inside the pop-up panel when clicking the edge
- FEAT/FLOW: single-stepping a paused flow. Pause and resume are in; a step button needs the
scheduler to keep its per-run progress between calls, which the one-shot executor does not —
without that it re-runs the first ready node instead of advancing. Needs a persistent
per-flow work queue that a step pops from and resume drains.
- FEAT/FLOW: single-stepping a paused flow. The work queue it needed now exists: pausing
parks claimed items per flow and resuming drains them, so a step button is a matter of
popping one parked item instead of all of them.
- FEAT/UI: interrupting a node that is already running. Pause holds nodes that have not been
submitted yet; one already executing runs to completion.
- BUG/UI: the enlarged panel (for code editing) should still maintain its floating style
@@ -46,8 +45,6 @@ Deferring because out of scope is fine, but don't mention deferring than.
- CHORE/UI: `make test-backend` cannot reach Postgres while the integrated stack is up —
`compose.local.yml` does `db: ports: !reset []`. Run it against the container's address, or
move the suite inside the compose network.
- BUG/FLOW: deleting a flow leaves its state in Redis — the value, timestamp, version and
history keys under `pipeline:{flow}.*` all survive. Clear the namespace on delete.
- CHORE/UI: the Playwright specs run against the development stack and leave their users and
flows behind, which is why the flowbar filled with `test_flow_*`. Give them their own data
or clean up after themselves, as `pytest` now does.