From f4cdf71795ea07690b53adde3a350edc8a0dc2d0 Mon Sep 17 00:00:00 2001 From: stroblme Date: Sun, 6 Sep 2026 17:16:01 +0200 Subject: [PATCH] Bound the work-queue durability claim to the appendfsync window Redis runs with appendfsync everysec, so an unclean stop loses up to a second of journaled entries. Name that where the docstrings promised a crash costs nothing, and state the setting and its trade-off on the configuration page. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01CL9zvnnvcp1mvA8o7impxk --- backend/fluksio/flow/pipeline.py | 3 ++- backend/fluksio/flow/queue.py | 4 ++++ docs/reference/configuration.md | 8 +++++++- 3 files changed, 13 insertions(+), 2 deletions(-) diff --git a/backend/fluksio/flow/pipeline.py b/backend/fluksio/flow/pipeline.py index 4746b1a..868599f 100644 --- a/backend/fluksio/flow/pipeline.py +++ b/backend/fluksio/flow/pipeline.py @@ -1522,7 +1522,8 @@ class Pipeline: With a work queue attached the event is journaled and the caller returns immediately — that is the path every external trigger takes, so - a crash mid-cascade loses nothing. Interactive callers (a manual run, a + a crash mid-cascade costs at most the last second of journaled work + rather than the whole wave. Interactive callers (a manual run, a draft preview) pass ``durable=False`` and get the old synchronous behaviour, because they are waiting for the result. diff --git a/backend/fluksio/flow/queue.py b/backend/fluksio/flow/queue.py index 03d5ec6..7c0ef72 100644 --- a/backend/fluksio/flow/queue.py +++ b/backend/fluksio/flow/queue.py @@ -5,6 +5,10 @@ poll — becomes a work item before anything runs. The item is journaled first and acknowledged only once the wave it started has quiesced, so an engine that dies mid-cascade picks the work up again on the way back rather than losing it. +Redis journals with ``appendfsync everysec``, so the bound is the last second: +an unclean stop of Redis or of its host loses up to a second of entries. An +engine that dies while Redis lives loses none of them. + Two implementations: Redis Streams, which is what makes the above true, and an in-memory one for tests and for running without Redis, where "durable" degrades honestly to "not". diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 3361680..2193570 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -54,7 +54,13 @@ An instance that already has a managed venv keeps it on upgrade under | `REDIS_PORT` | `6379` | | Flow state is the last value of every message, node memory, and the run queue. -Redis here is persistence, not a cache. Run it with append-only persistence on. +Redis here is persistence, not a cache. Run it with append-only persistence on +(`--appendonly yes`, which the compose stack sets). + +That writes with Redis's default `appendfsync everysec`, so an unclean stop of +Redis or of its host loses up to a second of queued work. `appendfsync always` +closes the window and cuts engine throughput to about a third, so the default +stands. ## Identity and access