diff --git a/backend/fluksio/flow/pipeline.py b/backend/fluksio/flow/pipeline.py index 4746b1a..868599f 100644 --- a/backend/fluksio/flow/pipeline.py +++ b/backend/fluksio/flow/pipeline.py @@ -1522,7 +1522,8 @@ class Pipeline: With a work queue attached the event is journaled and the caller returns immediately — that is the path every external trigger takes, so - a crash mid-cascade loses nothing. Interactive callers (a manual run, a + a crash mid-cascade costs at most the last second of journaled work + rather than the whole wave. Interactive callers (a manual run, a draft preview) pass ``durable=False`` and get the old synchronous behaviour, because they are waiting for the result. diff --git a/backend/fluksio/flow/queue.py b/backend/fluksio/flow/queue.py index 03d5ec6..7c0ef72 100644 --- a/backend/fluksio/flow/queue.py +++ b/backend/fluksio/flow/queue.py @@ -5,6 +5,10 @@ poll — becomes a work item before anything runs. The item is journaled first and acknowledged only once the wave it started has quiesced, so an engine that dies mid-cascade picks the work up again on the way back rather than losing it. +Redis journals with ``appendfsync everysec``, so the bound is the last second: +an unclean stop of Redis or of its host loses up to a second of entries. An +engine that dies while Redis lives loses none of them. + Two implementations: Redis Streams, which is what makes the above true, and an in-memory one for tests and for running without Redis, where "durable" degrades honestly to "not". diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 3361680..2193570 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -54,7 +54,13 @@ An instance that already has a managed venv keeps it on upgrade under | `REDIS_PORT` | `6379` | | Flow state is the last value of every message, node memory, and the run queue. -Redis here is persistence, not a cache. Run it with append-only persistence on. +Redis here is persistence, not a cache. Run it with append-only persistence on +(`--appendonly yes`, which the compose stack sets). + +That writes with Redis's default `appendfsync everysec`, so an unclean stop of +Redis or of its host loses up to a second of queued work. `appendfsync always` +closes the window and cuts engine throughput to about a third, so the default +stands. ## Identity and access