Bound warm workers' host memory and account local RAM
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 12s
Playwright Tests / test-playwright (2, 2) (push) Failing after 9s
pre-commit / pre-commit (push) Failing after 1m56s
Test Backend / test-backend (push) Failing after 2m50s
Compose Smoke Test / test-compose (push) Failing after 9s
Playwright Tests / merge-reports (push) Failing after 2m16s

A worker over FLOW_WORKER_MAX_RSS_MB (--worker-max-rss) when its node
returns is retired instead of kept warm, in the shared pool and in every
declared one; 0, the default, keeps them all. The engine now accounts its
own memory (FLOW_RAM / --ram, else the box or the cgroup limit less 1 GB),
so a node's `ram` declaration holds locally. A declared environment the
shared pool already has is that pool rather than a duplicate set of warm
workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-27 16:11:13 +02:00
co-authored by Claude Opus 5.5
parent cc4091166e
commit dafdcb76c4
10 changed files with 143 additions and 8 deletions
+9 -3
View File
@@ -44,6 +44,7 @@ from fluksio.flow.resources import (
ResourceAccountant,
fair_share_env,
machine_gpus,
machine_ram_mb,
)
from fluksio.flow.runs import RUN_STATE_TTL, RunService, sweep_artifacts
from fluksio.flow.secrets import init_secrets
@@ -256,9 +257,11 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
volatile = VolatileStore(_volatile_root(), settings.ARTIFACT_VOLATILE_BYTES)
artifacts.volatile = volatile
app.state.artifact_store = artifacts
# Zero means "work it out" for both, as it always has for cores.
# Zero means "work it out" for cards and memory, as it always has for cores.
accountant = ResourceAccountant(
cpus=settings.FLOW_CPUS, gpus=settings.FLOW_GPUS or machine_gpus()
cpus=settings.FLOW_CPUS,
gpus=settings.FLOW_GPUS or machine_gpus(),
ram_mb=settings.FLOW_RAM or machine_ram_mb(),
)
app.state.resources = accountant
# Every machine a node could run on: this one, and whatever attaches.
@@ -270,6 +273,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
python=modules.venv_python(),
size=settings.FLOW_MAX_WORKERS,
events=event_bus,
max_rss_mb=settings.FLOW_WORKER_MAX_RSS_MB,
# A worker in this container writes to the store directly; a remote one
# is given a URL instead. Node code calls the same two functions.
env={
@@ -332,10 +336,12 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
# Said out loud because it is the only way to tell that a settings file was
# read at all — the numbers are what someone raising them is looking for.
logger.info(
"Engine limits: workers=%s cascades=%s runs=%s",
"Engine limits: workers=%s cascades=%s runs=%s ram_mb=%s worker_max_rss_mb=%s",
settings.FLOW_MAX_WORKERS,
execution.max_cascades,
run_service.parallel,
accountant.ram_mb,
settings.FLOW_WORKER_MAX_RSS_MB,
)
watchdog = LoopWatchdog(event_bus)
app.state.watchdog = watchdog