Bound warm workers' host memory and account local RAM
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 12s
Playwright Tests / test-playwright (2, 2) (push) Failing after 9s
pre-commit / pre-commit (push) Failing after 1m56s
Test Backend / test-backend (push) Failing after 2m50s
Compose Smoke Test / test-compose (push) Failing after 9s
Playwright Tests / merge-reports (push) Failing after 2m16s
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 12s
Playwright Tests / test-playwright (2, 2) (push) Failing after 9s
pre-commit / pre-commit (push) Failing after 1m56s
Test Backend / test-backend (push) Failing after 2m50s
Compose Smoke Test / test-compose (push) Failing after 9s
Playwright Tests / merge-reports (push) Failing after 2m16s
A worker over FLOW_WORKER_MAX_RSS_MB (--worker-max-rss) when its node returns is retired instead of kept warm, in the shared pool and in every declared one; 0, the default, keeps them all. The engine now accounts its own memory (FLOW_RAM / --ram, else the box or the cgroup limit less 1 GB), so a node's `ram` declaration holds locally. A declared environment the shared pool already has is that pool rather than a duplicate set of warm workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -44,6 +44,7 @@ from fluksio.flow.resources import (
|
||||
ResourceAccountant,
|
||||
fair_share_env,
|
||||
machine_gpus,
|
||||
machine_ram_mb,
|
||||
)
|
||||
from fluksio.flow.runs import RUN_STATE_TTL, RunService, sweep_artifacts
|
||||
from fluksio.flow.secrets import init_secrets
|
||||
@@ -256,9 +257,11 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
volatile = VolatileStore(_volatile_root(), settings.ARTIFACT_VOLATILE_BYTES)
|
||||
artifacts.volatile = volatile
|
||||
app.state.artifact_store = artifacts
|
||||
# Zero means "work it out" for both, as it always has for cores.
|
||||
# Zero means "work it out" for cards and memory, as it always has for cores.
|
||||
accountant = ResourceAccountant(
|
||||
cpus=settings.FLOW_CPUS, gpus=settings.FLOW_GPUS or machine_gpus()
|
||||
cpus=settings.FLOW_CPUS,
|
||||
gpus=settings.FLOW_GPUS or machine_gpus(),
|
||||
ram_mb=settings.FLOW_RAM or machine_ram_mb(),
|
||||
)
|
||||
app.state.resources = accountant
|
||||
# Every machine a node could run on: this one, and whatever attaches.
|
||||
@@ -270,6 +273,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
python=modules.venv_python(),
|
||||
size=settings.FLOW_MAX_WORKERS,
|
||||
events=event_bus,
|
||||
max_rss_mb=settings.FLOW_WORKER_MAX_RSS_MB,
|
||||
# A worker in this container writes to the store directly; a remote one
|
||||
# is given a URL instead. Node code calls the same two functions.
|
||||
env={
|
||||
@@ -332,10 +336,12 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||
# Said out loud because it is the only way to tell that a settings file was
|
||||
# read at all — the numbers are what someone raising them is looking for.
|
||||
logger.info(
|
||||
"Engine limits: workers=%s cascades=%s runs=%s",
|
||||
"Engine limits: workers=%s cascades=%s runs=%s ram_mb=%s worker_max_rss_mb=%s",
|
||||
settings.FLOW_MAX_WORKERS,
|
||||
execution.max_cascades,
|
||||
run_service.parallel,
|
||||
accountant.ram_mb,
|
||||
settings.FLOW_WORKER_MAX_RSS_MB,
|
||||
)
|
||||
watchdog = LoopWatchdog(event_bus)
|
||||
app.state.watchdog = watchdog
|
||||
|
||||
Reference in New Issue
Block a user