Bound warm workers' host memory and account local RAM
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 12s
Playwright Tests / test-playwright (2, 2) (push) Failing after 9s
pre-commit / pre-commit (push) Failing after 1m56s
Test Backend / test-backend (push) Failing after 2m50s
Compose Smoke Test / test-compose (push) Failing after 9s
Playwright Tests / merge-reports (push) Failing after 2m16s

A worker over FLOW_WORKER_MAX_RSS_MB (--worker-max-rss) when its node
returns is retired instead of kept warm, in the shared pool and in every
declared one; 0, the default, keeps them all. The engine now accounts its
own memory (FLOW_RAM / --ram, else the box or the cgroup limit less 1 GB),
so a node's `ram` declaration holds locally. A declared environment the
shared pool already has is that pool rather than a duplicate set of warm
workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-27 16:11:13 +02:00
co-authored by Claude Opus 5.5
parent cc4091166e
commit dafdcb76c4
10 changed files with 143 additions and 8 deletions
+28
View File
@@ -31,6 +31,7 @@ import re
import threading
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path
from sqlmodel import Session
@@ -58,6 +59,12 @@ THREAD_VARS = (
#: in that process, and a node is not allowed to take the last of it.
ENGINE_RESERVE = 2
#: The same for memory, in megabytes.
ENGINE_RAM_RESERVE_MB = 1024
#: Where a container finds its own memory limit (cgroup v2).
CGROUP_MEMORY_MAX = Path("/sys/fs/cgroup/memory.max")
def machine_cpus() -> int:
"""Cores a node may be given here, when nobody said."""
@@ -76,6 +83,27 @@ def machine_gpus() -> int:
return len(glob.glob("/dev/nvidia[0-9]*"))
def machine_ram_mb() -> int | None:
"""Megabytes a node may be given here, when nobody said.
The box's memory, or the container's limit where that is lower — the
host's total is memory the engine would be killed for using. None where
neither can be asked, which leaves memory unaccounted rather than zero.
"""
try:
total = os.sysconf("SC_PHYS_PAGES") * os.sysconf("SC_PAGE_SIZE") // 2**20
except (AttributeError, ValueError, OSError):
return None
try:
# "max" when the container has no limit of its own.
limit = CGROUP_MEMORY_MAX.read_text().strip()
except OSError:
limit = ""
if limit.isdigit():
total = min(total, int(limit) // 2**20)
return max(1, total - ENGINE_RAM_RESERVE_MB)
@dataclass(frozen=True, slots=True)
class Allocation:
"""What one execution was actually given."""