Bound warm workers' host memory and account local RAM
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 12s
Playwright Tests / test-playwright (2, 2) (push) Failing after 9s
pre-commit / pre-commit (push) Failing after 1m56s
Test Backend / test-backend (push) Failing after 2m50s
Compose Smoke Test / test-compose (push) Failing after 9s
Playwright Tests / merge-reports (push) Failing after 2m16s
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 12s
Playwright Tests / test-playwright (2, 2) (push) Failing after 9s
pre-commit / pre-commit (push) Failing after 1m56s
Test Backend / test-backend (push) Failing after 2m50s
Compose Smoke Test / test-compose (push) Failing after 9s
Playwright Tests / merge-reports (push) Failing after 2m16s
A worker over FLOW_WORKER_MAX_RSS_MB (--worker-max-rss) when its node returns is retired instead of kept warm, in the shared pool and in every declared one; 0, the default, keeps them all. The engine now accounts its own memory (FLOW_RAM / --ram, else the box or the cgroup limit less 1 GB), so a node's `ram` declaration holds locally. A declared environment the shared pool already has is that pool rather than a duplicate set of warm workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -31,6 +31,7 @@ import re
|
||||
import threading
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from sqlmodel import Session
|
||||
|
||||
@@ -58,6 +59,12 @@ THREAD_VARS = (
|
||||
#: in that process, and a node is not allowed to take the last of it.
|
||||
ENGINE_RESERVE = 2
|
||||
|
||||
#: The same for memory, in megabytes.
|
||||
ENGINE_RAM_RESERVE_MB = 1024
|
||||
|
||||
#: Where a container finds its own memory limit (cgroup v2).
|
||||
CGROUP_MEMORY_MAX = Path("/sys/fs/cgroup/memory.max")
|
||||
|
||||
|
||||
def machine_cpus() -> int:
|
||||
"""Cores a node may be given here, when nobody said."""
|
||||
@@ -76,6 +83,27 @@ def machine_gpus() -> int:
|
||||
return len(glob.glob("/dev/nvidia[0-9]*"))
|
||||
|
||||
|
||||
def machine_ram_mb() -> int | None:
|
||||
"""Megabytes a node may be given here, when nobody said.
|
||||
|
||||
The box's memory, or the container's limit where that is lower — the
|
||||
host's total is memory the engine would be killed for using. None where
|
||||
neither can be asked, which leaves memory unaccounted rather than zero.
|
||||
"""
|
||||
try:
|
||||
total = os.sysconf("SC_PHYS_PAGES") * os.sysconf("SC_PAGE_SIZE") // 2**20
|
||||
except (AttributeError, ValueError, OSError):
|
||||
return None
|
||||
try:
|
||||
# "max" when the container has no limit of its own.
|
||||
limit = CGROUP_MEMORY_MAX.read_text().strip()
|
||||
except OSError:
|
||||
limit = ""
|
||||
if limit.isdigit():
|
||||
total = min(total, int(limit) // 2**20)
|
||||
return max(1, total - ENGINE_RAM_RESERVE_MB)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Allocation:
|
||||
"""What one execution was actually given."""
|
||||
|
||||
Reference in New Issue
Block a user