Retire the GPU workers when a run that held a card finishes

A worker that has run a jax node keeps holding the GPU after the run: XLA
preallocates most of the VRAM at import and never releases it, so the next
process OOMs on preallocation while a warm idle worker sits on the card.
Pools are kept warm on purpose — a library reads its environment at import,
so a warm worker cannot be re-told — but the end of a run is a point where
the memory should go back, and the environments carrying a GPU assignment
are exactly the pools that ran on one. Idle ones go now, busy ones when
they return.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc
This commit is contained in:
2026-08-29 13:53:11 +02:00
co-authored by Claude Opus 5
parent de87151c60
commit 53b49e5f68
3 changed files with 55 additions and 0 deletions
+18
View File
@@ -321,6 +321,24 @@ class PythonWorkerPool:
for child in children:
child.respawn_all()
def retire_gpu_children(self) -> None:
"""Retire the pools holding a card, so the VRAM goes back.
A library like JAX takes most of the device when it imports and never
releases it, so a warm worker that has run one such node is a held
card — and warm is the point of a pool, so nothing retires it. At the
end of a run there is something to key on: the environments carrying a
GPU assignment are exactly the pools that ran on one.
"""
with self._lock:
children = [
child
for key, child in self._children.items()
if any(name == "CUDA_VISIBLE_DEVICES" for name, _ in key)
]
for child in children:
child.respawn_all()
def _drain(self) -> list[_Worker | None]:
slots = []
while True: