Retire the GPU workers when a run that held a card finishes
A worker that has run a jax node keeps holding the GPU after the run: XLA preallocates most of the VRAM at import and never releases it, so the next process OOMs on preallocation while a warm idle worker sits on the card. Pools are kept warm on purpose — a library reads its environment at import, so a warm worker cannot be re-told — but the end of a run is a point where the memory should go back, and the environments carrying a GPU assignment are exactly the pools that ran on one. Idle ones go now, busy ones when they return. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc
This commit is contained in:
@@ -321,6 +321,24 @@ class PythonWorkerPool:
|
||||
for child in children:
|
||||
child.respawn_all()
|
||||
|
||||
def retire_gpu_children(self) -> None:
|
||||
"""Retire the pools holding a card, so the VRAM goes back.
|
||||
|
||||
A library like JAX takes most of the device when it imports and never
|
||||
releases it, so a warm worker that has run one such node is a held
|
||||
card — and warm is the point of a pool, so nothing retires it. At the
|
||||
end of a run there is something to key on: the environments carrying a
|
||||
GPU assignment are exactly the pools that ran on one.
|
||||
"""
|
||||
with self._lock:
|
||||
children = [
|
||||
child
|
||||
for key, child in self._children.items()
|
||||
if any(name == "CUDA_VISIBLE_DEVICES" for name, _ in key)
|
||||
]
|
||||
for child in children:
|
||||
child.respawn_all()
|
||||
|
||||
def _drain(self) -> list[_Worker | None]:
|
||||
slots = []
|
||||
while True:
|
||||
|
||||
Reference in New Issue
Block a user