Refuse what a node cannot publish, and stop timing out work that is fine

Four things the python SDK turned up, each fixed where every client sees it.

A key no port declares is now an error rather than a silent drop, on the
return, the yield and the emit alike — the contract the docs already stated.
The SDK reads literal yields at sync time, so a typo fails before anything
runs, and an emission of one fails the call rather than being logged where
nobody looks.

NaN and infinity are refused at the port. JSON cannot spell either, so one
that travelled came back as a 500, a socket frame that stopped the canvas, or
a metric batch the database dropped whole.

An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the
engine — so the CLI, the run dialog and a python caller mean the same thing,
and a sweep can pass one at all.

Node timeouts are off by default. The clock measured silence, which a training
node is full of, and remote workers had already stopped enforcing it — their
heartbeat reset it. Now a heartbeat proves the agent rather than the node,
ninety seconds of nothing fails the call either way, and the engine touches
work it is still running so a long node is not redelivered at sixty seconds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
This commit is contained in:
2026-08-25 07:30:14 +02:00
co-authored by Claude Opus 5
parent c33fa404a4
commit 93374a310e
32 changed files with 968 additions and 67 deletions
+23 -3
View File
@@ -31,6 +31,10 @@ CLAIM_BLOCK_MS = 1000
# Long enough that a busy cascade is not mistaken for a dead one.
RECLAIM_IDLE_MS = 60_000
RECLAIM_INTERVAL_S = 30.0
# How often to tell the queue that what we hold is still being worked on. A
# node may run for as long as it likes, so what marks an item abandoned is this
# stopping — which is what an engine that died does.
TOUCH_INTERVAL_S = 20.0
DELAYED_INTERVAL_S = 1.0
MAX_CASCADES = 4
# How long a reload waits for claimed work to finish before rebuilding anyway.
@@ -54,6 +58,8 @@ class ExecutionService:
self._intake.set()
self._inflight = 0
self._inflight_lock = threading.Condition()
# Entry ids claimed and still running, under _inflight_lock.
self._active: set[str] = set()
self.node_pool = ThreadPoolExecutor(
max_workers=max_workers or 4, thread_name_prefix="node"
)
@@ -145,6 +151,7 @@ class ExecutionService:
def _tick(self) -> None:
"""Promote delayed items, and take back what a dead engine dropped."""
last_reclaim = 0.0
last_touch = 0.0
while not self._stop.is_set():
self._stop.wait(DELAYED_INTERVAL_S)
if self._stop.is_set():
@@ -155,6 +162,16 @@ class ExecutionService:
logger.error("Could not promote delayed work: %s", exc)
now = time.monotonic()
if now - last_touch >= TOUCH_INTERVAL_S:
last_touch = now
with self._inflight_lock:
running = list(self._active)
try:
self.queue.touch(running)
except Exception as exc:
logger.error("Could not touch claimed work: %s", exc)
if now - last_reclaim < RECLAIM_INTERVAL_S:
continue
last_reclaim = now
@@ -172,15 +189,18 @@ class ExecutionService:
def _dispatch(self, item: WorkItem) -> None:
with self._inflight_lock:
self._inflight += 1
if item.entry_id:
self._active.add(item.entry_id)
try:
self._cascade_pool.submit(self._handle, item)
except RuntimeError:
# Pool already shutting down.
self._done()
self._done(item)
def _done(self) -> None:
def _done(self, item: WorkItem) -> None:
with self._inflight_lock:
self._inflight -= 1
self._active.discard(item.entry_id)
self._inflight_lock.notify_all()
# -------------------------------------------------------------------------
@@ -222,7 +242,7 @@ class ExecutionService:
logger.error(
"Could not acknowledge work for '%s': %s", item.node, exc
)
self._done()
self._done(item)
def _run_item(self, item: WorkItem) -> bool:
"""Run one item. False means it was not handled and must come back."""