Refuse what a node cannot publish, and stop timing out work that is fine
Four things the python SDK turned up, each fixed where every client sees it. A key no port declares is now an error rather than a silent drop, on the return, the yield and the emit alike — the contract the docs already stated. The SDK reads literal yields at sync time, so a typo fails before anything runs, and an emission of one fails the call rather than being logged where nobody looks. NaN and infinity are refused at the port. JSON cannot spell either, so one that travelled came back as a 500, a socket frame that stopped the canvas, or a metric batch the database dropped whole. An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the engine — so the CLI, the run dialog and a python caller mean the same thing, and a sweep can pass one at all. Node timeouts are off by default. The clock measured silence, which a training node is full of, and remote workers had already stopped enforcing it — their heartbeat reset it. Now a heartbeat proves the agent rather than the node, ninety seconds of nothing fails the call either way, and the engine touches work it is still running so a long node is not redelivered at sixty seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
This commit is contained in:
@@ -30,14 +30,17 @@ import time
|
||||
from collections.abc import Callable
|
||||
from typing import Any
|
||||
|
||||
from fluksio.flow.nodes.base import NodeOutputError
|
||||
from fluksio.flow.workers import NodeTimeout, RemoteError, _remote_class
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: How long the loop is given to accept a frame we are handing it.
|
||||
SEND_TIMEOUT_S = 30.0
|
||||
#: A worker that has said nothing for this long is treated as gone. It sends a
|
||||
#: heartbeat while it is executing, so this only ever catches a dead socket.
|
||||
#: A worker that has said nothing at all for this long — not even a heartbeat —
|
||||
#: is treated as gone. It beats every ten seconds while it is executing, so this
|
||||
#: catches a dead socket rather than a slow node, and it is what bounds a call
|
||||
#: whose node has no timeout of its own.
|
||||
SILENCE_S = 90.0
|
||||
#: Protocol version this engine speaks. A worker announcing anything else is
|
||||
#: refused rather than half-understood.
|
||||
@@ -125,15 +128,27 @@ class RemoteWorker:
|
||||
future = asyncio.run_coroutine_threadsafe(self._send(payload), self._loop)
|
||||
future.result(timeout=SEND_TIMEOUT_S)
|
||||
|
||||
# The node's own deadline measures silence, so a node reporting its
|
||||
# progress is never mistaken for a hung one. A heartbeat is not
|
||||
# progress: it says the agent is alive, which is what SILENCE_S
|
||||
# asks, and says nothing about the node — so it feeds the liveness
|
||||
# bound below and never the node's own.
|
||||
deadline = time.monotonic() + timeout if timeout > 0 else None
|
||||
while True:
|
||||
wait = SILENCE_S
|
||||
if deadline is not None:
|
||||
wait = min(SILENCE_S, deadline - time.monotonic())
|
||||
try:
|
||||
# Reset per frame: the deadline measures silence, so a node
|
||||
# reporting its progress is never mistaken for a hung one.
|
||||
message = inbox.get(timeout=timeout)
|
||||
message = inbox.get(timeout=max(wait, 0.0))
|
||||
except queue.Empty:
|
||||
self.cancel(call_id)
|
||||
raise NodeTimeout(
|
||||
f"'{self.name}' was silent for {timeout}s"
|
||||
if deadline is not None and time.monotonic() >= deadline:
|
||||
raise NodeTimeout(
|
||||
f"'{self.name}' was silent for {timeout}s"
|
||||
) from None
|
||||
raise RemoteError(
|
||||
f"worker '{self.name}' sent nothing for "
|
||||
f"{SILENCE_S:.0f}s and is presumed gone"
|
||||
) from None
|
||||
if message is None:
|
||||
raise RemoteError(f"worker '{self.name}' went away mid-call")
|
||||
@@ -144,12 +159,19 @@ class RemoteWorker:
|
||||
if on_event is not None:
|
||||
try:
|
||||
on_event(message)
|
||||
except NodeOutputError:
|
||||
# A port the node never declared. Stop the call
|
||||
# rather than let the rest of its emissions arrive.
|
||||
self.cancel(call_id)
|
||||
raise
|
||||
except Exception:
|
||||
logger.exception("Could not record a worker event")
|
||||
if deadline is not None:
|
||||
deadline = time.monotonic() + timeout
|
||||
continue
|
||||
return message
|
||||
except Exception as exc:
|
||||
if isinstance(exc, (NodeTimeout, RemoteError)):
|
||||
if isinstance(exc, (NodeTimeout, RemoteError, NodeOutputError)):
|
||||
raise
|
||||
raise RemoteError(f"worker '{self.name}': {exc}") from exc
|
||||
finally:
|
||||
|
||||
Reference in New Issue
Block a user