Refuse what a node cannot publish, and stop timing out work that is fine

Four things the python SDK turned up, each fixed where every client sees it.

A key no port declares is now an error rather than a silent drop, on the
return, the yield and the emit alike — the contract the docs already stated.
The SDK reads literal yields at sync time, so a typo fails before anything
runs, and an emission of one fails the call rather than being logged where
nobody looks.

NaN and infinity are refused at the port. JSON cannot spell either, so one
that travelled came back as a 500, a socket frame that stopped the canvas, or
a metric batch the database dropped whole.

An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the
engine — so the CLI, the run dialog and a python caller mean the same thing,
and a sweep can pass one at all.

Node timeouts are off by default. The clock measured silence, which a training
node is full of, and remote workers had already stopped enforcing it — their
heartbeat reset it. Now a heartbeat proves the agent rather than the node,
ninety seconds of nothing fails the call either way, and the engine touches
work it is still running so a long node is not redelivered at sixty seconds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
This commit is contained in:
2026-08-25 07:30:14 +02:00
co-authored by Claude Opus 5
parent c33fa404a4
commit 93374a310e
32 changed files with 968 additions and 67 deletions
+30 -8
View File
@@ -30,14 +30,17 @@ import time
from collections.abc import Callable
from typing import Any
from fluksio.flow.nodes.base import NodeOutputError
from fluksio.flow.workers import NodeTimeout, RemoteError, _remote_class
logger = logging.getLogger(__name__)
#: How long the loop is given to accept a frame we are handing it.
SEND_TIMEOUT_S = 30.0
#: A worker that has said nothing for this long is treated as gone. It sends a
#: heartbeat while it is executing, so this only ever catches a dead socket.
#: A worker that has said nothing at all for this long — not even a heartbeat —
#: is treated as gone. It beats every ten seconds while it is executing, so this
#: catches a dead socket rather than a slow node, and it is what bounds a call
#: whose node has no timeout of its own.
SILENCE_S = 90.0
#: Protocol version this engine speaks. A worker announcing anything else is
#: refused rather than half-understood.
@@ -125,15 +128,27 @@ class RemoteWorker:
future = asyncio.run_coroutine_threadsafe(self._send(payload), self._loop)
future.result(timeout=SEND_TIMEOUT_S)
# The node's own deadline measures silence, so a node reporting its
# progress is never mistaken for a hung one. A heartbeat is not
# progress: it says the agent is alive, which is what SILENCE_S
# asks, and says nothing about the node — so it feeds the liveness
# bound below and never the node's own.
deadline = time.monotonic() + timeout if timeout > 0 else None
while True:
wait = SILENCE_S
if deadline is not None:
wait = min(SILENCE_S, deadline - time.monotonic())
try:
# Reset per frame: the deadline measures silence, so a node
# reporting its progress is never mistaken for a hung one.
message = inbox.get(timeout=timeout)
message = inbox.get(timeout=max(wait, 0.0))
except queue.Empty:
self.cancel(call_id)
raise NodeTimeout(
f"'{self.name}' was silent for {timeout}s"
if deadline is not None and time.monotonic() >= deadline:
raise NodeTimeout(
f"'{self.name}' was silent for {timeout}s"
) from None
raise RemoteError(
f"worker '{self.name}' sent nothing for "
f"{SILENCE_S:.0f}s and is presumed gone"
) from None
if message is None:
raise RemoteError(f"worker '{self.name}' went away mid-call")
@@ -144,12 +159,19 @@ class RemoteWorker:
if on_event is not None:
try:
on_event(message)
except NodeOutputError:
# A port the node never declared. Stop the call
# rather than let the rest of its emissions arrive.
self.cancel(call_id)
raise
except Exception:
logger.exception("Could not record a worker event")
if deadline is not None:
deadline = time.monotonic() + timeout
continue
return message
except Exception as exc:
if isinstance(exc, (NodeTimeout, RemoteError)):
if isinstance(exc, (NodeTimeout, RemoteError, NodeOutputError)):
raise
raise RemoteError(f"worker '{self.name}': {exc}") from exc
finally: