Refuse what a node cannot publish, and stop timing out work that is fine
Four things the python SDK turned up, each fixed where every client sees it. A key no port declares is now an error rather than a silent drop, on the return, the yield and the emit alike — the contract the docs already stated. The SDK reads literal yields at sync time, so a typo fails before anything runs, and an emission of one fails the call rather than being logged where nobody looks. NaN and infinity are refused at the port. JSON cannot spell either, so one that travelled came back as a 500, a socket frame that stopped the canvas, or a metric batch the database dropped whole. An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the engine — so the CLI, the run dialog and a python caller mean the same thing, and a sweep can pass one at all. Node timeouts are off by default. The clock measured silence, which a training node is full of, and remote workers had already stopped enforcing it — their heartbeat reset it. Now a heartbeat proves the agent rather than the node, ninety seconds of nothing fails the call either way, and the engine touches work it is still running so a long node is not redelivered at sixty seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
This commit is contained in:
@@ -1,7 +1,9 @@
|
||||
"""The work queue, and what the execution service does with it."""
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
from fluksio.flow import executor
|
||||
from fluksio.flow.executor import ExecutionService
|
||||
from fluksio.flow.messages import DType, MessageSpec
|
||||
from fluksio.flow.nodes import Node
|
||||
@@ -264,3 +266,72 @@ def test_a_replayed_item_does_not_repeat_a_side_effect():
|
||||
service._run_item(item)
|
||||
|
||||
assert calls == [5.0]
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Telling the queue that a long node is working, not lost
|
||||
#
|
||||
# What marks an item abandoned is nobody touching it. A node with no timeout
|
||||
# may run for hours, so the engine holding it says so on a timer — and an
|
||||
# engine that died says nothing, which is the distinction the reaper needs.
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
|
||||
class RecordingQueue(MemoryWorkQueue):
|
||||
"""A memory queue that writes down what it was asked to hold on to."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.touched: list[list[str]] = []
|
||||
|
||||
def touch(self, entry_ids: list[str]) -> None:
|
||||
self.touched.append(list(entry_ids))
|
||||
|
||||
|
||||
def test_work_in_flight_is_touched_until_it_finishes(monkeypatch):
|
||||
monkeypatch.setattr(executor, "TOUCH_INTERVAL_S", 0.0)
|
||||
monkeypatch.setattr(executor, "DELAYED_INTERVAL_S", 0.05)
|
||||
running = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
def slow(reading, params):
|
||||
running.set()
|
||||
release.wait(5)
|
||||
return {"doubled": reading * 2}
|
||||
|
||||
source = Node(
|
||||
f=lambda params: None,
|
||||
provides=[MessageSpec(name="reading", dtype=DType.FLOAT)],
|
||||
name="source",
|
||||
)
|
||||
consumer = Node(
|
||||
f=slow,
|
||||
requires=[MessageSpec(name="reading", dtype=DType.FLOAT)],
|
||||
provides=[MessageSpec(name="doubled", dtype=DType.FLOAT)],
|
||||
name="consumer",
|
||||
)
|
||||
source.assign_flow("f", "source")
|
||||
consumer.assign_flow("f", "consumer")
|
||||
|
||||
queue = RecordingQueue()
|
||||
pipeline = Pipeline(
|
||||
nodes=[source, consumer], state=MemoryState(), work_queue=queue
|
||||
)
|
||||
service = ExecutionService(queue)
|
||||
service.bind(pipeline)
|
||||
service.start()
|
||||
try:
|
||||
source.inject({"reading": 3.0})
|
||||
assert running.wait(5)
|
||||
# Give the timer a couple of passes while the node is still in there.
|
||||
time.sleep(0.2)
|
||||
held = [ids for ids in queue.touched if ids]
|
||||
assert held, "a running item was never touched"
|
||||
|
||||
release.set()
|
||||
time.sleep(0.3)
|
||||
# Once it is done it is acknowledged, so there is nothing to hold.
|
||||
assert queue.touched[-1] == []
|
||||
finally:
|
||||
release.set()
|
||||
service.stop()
|
||||
|
||||
Reference in New Issue
Block a user