Refuse what a node cannot publish, and stop timing out work that is fine
Four things the python SDK turned up, each fixed where every client sees it. A key no port declares is now an error rather than a silent drop, on the return, the yield and the emit alike — the contract the docs already stated. The SDK reads literal yields at sync time, so a typo fails before anything runs, and an emission of one fails the call rather than being logged where nobody looks. NaN and infinity are refused at the port. JSON cannot spell either, so one that travelled came back as a 500, a socket frame that stopped the canvas, or a metric batch the database dropped whole. An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the engine — so the CLI, the run dialog and a python caller mean the same thing, and a sweep can pass one at all. Node timeouts are off by default. The clock measured silence, which a training node is full of, and remote workers had already stopped enforcing it — their heartbeat reset it. Now a heartbeat proves the agent rather than the node, ninety seconds of nothing fails the call either way, and the engine touches work it is still running so a long node is not redelivered at sixty seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
This commit is contained in:
@@ -182,6 +182,20 @@ the numbers the run reports as they arrive:
|
||||
Ctrl-C while either is waiting cancels the run on the engine rather than only
|
||||
stopping the watching, and exits 130.
|
||||
|
||||
An input declared as an `artifact` takes the file a previous run produced,
|
||||
named rather than typed out:
|
||||
|
||||
```sh
|
||||
fluksio run evaluate --dataset @run:1758042000123-9f2ab41c.dataset
|
||||
fluksio run evaluate --dataset sha256:6dd1f0…
|
||||
```
|
||||
|
||||
`@run:<id>.<output>` is what that run's output was, and a bare digest is the
|
||||
content itself; the engine resolves either into the reference. Passing the
|
||||
whole reference as JSON still works and is what a script that already holds one
|
||||
does — which is the same thing `flow.submit(dataset=run.result["dataset"])`
|
||||
does from Python.
|
||||
|
||||
`--no-sync` runs what is already on the engine. Worth it in a tight loop where
|
||||
you know nothing changed, since syncing retires the workers and the next call
|
||||
pays its imports again. A directory that declares no flows syncs nothing and
|
||||
|
||||
+9
-6
@@ -131,13 +131,16 @@ acknowledged from the canvas.
|
||||
|
||||
## Timeouts
|
||||
|
||||
`timeout` on a node is how many seconds its code may run before it is stopped.
|
||||
The default is 30, and it covers the *first* call's imports, which can be much
|
||||
slower than the body — a node importing torch is not being slow, it is loading.
|
||||
`timeout` on a node is how many seconds its code may be *silent* before it is
|
||||
stopped. A yield or an `emit` resets the clock, and the first call's imports are
|
||||
not charged to it — a node importing torch is not being slow, it is loading.
|
||||
|
||||
Above 60 seconds, a live flow may deliver the same work again while the node is
|
||||
still running. In a batch run, which never redelivers, it is an idle timeout
|
||||
instead: silence this long is a kill.
|
||||
There is no timeout by default. Training runs for hours and a node that reports
|
||||
nothing is usually working, so the engine waits: what fails a call is the worker
|
||||
dying, which arrives at once rather than after a deadline. Set a timeout on the
|
||||
nodes where silence means stuck — an HTTP call that should answer in seconds,
|
||||
a loop that can spin — either per node or, for all of them,
|
||||
with `FLOW_NODE_TIMEOUT`. `timeout = 0` says explicitly that this node has none.
|
||||
|
||||
## Running a node somewhere else
|
||||
|
||||
|
||||
@@ -100,8 +100,12 @@ you want once it does.
|
||||
other runs of the same node alone.
|
||||
- **If the worker disappears mid-call**, the run fails in seconds with
|
||||
`worker went away mid-call` rather than waiting out its timeout.
|
||||
- **A worker sends a heartbeat while it executes**, so a long node is
|
||||
distinguishable from a dead socket. Ninety seconds of silence is gone.
|
||||
- **A worker sends a heartbeat every ten seconds while it executes**, so a long
|
||||
node is distinguishable from a dead socket. Ninety seconds of nothing at all —
|
||||
not even a heartbeat — fails the call as gone. A heartbeat says the *agent* is
|
||||
alive and nothing about the node, so it never satisfies a node's own timeout:
|
||||
one set to thirty seconds fires after thirty seconds of the node reporting
|
||||
nothing, wherever it runs.
|
||||
|
||||
## Artifacts across machines
|
||||
|
||||
|
||||
Reference in New Issue
Block a user