Refuse what a node cannot publish, and stop timing out work that is fine
Four things the python SDK turned up, each fixed where every client sees it. A key no port declares is now an error rather than a silent drop, on the return, the yield and the emit alike — the contract the docs already stated. The SDK reads literal yields at sync time, so a typo fails before anything runs, and an emission of one fails the call rather than being logged where nobody looks. NaN and infinity are refused at the port. JSON cannot spell either, so one that travelled came back as a 500, a socket frame that stopped the canvas, or a metric batch the database dropped whole. An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the engine — so the CLI, the run dialog and a python caller mean the same thing, and a sweep can pass one at all. Node timeouts are off by default. The clock measured silence, which a training node is full of, and remote workers had already stopped enforcing it — their heartbeat reset it. Now a heartbeat proves the agent rather than the node, ninety seconds of nothing fails the call either way, and the engine touches work it is still running so a long node is not redelivered at sixty seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
This commit is contained in:
@@ -182,6 +182,20 @@ the numbers the run reports as they arrive:
|
||||
Ctrl-C while either is waiting cancels the run on the engine rather than only
|
||||
stopping the watching, and exits 130.
|
||||
|
||||
An input declared as an `artifact` takes the file a previous run produced,
|
||||
named rather than typed out:
|
||||
|
||||
```sh
|
||||
fluksio run evaluate --dataset @run:1758042000123-9f2ab41c.dataset
|
||||
fluksio run evaluate --dataset sha256:6dd1f0…
|
||||
```
|
||||
|
||||
`@run:<id>.<output>` is what that run's output was, and a bare digest is the
|
||||
content itself; the engine resolves either into the reference. Passing the
|
||||
whole reference as JSON still works and is what a script that already holds one
|
||||
does — which is the same thing `flow.submit(dataset=run.result["dataset"])`
|
||||
does from Python.
|
||||
|
||||
`--no-sync` runs what is already on the engine. Worth it in a tight loop where
|
||||
you know nothing changed, since syncing retires the workers and the next call
|
||||
pays its imports again. A directory that declares no flows syncs nothing and
|
||||
|
||||
+9
-6
@@ -131,13 +131,16 @@ acknowledged from the canvas.
|
||||
|
||||
## Timeouts
|
||||
|
||||
`timeout` on a node is how many seconds its code may run before it is stopped.
|
||||
The default is 30, and it covers the *first* call's imports, which can be much
|
||||
slower than the body — a node importing torch is not being slow, it is loading.
|
||||
`timeout` on a node is how many seconds its code may be *silent* before it is
|
||||
stopped. A yield or an `emit` resets the clock, and the first call's imports are
|
||||
not charged to it — a node importing torch is not being slow, it is loading.
|
||||
|
||||
Above 60 seconds, a live flow may deliver the same work again while the node is
|
||||
still running. In a batch run, which never redelivers, it is an idle timeout
|
||||
instead: silence this long is a kill.
|
||||
There is no timeout by default. Training runs for hours and a node that reports
|
||||
nothing is usually working, so the engine waits: what fails a call is the worker
|
||||
dying, which arrives at once rather than after a deadline. Set a timeout on the
|
||||
nodes where silence means stuck — an HTTP call that should answer in seconds,
|
||||
a loop that can spin — either per node or, for all of them,
|
||||
with `FLOW_NODE_TIMEOUT`. `timeout = 0` says explicitly that this node has none.
|
||||
|
||||
## Running a node somewhere else
|
||||
|
||||
|
||||
@@ -100,8 +100,12 @@ you want once it does.
|
||||
other runs of the same node alone.
|
||||
- **If the worker disappears mid-call**, the run fails in seconds with
|
||||
`worker went away mid-call` rather than waiting out its timeout.
|
||||
- **A worker sends a heartbeat while it executes**, so a long node is
|
||||
distinguishable from a dead socket. Ninety seconds of silence is gone.
|
||||
- **A worker sends a heartbeat every ten seconds while it executes**, so a long
|
||||
node is distinguishable from a dead socket. Ninety seconds of nothing at all —
|
||||
not even a heartbeat — fails the call as gone. A heartbeat says the *agent* is
|
||||
alive and nothing about the node, so it never satisfies a node's own timeout:
|
||||
one set to thirty seconds fires after thirty seconds of the node reporting
|
||||
nothing, wherever it runs.
|
||||
|
||||
## Artifacts across machines
|
||||
|
||||
|
||||
@@ -148,7 +148,9 @@ display not to be flooded, not for the curve to have holes in it.
|
||||
Emitting has a second effect: **a node's timeout measures silence, not
|
||||
duration**. A node that yields every few seconds can run for hours under a
|
||||
`timeout` of 300; one that says nothing for longer than its timeout is killed.
|
||||
Set `timeout` on a long node to how long it may plausibly go quiet.
|
||||
There is no timeout unless one is set — a training node that reports nothing is
|
||||
usually working — so set `timeout` where going quiet means stuck, at how long
|
||||
the node may plausibly be.
|
||||
|
||||
In a *live* flow, an emission also wakes whatever is downstream of it, exactly
|
||||
as a subscriber publishing does. In a run it does not: a run's graph is
|
||||
|
||||
@@ -56,10 +56,11 @@ A Function node gets a second settings section where you add your own: a
|
||||
setting is a constant of this node's code and arrives as an argument beside its
|
||||
ports. See [Where a node's values come from](../concepts/values.md).
|
||||
|
||||
**Timeout** (Function nodes) is how many seconds the code may run before it is
|
||||
stopped. Once a node streams, this becomes an *idle* timeout: it measures
|
||||
silence, not duration, so a node that yields every few seconds can run for
|
||||
hours under a timeout of 300.
|
||||
**Timeout** (Function nodes) is how many seconds the code may be *silent*
|
||||
before it is stopped: it measures silence, not duration, so a node that yields
|
||||
every few seconds can run for hours under a timeout of 300. Left empty there is
|
||||
no limit, which is the default — set one where a node going quiet means it is
|
||||
stuck rather than working. A `0` says so explicitly.
|
||||
|
||||
**Code** is the editor. It saves as you type and applies on ⌘S; the node
|
||||
reloads without the flow stopping. A node file defines `process(...)`; if it
|
||||
|
||||
@@ -123,7 +123,7 @@ warning into a refusal to start.
|
||||
| Variable | Default | Notes |
|
||||
|---|---|---|
|
||||
| `FLOW_MAX_WORKERS` | `4` | node-code subprocesses run in parallel |
|
||||
| `FLOW_NODE_TIMEOUT` | `30.0` | seconds a node may run, unless it sets its own |
|
||||
| `FLOW_NODE_TIMEOUT` | `0` | seconds a node may be silent, unless it sets its own; 0 is no limit |
|
||||
| `OBS_RETENTION_DAYS` | `30` | how long metrics, events and run records are kept |
|
||||
|
||||
## Agents
|
||||
|
||||
@@ -24,6 +24,12 @@ the state backend, the work queue and the worker protocol unchanged.
|
||||
is not a measurement, and a switch bound to a temperature is a mistake worth
|
||||
catching.
|
||||
|
||||
`NaN` and infinity are refused, wherever they sit — including inside a `json`,
|
||||
`record`, `series` or `list`. JSON cannot spell either, so one that travelled
|
||||
would come back as a response nobody can parse and a row the database rejects,
|
||||
a long way from the node that made it. An empty subset or a division with no
|
||||
denominator is what usually produces one; publish `None` instead.
|
||||
|
||||
## The structured ones
|
||||
|
||||
These are *declared shapes* rather than "some JSON", which is what makes a
|
||||
@@ -93,6 +99,12 @@ stays valid wherever the store is reachable from, including on another machine.
|
||||
Node code produces one with `fluksio.save_artifact` and opens one with
|
||||
`fluksio.load_artifact`. See [Writing node code](../code/nodes.md#bytes-artifacts).
|
||||
|
||||
As a *run parameter* it is also accepted as text, since nobody wants to paste
|
||||
the object into a shell: `@run:<id>.<output>` names what a run produced, and a
|
||||
bare `sha256:…` digest names the content. Both resolve to the reference above
|
||||
before the run starts, so the CLI, the run dialog and a python caller all mean
|
||||
the same thing.
|
||||
|
||||
### `json`
|
||||
|
||||
Anything JSON-serializable. The escape hatch, and the right answer when a
|
||||
|
||||
Reference in New Issue
Block a user