Survive a busy engine: retry, idempotent submit, resilient waiting
A driver script died of one slow answer: httpx.ReadTimeout out of RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which cost a sweep 78 of its 84 runs. - Split the timeout (5 s connect, 120 s read): a wrong URL fails at once, and a busy engine gets longer than the slowest thing it does on purpose (a 60 s compile, a 15 s rebuild wait). - Retry idempotent calls three times on a transport error or 502/503/504. 503 is the engine's own "ask again" — it is what RebuildBusy answers. - Submit carries a key the engine stores with the run, so a retry after a timeout returns that run instead of starting a second. A sweep keys every entry, so a half-created one recreates only what is missing. - wait() and --follow tolerate five failed polls in a row; a 404 still stops at once, because that is an answer rather than a gap. - CLI says "engine not answering" and names the run still on the engine, instead of printing a traceback. - runs: clamp the params column to 80 characters; events() takes the flow/since/until the endpoint already had; RunHandle.failures answers "what killed this run" from the run's own node rows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -449,7 +449,33 @@ curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $
|
||||
The run carries its parameters, a digest of them, the seed, its result, how
|
||||
long each node took, what it logged, every artifact it produced, and both
|
||||
commits — the flow store's and your repository's. That is the answer to "what
|
||||
was the learning rate on the run that got 94%?".
|
||||
was the learning rate on the run that got 94%?". When a run failed,
|
||||
`run.failures` is the node that did it, with its traceback and its logs.
|
||||
|
||||
### When the engine is busy
|
||||
|
||||
A driver script outlives the engine being slow, because a sweep is hours long
|
||||
and a blip is seconds:
|
||||
|
||||
- **Reads are repeated.** Every GET, and the writes that mean the same thing
|
||||
twice, are tried up to three more times on a transport error or a `502`,
|
||||
`503` or `504`, waiting 1 s, 2 s, then 4 s. `Client(retries=0)` turns that
|
||||
off.
|
||||
- **Submitting is safe to repeat.** Each `submit()` mints a key the engine
|
||||
stores with the run, so an attempt that timed out on the way back is
|
||||
answered with the run it already made rather than starting a second one. A
|
||||
sweep keys every entry, so retrying a half-created one creates only what is
|
||||
missing.
|
||||
- **Waiting tolerates a gap.** `wait()` and `--follow` survive five failed
|
||||
polls in a row before giving up. An answer is not a gap: a `404` for a run
|
||||
that is gone stops immediately.
|
||||
- **Timeouts are split** — 5 s to connect, 120 s to read, so a wrong URL fails
|
||||
at once while a busy engine is given longer than the slowest thing it does
|
||||
on purpose. `Client(timeout=…)` takes a number or an `httpx.Timeout`.
|
||||
|
||||
A command that still cannot reach the engine says
|
||||
`fluksio: engine not answering (…)` and names the run it started, which is
|
||||
running on the engine regardless — `fluksio runs` finds it again.
|
||||
|
||||
## Sweep it
|
||||
|
||||
|
||||
Reference in New Issue
Block a user