Survive a busy engine: retry, idempotent submit, resilient waiting

A driver script died of one slow answer: httpx.ReadTimeout out of
RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which
cost a sweep 78 of its 84 runs.

- Split the timeout (5 s connect, 120 s read): a wrong URL fails at once,
  and a busy engine gets longer than the slowest thing it does on purpose
  (a 60 s compile, a 15 s rebuild wait).
- Retry idempotent calls three times on a transport error or 502/503/504.
  503 is the engine's own "ask again" — it is what RebuildBusy answers.
- Submit carries a key the engine stores with the run, so a retry after a
  timeout returns that run instead of starting a second. A sweep keys every
  entry, so a half-created one recreates only what is missing.
- wait() and --follow tolerate five failed polls in a row; a 404 still stops
  at once, because that is an answer rather than a gap.
- CLI says "engine not answering" and names the run still on the engine,
  instead of printing a traceback.
- runs: clamp the params column to 80 characters; events() takes the
  flow/since/until the endpoint already had; RunHandle.failures answers
  "what killed this run" from the run's own node rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-26 21:19:09 +02:00
co-authored by Claude Opus 5
parent 2cf45e4f39
commit 1f7c6646f1
12 changed files with 503 additions and 31 deletions
+4 -2
View File
@@ -273,8 +273,10 @@ fluksio runs [--flow train] [--limit 20]
The runs an engine has recorded, newest first: id, status, flow, duration, the
commit of the repository it came from, and its parameters. Statuses are
coloured when a terminal is reading the output — `ok` green, `error` red,
`cached` cyan. `--local` reads the same history from an in-process engine,
without one having to be served.
`cached` cyan. Parameters are clamped to 80 characters so a flow taking a few
kB of JSON still lists as a table; `Client.runs()` is where the whole value is
read. `--local` reads the same history from an in-process engine, without one
having to be served.
### `fluksio sweep`
+27 -1
View File
@@ -449,7 +449,33 @@ curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $
The run carries its parameters, a digest of them, the seed, its result, how
long each node took, what it logged, every artifact it produced, and both
commits — the flow store's and your repository's. That is the answer to "what
was the learning rate on the run that got 94%?".
was the learning rate on the run that got 94%?". When a run failed,
`run.failures` is the node that did it, with its traceback and its logs.
### When the engine is busy
A driver script outlives the engine being slow, because a sweep is hours long
and a blip is seconds:
- **Reads are repeated.** Every GET, and the writes that mean the same thing
twice, are tried up to three more times on a transport error or a `502`,
`503` or `504`, waiting 1 s, 2 s, then 4 s. `Client(retries=0)` turns that
off.
- **Submitting is safe to repeat.** Each `submit()` mints a key the engine
stores with the run, so an attempt that timed out on the way back is
answered with the run it already made rather than starting a second one. A
sweep keys every entry, so retrying a half-created one creates only what is
missing.
- **Waiting tolerates a gap.** `wait()` and `--follow` survive five failed
polls in a row before giving up. An answer is not a gap: a `404` for a run
that is gone stops immediately.
- **Timeouts are split** — 5 s to connect, 120 s to read, so a wrong URL fails
at once while a busy engine is given longer than the slowest thing it does
on purpose. `Client(timeout=…)` takes a number or an `httpx.Timeout`.
A command that still cannot reach the engine says
`fluksio: engine not answering (…)` and names the run it started, which is
running on the engine regardless — `fluksio runs` finds it again.
## Sweep it