Survive a busy engine: retry, idempotent submit, resilient waiting

A driver script died of one slow answer: httpx.ReadTimeout out of
RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which
cost a sweep 78 of its 84 runs.

- Split the timeout (5 s connect, 120 s read): a wrong URL fails at once,
  and a busy engine gets longer than the slowest thing it does on purpose
  (a 60 s compile, a 15 s rebuild wait).
- Retry idempotent calls three times on a transport error or 502/503/504.
  503 is the engine's own "ask again" — it is what RebuildBusy answers.
- Submit carries a key the engine stores with the run, so a retry after a
  timeout returns that run instead of starting a second. A sweep keys every
  entry, so a half-created one recreates only what is missing.
- wait() and --follow tolerate five failed polls in a row; a 404 still stops
  at once, because that is an answer rather than a gap.
- CLI says "engine not answering" and names the run still on the engine,
  instead of printing a traceback.
- runs: clamp the params column to 80 characters; events() takes the
  flow/since/until the endpoint already had; RunHandle.failures answers
  "what killed this run" from the run's own node rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-26 21:19:09 +02:00
co-authored by Claude Opus 5
parent 2cf45e4f39
commit 1f7c6646f1
12 changed files with 503 additions and 31 deletions
+8
View File
@@ -50,11 +50,17 @@ class RunCreate(BaseModel):
no_cache: bool = False
#: Who is asking. The dashboard leaves it, and is the "api" default.
cause: RunCause = "api"
#: A key the caller minted for this submission. Sending it again returns
#: the run it already made, so a retry after a timeout cannot double-submit.
idempotency_key: str | None = Field(default=None, max_length=64)
class SweepEntry(BaseModel):
params: dict[str, Any] = Field(default_factory=dict)
seed: int | None = None
#: One per entry, so retrying a half-created sweep recreates only the runs
#: whose rows never landed.
idempotency_key: str | None = Field(default=None, max_length=64)
class SweepCreate(BaseModel):
@@ -184,6 +190,7 @@ async def create_run(
actor=user.email,
draft=body.draft,
no_cache=body.no_cache,
idempotency_key=body.idempotency_key,
)
except FlowNotFound as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
@@ -221,6 +228,7 @@ async def create_sweep(
actor=user.email,
draft=body.draft,
no_cache=body.no_cache,
idempotency_key=entry.idempotency_key,
)
for entry in body.runs
]