Survive a busy engine: retry, idempotent submit, resilient waiting
A driver script died of one slow answer: httpx.ReadTimeout out of RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which cost a sweep 78 of its 84 runs. - Split the timeout (5 s connect, 120 s read): a wrong URL fails at once, and a busy engine gets longer than the slowest thing it does on purpose (a 60 s compile, a 15 s rebuild wait). - Retry idempotent calls three times on a transport error or 502/503/504. 503 is the engine's own "ask again" — it is what RebuildBusy answers. - Submit carries a key the engine stores with the run, so a retry after a timeout returns that run instead of starting a second. A sweep keys every entry, so a half-created one recreates only what is missing. - wait() and --follow tolerate five failed polls in a row; a 404 still stops at once, because that is an answer rather than a gap. - CLI says "engine not answering" and names the run still on the engine, instead of printing a traceback. - runs: clamp the params column to 80 characters; events() takes the flow/since/until the endpoint already had; RunHandle.failures answers "what killed this run" from the run's own node rows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -340,6 +340,12 @@ class Run(SQLModel, table=True):
|
||||
group_id: str | None = Field(default=None, index=True, max_length=64)
|
||||
#: The run this one was made from, on a retry.
|
||||
parent_id: str | None = Field(default=None, max_length=64)
|
||||
#: A key the caller made up for one submission, so retrying a submit that
|
||||
#: may already have landed returns that run instead of starting a second.
|
||||
#: Unique where it is set; null for anything submitted without one.
|
||||
idempotency_key: str | None = Field(
|
||||
default=None, unique=True, index=True, max_length=64
|
||||
)
|
||||
#: Where it was asked for: api, cli, sdk, hook or sweep.
|
||||
cause: str = Field(default="api", max_length=32)
|
||||
#: Re-execute every node, whatever the stage cache holds for it.
|
||||
|
||||
Reference in New Issue
Block a user