Survive a busy engine: retry, idempotent submit, resilient waiting
A driver script died of one slow answer: httpx.ReadTimeout out of RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which cost a sweep 78 of its 84 runs. - Split the timeout (5 s connect, 120 s read): a wrong URL fails at once, and a busy engine gets longer than the slowest thing it does on purpose (a 60 s compile, a 15 s rebuild wait). - Retry idempotent calls three times on a transport error or 502/503/504. 503 is the engine's own "ask again" — it is what RebuildBusy answers. - Submit carries a key the engine stores with the run, so a retry after a timeout returns that run instead of starting a second. A sweep keys every entry, so a half-created one recreates only what is missing. - wait() and --follow tolerate five failed polls in a row; a 404 still stops at once, because that is an answer rather than a gap. - CLI says "engine not answering" and names the run still on the engine, instead of printing a traceback. - runs: clamp the params column to 80 characters; events() takes the flow/since/until the endpoint already had; RunHandle.failures answers "what killed this run" from the run's own node rows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,40 @@
|
||||
"""run.idempotency_key
|
||||
|
||||
Submitting is a POST, so a client that retries one after a timeout cannot know
|
||||
whether the first attempt landed. A key the caller mints per submission makes
|
||||
the answer a lookup: the same key returns the run it already created.
|
||||
|
||||
Revision ID: f2c6a8d15e93
|
||||
Revises: d1f7a3c8b204
|
||||
Create Date: 2026-08-26
|
||||
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
import sqlmodel.sql.sqltypes
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "f2c6a8d15e93"
|
||||
down_revision = "d1f7a3c8b204"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column(
|
||||
"run",
|
||||
sa.Column(
|
||||
"idempotency_key",
|
||||
sqlmodel.sql.sqltypes.AutoString(length=64),
|
||||
nullable=True,
|
||||
),
|
||||
)
|
||||
# Nullable and unique: every run submitted without a key stays NULL, and
|
||||
# both SQLite and Postgres allow as many of those as there are runs.
|
||||
op.create_index("ix_run_idempotency_key", "run", ["idempotency_key"], unique=True)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_index("ix_run_idempotency_key", table_name="run")
|
||||
op.drop_column("run", "idempotency_key")
|
||||
Reference in New Issue
Block a user