Survive a busy engine: retry, idempotent submit, resilient waiting

A driver script died of one slow answer: httpx.ReadTimeout out of
RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which
cost a sweep 78 of its 84 runs.

- Split the timeout (5 s connect, 120 s read): a wrong URL fails at once,
  and a busy engine gets longer than the slowest thing it does on purpose
  (a 60 s compile, a 15 s rebuild wait).
- Retry idempotent calls three times on a transport error or 502/503/504.
  503 is the engine's own "ask again" — it is what RebuildBusy answers.
- Submit carries a key the engine stores with the run, so a retry after a
  timeout returns that run instead of starting a second. A sweep keys every
  entry, so a half-created one recreates only what is missing.
- wait() and --follow tolerate five failed polls in a row; a 404 still stops
  at once, because that is an answer rather than a gap.
- CLI says "engine not answering" and names the run still on the engine,
  instead of printing a traceback.
- runs: clamp the params column to 80 characters; events() takes the
  flow/since/until the endpoint already had; RunHandle.failures answers
  "what killed this run" from the run's own node rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

backend/fluksio/flow/runs.py

@@ -42,6 +42,7 @@ from typing import Any
 
 from sqlalchemy import update
 from sqlalchemy.dialects.sqlite import insert as upsert
+from sqlalchemy.exc import IntegrityError
 from sqlmodel import Session, col, select
 
 from fluksio.core.db import engine as db_engine
@@ -549,8 +550,16 @@ class RunService:
         actor: str = "",
         draft: bool = False,
         no_cache: bool = False,
+        idempotency_key: str | None = None,
     ) -> Run:
         """Journal a run and wake an engine up for it. Never blocks on it."""
+        # Before anything else, including reading the flow: a caller retrying a
+        # submit it never got an answer for is owed the run that answer was
+        # about, whatever the flow says now.
+        if idempotency_key:
+            existing = self._by_key(idempotency_key)
+            if existing is not None:
+                return existing
         flow = self.controller.store.read_flow(flow_name, draft=draft)
         issues = batch_issues(flow)
         if issues:
@@ -580,15 +589,33 @@ class RunService:
             labels=required_labels(flow),
             created_at=datetime.now(UTC),
             actor=actor,
+            idempotency_key=idempotency_key,
         )
         with Session(db_engine) as session:
             session.add(run)
-            session.commit()
+            try:
+                session.commit()
+            except IntegrityError:
+                # Two retries of one submit raced here. The unique index is
+                # what decided which of them is the run; this one reads it
+                # back rather than queueing a second execution of it.
+                session.rollback()
+                existing = self._by_key(idempotency_key) if idempotency_key else None
+                if existing is None:
+                    raise
+                return existing
             session.refresh(run)
 
         self.queue.add(WorkItem(kind="run", node="", flow=flow.name, run_id=run.id))
         return run
 
+    def _by_key(self, idempotency_key: str) -> Run | None:
+        """The run a key already made, if it made one."""
+        with Session(db_engine) as session:
+            return session.exec(
+                select(Run).where(col(Run.idempotency_key) == idempotency_key)
+            ).first()
+
     def cancel(self, run_id: str) -> bool:
         """Stop a run: kill what it is executing, schedule nothing further."""
         with self._lock: