Close the nine open SDK tasks: one engine per directory, a tabbed dashboard, re-pairing, run recovery
Docs / docs (push) Successful in 27s
Playwright Tests / test-playwright (1, 2) (push) Failing after 17s
Playwright Tests / test-playwright (2, 2) (push) Failing after 12s
pre-commit / pre-commit (push) Failing after 1m59s
Test Backend / test-backend (push) Failing after 2m30s
Compose Smoke Test / test-compose (push) Failing after 13s
Playwright Tests / merge-reports (push) Failing after 2m19s

serve: refuse a second engine for one data directory whatever port it was
asked for, using the pidfile and a token this directory signed. The check
runs before the database is touched and before the credential is written,
which is what left every later CLI call pointing at a dead port.

The terminal dashboard is three tabs (Overview, Runs, Logs) with the toolbar
following the focused pane, the engine's output goes to serve.log rather than
down a pipe, and closing the screen stops both reader threads so the prompt
comes back. It adopts a running engine on every start, so stop/start and
restart work on one it did not start, and a stop waits for the process to be
gone before the next start. Enrolment reports itself in the modal.

enroll: a new claim code replaces the pairing instead of being refused. The
code is redeemed before anything is written, mappings to a portal being left
are cleared, and a running engine redials when the stored enrolment changes.

runs: an engine re-queues the runs left `queued` by the one before it, and
`fluksio retry <id>` / `retry --group <sweep>` submits an interrupted run
again with the same inputs and group, recorded through Run.parent_id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9BoNGq6V9MdRWAte7JBuC
This commit is contained in:
2026-08-31 17:37:13 +02:00
co-authored by Claude Opus 5
parent bdad6d7fc2
commit 8a94bf10d7
14 changed files with 873 additions and 140 deletions
+52
View File
@@ -799,6 +799,28 @@ class RunService:
target=self._keep_leases, name="run-leases", daemon=True
)
self._keeper.start()
self._requeue_queued()
def _requeue_queued(self) -> None:
"""Wake the engine for runs an earlier process was holding.
A row is journaled `queued` and then a work item is added, and the two
are not one transaction — an in-memory queue loses the item with the
process, and a stream item nobody claimed is nobody's. The row is the
record, so this reads it back. Nothing is re-executed by it: claiming
is a compare-and-set out of `queued`, so an item that did survive is
claimed once whatever this adds.
"""
with Session(db_engine) as session:
waiting = session.exec(
select(Run.id, Run.flow).where(col(Run.status) == "queued")
).all()
for run_id, flow in waiting:
self.queue.add(WorkItem(kind="run", node="", flow=flow, run_id=run_id))
if waiting:
logger.info(
"Requeued %d run(s) left queued by an earlier engine", len(waiting)
)
def stop(self) -> None:
self._stop.set()
@@ -828,6 +850,7 @@ class RunService:
draft: bool = False,
no_cache: bool = False,
idempotency_key: str | None = None,
parent_id: str | None = None,
) -> Run:
"""Journal a run and wake an engine up for it. Never blocks on it."""
# Before anything else, including reading the flow: a caller retrying a
@@ -889,6 +912,7 @@ class RunService:
created_at=datetime.now(UTC),
actor=actor,
idempotency_key=idempotency_key,
parent_id=parent_id,
)
with Session(db_engine) as session:
session.add(run)
@@ -940,6 +964,34 @@ class RunService:
self.controller.remote.cancel_run(run_id)
return True
def retry(self, run_id: str, actor: str = "") -> Run:
"""Submit the same run again, as a run of its own.
What an engine that stopped mid-run leaves behind is a run marked
`abandoned`, and re-issuing the whole sweep to recover one config is
the blunt way back. This is the sharp one: the same flow, the same
inputs, the same group, so a sweep is completed rather than repeated.
The stage cache is what makes it cheap — the nodes that finished are
restored rather than run again.
"""
with Session(db_engine) as session:
run = session.get(Run, run_id)
if run is None:
raise RunRejected(f"No run {run_id}")
if run.status in ("queued", "running"):
raise RunRejected("That run has not finished; cancel it first")
source = run.model_copy()
return self.submit(
source.flow,
params=dict(source.params),
seed=source.seed,
group_id=source.group_id,
cause="retry",
actor=actor,
no_cache=source.no_cache,
parent_id=source.id,
)
def _flow_of(self, run_id: str) -> str | None:
with Session(db_engine) as session:
run = session.get(Run, run_id)