Close the nine open SDK tasks: one engine per directory, a tabbed dashboard, re-pairing, run recovery
Docs / docs (push) Successful in 27s
Playwright Tests / test-playwright (1, 2) (push) Failing after 17s
Playwright Tests / test-playwright (2, 2) (push) Failing after 12s
pre-commit / pre-commit (push) Failing after 1m59s
Test Backend / test-backend (push) Failing after 2m30s
Compose Smoke Test / test-compose (push) Failing after 13s
Playwright Tests / merge-reports (push) Failing after 2m19s
Docs / docs (push) Successful in 27s
Playwright Tests / test-playwright (1, 2) (push) Failing after 17s
Playwright Tests / test-playwright (2, 2) (push) Failing after 12s
pre-commit / pre-commit (push) Failing after 1m59s
Test Backend / test-backend (push) Failing after 2m30s
Compose Smoke Test / test-compose (push) Failing after 13s
Playwright Tests / merge-reports (push) Failing after 2m19s
serve: refuse a second engine for one data directory whatever port it was asked for, using the pidfile and a token this directory signed. The check runs before the database is touched and before the credential is written, which is what left every later CLI call pointing at a dead port. The terminal dashboard is three tabs (Overview, Runs, Logs) with the toolbar following the focused pane, the engine's output goes to serve.log rather than down a pipe, and closing the screen stops both reader threads so the prompt comes back. It adopts a running engine on every start, so stop/start and restart work on one it did not start, and a stop waits for the process to be gone before the next start. Enrolment reports itself in the modal. enroll: a new claim code replaces the pairing instead of being refused. The code is redeemed before anything is written, mappings to a portal being left are cleared, and a running engine redials when the stored enrolment changes. runs: an engine re-queues the runs left `queued` by the one before it, and `fluksio retry <id>` / `retry --group <sweep>` submits an interrupted run again with the same inputs and group, recorded through Run.parent_id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9BoNGq6V9MdRWAte7JBuC
This commit is contained in:
@@ -799,6 +799,28 @@ class RunService:
|
||||
target=self._keep_leases, name="run-leases", daemon=True
|
||||
)
|
||||
self._keeper.start()
|
||||
self._requeue_queued()
|
||||
|
||||
def _requeue_queued(self) -> None:
|
||||
"""Wake the engine for runs an earlier process was holding.
|
||||
|
||||
A row is journaled `queued` and then a work item is added, and the two
|
||||
are not one transaction — an in-memory queue loses the item with the
|
||||
process, and a stream item nobody claimed is nobody's. The row is the
|
||||
record, so this reads it back. Nothing is re-executed by it: claiming
|
||||
is a compare-and-set out of `queued`, so an item that did survive is
|
||||
claimed once whatever this adds.
|
||||
"""
|
||||
with Session(db_engine) as session:
|
||||
waiting = session.exec(
|
||||
select(Run.id, Run.flow).where(col(Run.status) == "queued")
|
||||
).all()
|
||||
for run_id, flow in waiting:
|
||||
self.queue.add(WorkItem(kind="run", node="", flow=flow, run_id=run_id))
|
||||
if waiting:
|
||||
logger.info(
|
||||
"Requeued %d run(s) left queued by an earlier engine", len(waiting)
|
||||
)
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stop.set()
|
||||
@@ -828,6 +850,7 @@ class RunService:
|
||||
draft: bool = False,
|
||||
no_cache: bool = False,
|
||||
idempotency_key: str | None = None,
|
||||
parent_id: str | None = None,
|
||||
) -> Run:
|
||||
"""Journal a run and wake an engine up for it. Never blocks on it."""
|
||||
# Before anything else, including reading the flow: a caller retrying a
|
||||
@@ -889,6 +912,7 @@ class RunService:
|
||||
created_at=datetime.now(UTC),
|
||||
actor=actor,
|
||||
idempotency_key=idempotency_key,
|
||||
parent_id=parent_id,
|
||||
)
|
||||
with Session(db_engine) as session:
|
||||
session.add(run)
|
||||
@@ -940,6 +964,34 @@ class RunService:
|
||||
self.controller.remote.cancel_run(run_id)
|
||||
return True
|
||||
|
||||
def retry(self, run_id: str, actor: str = "") -> Run:
|
||||
"""Submit the same run again, as a run of its own.
|
||||
|
||||
What an engine that stopped mid-run leaves behind is a run marked
|
||||
`abandoned`, and re-issuing the whole sweep to recover one config is
|
||||
the blunt way back. This is the sharp one: the same flow, the same
|
||||
inputs, the same group, so a sweep is completed rather than repeated.
|
||||
The stage cache is what makes it cheap — the nodes that finished are
|
||||
restored rather than run again.
|
||||
"""
|
||||
with Session(db_engine) as session:
|
||||
run = session.get(Run, run_id)
|
||||
if run is None:
|
||||
raise RunRejected(f"No run {run_id}")
|
||||
if run.status in ("queued", "running"):
|
||||
raise RunRejected("That run has not finished; cancel it first")
|
||||
source = run.model_copy()
|
||||
return self.submit(
|
||||
source.flow,
|
||||
params=dict(source.params),
|
||||
seed=source.seed,
|
||||
group_id=source.group_id,
|
||||
cause="retry",
|
||||
actor=actor,
|
||||
no_cache=source.no_cache,
|
||||
parent_id=source.id,
|
||||
)
|
||||
|
||||
def _flow_of(self, run_id: str) -> str | None:
|
||||
with Session(db_engine) as session:
|
||||
run = session.get(Run, run_id)
|
||||
|
||||
Reference in New Issue
Block a user