Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Close the nine open SDK tasks: one engine per directory, a tabbed dashboard, re-pairing, run recovery
Docs / docs (push) Successful in 27s
Details
Playwright Tests / test-playwright (1, 2) (push) Failing after 17s
Details
Playwright Tests / test-playwright (2, 2) (push) Failing after 12s
Details
pre-commit / pre-commit (push) Failing after 1m59s
Details
Test Backend / test-backend (push) Failing after 2m30s
Details
Compose Smoke Test / test-compose (push) Failing after 13s
Details
Playwright Tests / merge-reports (push) Failing after 2m19s
Details

Browse Source 
serve: refuse a second engine for one data directory whatever port it was
asked for, using the pidfile and a token this directory signed. The check
runs before the database is touched and before the credential is written,
which is what left every later CLI call pointing at a dead port.

The terminal dashboard is three tabs (Overview, Runs, Logs) with the toolbar
following the focused pane, the engine's output goes to serve.log rather than
down a pipe, and closing the screen stops both reader threads so the prompt
comes back. It adopts a running engine on every start, so stop/start and
restart work on one it did not start, and a stop waits for the process to be
gone before the next start. Enrolment reports itself in the modal.

enroll: a new claim code replaces the pairing instead of being refused. The
code is redeemed before anything is written, mappings to a portal being left
are cleared, and a running engine redials when the stored enrolment changes.

runs: an engine re-queues the runs left `queued` by the one before it, and
`fluksio retry <id>` / `retry --group <sweep>` submits an interrupted run
again with the same inputs and group, recorded through Run.parent_id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9BoNGq6V9MdRWAte7JBuC
This commit is contained in:
stroblme
2026-08-31 17:37:13 +02:00
co-authored by Claude Opus 5
parent bdad6d7fc2
commit 8a94bf10d7
14 changed files with 873 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/fluksio/tui/app.py
+302 -92
View File
@@ -12,14 +12,26 @@ import signal
import subprocess
import sys
import threading
import time
from pathlib import Path
from typing import Any
from textual import work
from textual.app import App, ComposeResult
from textual.containers import Horizontal, Vertical
from textual.containers import Horizontal, Vertical, VerticalScroll
from textual.screen import ModalScreen
from textual.widgets import Button, DataTable, Footer, Header, Input, Label, RichLog
from textual.widgets import (
Button,
DataTable,
Footer,
Header,
Input,
Label,
RichLog,
Static,
TabbedContent,
TabPane,
)
from fluksio.cli import (
DEFAULT_PORT,
@@ -43,6 +55,9 @@ STARTUP_TRIES = 60
#: split the browser makes for the same reason.
MAX_PICKED = 20
#: How many runs the table holds, now that it owns a whole tab.
RUNS_SHOWN = 50
#: What the dashboard subscribes to. `message_value` is most of what the bus
#: carries and none of what this screen draws.
KINDS = frozenset(
@@ -53,6 +68,33 @@ KINDS = frozenset(
#: runs should cost one read of the engine, not twenty.
COALESCE_S = 0.25
#: Where the engine's own output goes, beside the data it is serving. A file
#: rather than a pipe because the screen is meant to be closed while the
#: engine keeps running: a pipe whose reader has gone breaks the next write,
#: and a node's `print` is one of those writes. It also means the log of an
#: engine this screen only *adopted* can be read here.
LOG_NAME = "serve.log"
#: How much of it to read back when the screen opens.
LOG_TAIL_BYTES = 64 * 1024
#: What it is truncated to when this screen starts an engine of its own.
#: ponytail: a size check, not rotation — add rotation when somebody wants
#: yesterday's log.
LOG_KEEP_BYTES = 5 * 1024 * 1024
#: How long the tail waits when the file has nothing new. Also how long
#: closing the screen waits for that thread.
LOG_POLL_S = 0.25
#: How long a stop waits for the engine to be gone before killing it, and how
#: long it waits for an adopted one, which it can only ask.
STOP_WAIT_S = 10.0
#: The width of the runs table's five fixed columns, padding included, which
#: is what is left over for the inputs.
FIXED_COLUMNS = 54
def child_argv(argv: list[str]) -> list[str]:
"""The same command, told to serve without a dashboard of its own.
@@ -64,27 +106,98 @@ def child_argv(argv: list[str]) -> list[str]:
return [sys.executable, "-m", "fluksio.cli", *passed, "--plain"]
class Enroll(ModalScreen[tuple[str, str] | None]):
"""The claim code a portal minted, and which portal minted it."""
class Enroll(ModalScreen[None]):
"""The claim code a portal minted, and which portal minted it.
The work happens here rather than back on the dashboard: enrolment is a
round trip to the portal, and the person who pressed the button is looking
at this modal while it happens.
"""
BINDINGS = [("escape", "dismiss(None)", "cancel")]
def __init__(self, data_dir: Path) -> None:
super().__init__()
self.data_dir = data_dir
def compose(self) -> ComposeResult:
with Vertical(id="enroll"):
yield Label("Pair this instance with a portal")
yield Input(placeholder="claim code", id="code")
yield Input(value=DEFAULT_PORTAL, id="portal")
yield Label("", id="note")
with Horizontal():
yield Button("Enroll", variant="primary", id="go")
yield Button("Cancel", id="cancel")
def on_mount(self) -> None:
self.query_one("#code", Input).focus()
def on_button_pressed(self, event: Button.Pressed) -> None:
if event.button.id == "cancel":
self.dismiss(None)
return
self.submit()
def on_input_submitted(self, event: Input.Submitted) -> None:
self.submit()
def submit(self) -> None:
code = self.query_one("#code", Input).value.strip()
portal = self.query_one("#portal", Input).value.strip()
self.dismiss((code, portal or DEFAULT_PORTAL) if code else None)
portal = self.query_one("#portal", Input).value.strip() or DEFAULT_PORTAL
if not code:
self.note("A claim code is what pairs this instance.")
return
self.busy(True)
self.note(f"Enrolling with {portal}…")
self.enroll(code, portal)
def note(self, message: str) -> None:
self.query_one("#note", Label).update(message)
def busy(self, working: bool) -> None:
for one in self.query(Input):
one.disabled = working
for button in self.query(Button):
button.disabled = working
@work(thread=True)
def enroll(self, code: str, portal: str) -> None:
"""`fluksio enroll`, as its own process for the same reason serve is.
A running engine picks the configuration up on its own; this screen
only has to report what the command said.
"""
done = subprocess.run( # noqa: S603
[
sys.executable,
"-m",
"fluksio.cli",
"enroll",
code,
"--portal",
portal,
"--data-dir",
str(self.data_dir),
],
capture_output=True,
text=True,
)
self.app.call_from_thread(
self.finished, done.returncode, done.stdout + done.stderr
)
def finished(self, code: int, output: str) -> None:
lines = [line for line in output.splitlines() if line.strip()]
for line in lines:
self.app.note(line) # type: ignore[attr-defined]
said = lines[-1] if lines else ""
if code == 0:
self.app.notify(said or "Connected to the portal.")
self.dismiss(None)
return
self.busy(False)
self.note(said or "Enrolment failed; the log has what it said.")
class Artifacts(ModalScreen[None]):
@@ -168,26 +281,41 @@ class Artifacts(ModalScreen[None]):
self.app.call_from_thread(self.note, f"Wrote {out.resolve()}")
class RunsTable(DataTable[str]):
"""The runs, and the keys that only mean anything while they are in front.
Textual resolves a binding from the focused widget outwards, so hanging
them here is what makes the footer read as run tools on this tab and as
engine tools on the others.
"""
BINDINGS = [
("space", "app.pick", "pick for comparison"),
("c", "app.cancel_run", "cancel run"),
("a", "app.artifacts", "artifacts"),
]
class ServeApp(App[int]):
"""One screen: how the engine is, what has run, and what it is saying."""
"""Three tabs: how the engine is, what has run, and what it is saying."""
CSS = """
#status { height: auto; padding: 0 1; }
#runs { height: 2fr; }
#log { height: 1fr; border-top: solid $panel; }
TabbedContent { height: 1fr; }
TabPane { height: 1fr; padding: 0; }
#overview { padding: 0 1; }
#enroll { width: 60; height: auto; padding: 1 2; background: $surface; }
#enroll #note { color: $text-muted; height: auto; }
#artifacts { width: 80; height: auto; padding: 1 2; background: $surface; }
#artifacts DataTable { height: auto; max-height: 14; }
"""
BINDINGS = [
("q", "quit", "quit (engine keeps running)"),
("1", "tab('overview-tab')", "overview"),
("2", "tab('runs-tab')", "runs"),
("3", "tab('logs-tab')", "logs"),
("s", "stop_start", "stop/start"),
("r", "restart", "restart"),
("c", "cancel_run", "cancel run"),
("space", "pick", "pick for comparison"),
("enter", "compare", "compare"),
("a", "artifacts", "artifacts"),
("e", "enroll", "enroll"),
]
@@ -195,7 +323,7 @@ class ServeApp(App[int]):
super().__init__()
self.args = args
self.data_dir: Path = _data_dir(args.data_dir, args.shared)
self.child: subprocess.Popen[str] | None = None
self.child: subprocess.Popen[bytes] | None = None
#: The pid of an engine this dashboard did not start. Only ever one
#: whose data directory is this one — the probe is what proves it.
self.adopted: int | None = None
@@ -207,43 +335,92 @@ class ServeApp(App[int]):
#: cascade of events costs one read of the engine rather than one each.
self.stirred = False
self.stop_stream = threading.Event()
self.stop_log = threading.Event()
#: The credential's mtime when the engine under this screen was
#: started, so its own is told from the one before it.
self.config_stamp = 0
# -- layout ---------------------------------------------------------------
def compose(self) -> ComposeResult:
yield Header()
yield RichLog(id="status", markup=True, wrap=True)
table: DataTable[str] = DataTable(id="runs", cursor_type="row")
yield table
yield RichLog(id="log", markup=False, highlight=False, max_lines=2000)
with TabbedContent(initial="overview-tab"):
with TabPane("Overview", id="overview-tab"):
with VerticalScroll():
yield Static(id="overview")
with TabPane("Runs", id="runs-tab"):
yield RunsTable(id="runs", cursor_type="row")
with TabPane("Logs", id="logs-tab"):
yield RichLog(id="log", markup=False, highlight=False, max_lines=5000)
yield Footer()
def on_mount(self) -> None:
self.title = f"fluksio — {self.data_dir}"
table = self.query_one("#runs", DataTable)
table.add_columns(" ", "run", "status", "flow", "took", "params")
table.focus()
self.start_engine(first=True)
table = self.query_one("#runs", RunsTable)
table.add_column(" ", width=1)
table.add_column("run", width=8)
table.add_column("status", width=9)
table.add_column("flow", width=16)
table.add_column("took", width=8)
table.add_column("params")
self.tail_log()
self.start_engine()
# The socket is what makes a run that starts and finishes between two
# ticks visible; this stays as the heartbeat for what is not on the bus.
self.set_interval(WATCH_INTERVAL_S, self.refresh_panels)
self.set_interval(COALESCE_S, self.drain)
def action_tab(self, tab: str) -> None:
self.query_one(TabbedContent).active = tab
def on_tabbed_content_tab_activated(
self, event: TabbedContent.TabActivated
) -> None:
"""Focus what the tab is about, so its keys are the ones in the footer."""
focusable = event.pane.query("RunsTable, RichLog, VerticalScroll")
if focusable:
focusable.first().focus()
async def action_quit(self) -> None:
"""Close the screen and leave the engine running.
The threads are told first: both are worker threads on the loop's own
executor, which is joined while the loop closes — so a reader still
blocked on the log or the socket would hold the process after the
screen is gone, which is what `q` used to do.
"""
self.stop_log.set()
self.stop_stream.set()
self.exit(0)
def on_unmount(self) -> None:
self.stop_log.set()
self.stop_stream.set()
# -- the engine under the screen ------------------------------------------
def note(self, message: str) -> None:
self.query_one("#log", RichLog).write(message)
def start_engine(self, first: bool = False) -> None:
def say(self, message: str) -> None:
"""`note`, from a worker thread."""
self.call_from_thread(self.note, message)
def log_path(self) -> Path:
return self.data_dir / LOG_NAME
def start_engine(self) -> None:
"""Adopt whatever is already serving this directory, or start one."""
host = self.args.host
reachable = "127.0.0.1" if host in ("0.0.0.0", "::", "") else host
wanted = self.args.port or DEFAULT_PORT
# Where this directory's engine last said it was, which is not always
# where it was asked to be: a taken port moves.
running = read_pidfile(self.data_dir)
wanted = running["port"] if running else (self.args.port or DEFAULT_PORT)
url = f"http://{reachable}:{wanted}"
who = probe_engine(url, _token_for(self.data_dir)) if first else "other"
who = probe_engine(url, _token_for(self.data_dir))
if who == "ours":
running = read_pidfile(self.data_dir)
self.adopted = running["pid"] if running else None
self.url = url
named = f" (pid {self.adopted})" if self.adopted else ""
@@ -255,25 +432,47 @@ class ServeApp(App[int]):
if who == "foreign":
self.note(f"Port {wanted} holds another instance's Fluksio.")
self.child = subprocess.Popen( # noqa: S603
child_argv(sys.argv[1:]),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
bufsize=1,
env={**os.environ, "PYTHONUNBUFFERED": "1"},
)
self.tail_child(self.child)
# What the credential looked like before the child wrote its own, so
# the wait below cannot mistake the last engine's url for this one's.
self.config_stamp = self._config_stamp()
path = self.log_path()
if path.exists() and path.stat().st_size > LOG_KEEP_BYTES:
path.write_text("")
handle = path.open("ab", buffering=0)
try:
self.child = subprocess.Popen( # noqa: S603
child_argv(sys.argv[1:]),
stdout=handle,
stderr=subprocess.STDOUT,
env={**os.environ, "PYTHONUNBUFFERED": "1"},
)
finally:
# The child holds a descriptor of its own; this one is what would
# otherwise keep the file open for as long as the screen lives.
handle.close()
self.await_engine()
@work(thread=True, exclusive=False)
def tail_child(self, child: subprocess.Popen[str]) -> None:
@work(thread=True, exclusive=True, group="log")
def tail_log(self) -> None:
"""The engine's own output, which is why a second terminal was needed."""
if child.stdout is None:
return
for line in child.stdout:
self.call_from_thread(self.note, line.rstrip())
self.call_from_thread(self.note, f"The engine stopped ({child.wait()}).")
path = self.log_path()
while not self.stop_log.is_set() and self.is_running:
try:
with path.open("r", errors="replace") as handle:
handle.seek(max(0, path.stat().st_size - LOG_TAIL_BYTES))
if handle.tell():
handle.readline() # whatever line the seek landed in
while not self.stop_log.is_set() and self.is_running:
line = handle.readline()
if line:
self.call_from_thread(self.note, line.rstrip())
continue
if path.stat().st_size < handle.tell():
break # truncated under us; read it from the top
time.sleep(LOG_POLL_S)
except OSError:
# Not written yet, or gone. Either way it may appear.
time.sleep(LOG_POLL_S)
@work(thread=True, exclusive=True, group="startup")
def await_engine(self) -> None:
@@ -282,8 +481,6 @@ class ServeApp(App[int]):
The port is the child's to choose — it moves off a taken one — and
`client.json` is where it says which it took.
"""
import time
for _ in range(STARTUP_TRIES):
if self.child is not None and self.child.poll() is not None:
return
@@ -291,6 +488,11 @@ class ServeApp(App[int]):
stored = json.loads(config_path(self.data_dir).read_text())
except (OSError, ValueError):
stored = {}
if self._config_stamp() == self.config_stamp:
# Still the one the last engine left, which names a port this
# one may not have taken.
time.sleep(0.5)
continue
if stored.get("url") and stored.get("token"):
self.url = str(stored["url"])
self.call_from_thread(self.connect)
@@ -313,23 +515,56 @@ class ServeApp(App[int]):
self.watch_engine(self.client.url, self.client.token)
self.refresh_panels()
def _config_stamp(self) -> int:
"""When the credential was last written, or 0 if it never was."""
try:
return config_path(self.data_dir).stat().st_mtime_ns
except OSError:
return 0
def engine_pid(self) -> int | None:
return self.child.pid if self.child is not None else self.adopted
def stop_engine(self) -> None:
"""Stop the engine and wait for it to be gone. **From a thread.**
Waiting is what makes `r` work: a new engine started while the old one
still holds the port would move off it, and the screen would end up
watching one engine while the client talks to another. An engine
draining its flows takes as long as it takes, which is why this is not
something to do on the loop that draws.
"""
pid = self.engine_pid()
if self.child is not None:
self.note(f"Stopping the engine (pid {self.child.pid}).")
self.say(f"Stopping the engine (pid {self.child.pid}).")
self.child.terminate()
try:
self.child.wait(timeout=STOP_WAIT_S)
except subprocess.TimeoutExpired:
self.child.kill()
self.child = None
elif self.adopted is not None:
self.note(f"Stopping the adopted engine (pid {self.adopted}).")
self.say(f"Stopping the adopted engine (pid {self.adopted}).")
try:
os.kill(self.adopted, signal.SIGTERM)
except OSError as exc:
self.note(f"Could not stop it: {exc}")
self.say(f"Could not stop it: {exc}")
self._await_exit(self.adopted)
self.adopted = None
self.stop_stream.set()
self.client = None
if pid is not None:
self.say("The engine stopped.")
def _await_exit(self, pid: int) -> None:
deadline = time.time() + STOP_WAIT_S
while time.time() < deadline:
try:
os.kill(pid, 0)
except OSError:
return
time.sleep(0.2)
self.say(f"pid {pid} is still running.")
# -- the engine's own events ----------------------------------------------
@@ -371,28 +606,30 @@ class ServeApp(App[int]):
return
try:
screen = _status_screen(client)
rows = client.runs(limit=20)
rows = client.runs(limit=RUNS_SHOWN)
except Exception as exc:
self.call_from_thread(self.show_offline, exc)
return
self.call_from_thread(self.show, screen, rows)
def show(self, screen: Any, rows: list[dict[str, Any]]) -> None:
status = self.query_one("#status", RichLog)
status.clear()
status.write(screen)
table = self.query_one("#runs", DataTable)
self.query_one("#overview", Static).update(screen)
table = self.query_one("#runs", RunsTable)
cursor = table.cursor_row
# Whatever the fixed columns do not use, so a wide terminal shows the
# inputs rather than a stripe of empty table.
room = max(20, table.size.width - FIXED_COLUMNS)
table.clear()
for row in rows:
run_id = str(row.get("id", ""))
params = json.dumps(row.get("params") or {})
table.add_row(
"·" if run_id in self.picked else " ",
run_id[-8:],
str(row.get("status", "")),
str(row.get("flow", "")),
_dur(row.get("duration_ms")),
json.dumps(row.get("params") or {})[:60],
params.ljust(room) if len(params) <= room else params[: room - 1] + "…",
key=run_id,
)
# The rows are rewritten wholesale every refresh, and a cursor that
@@ -402,22 +639,25 @@ class ServeApp(App[int]):
table.move_cursor(row=cursor)
def show_offline(self, exc: Exception) -> None:
status = self.query_one("#status", RichLog)
status.clear()
which = "starting" if self.engine_pid() is not None else "not running"
status.write(f"[yellow]The engine is {which}.[/] ({type(exc).__name__})")
self.query_one("#overview", Static).update(
f"[yellow]The engine is {which}.[/] ({type(exc).__name__})"
)
# -- keys -----------------------------------------------------------------
@work(thread=True, group="engine")
def action_stop_start(self) -> None:
if self.engine_pid() is not None:
self.stop_engine()
else:
self.start_engine()
self.call_from_thread(self.start_engine)
@work(thread=True, group="engine")
def action_restart(self) -> None:
"""Stop, wait for the port to come back, start."""
self.stop_engine()
self.start_engine()
self.call_from_thread(self.start_engine)
def cursor_run(self) -> str:
"""The whole id of the run the cursor is on.
@@ -425,7 +665,7 @@ class ServeApp(App[int]):
The cell holds its tail, which is what fits in a column; the row's key
is the whole of it, which is what the engine is asked about.
"""
table = self.query_one("#runs", DataTable)
table = self.query_one("#runs", RunsTable)
if self.client is None or not table.row_count:
return ""
key = table.coordinate_to_cell_key(table.cursor_coordinate).row_key
@@ -482,38 +722,7 @@ class ServeApp(App[int]):
self.call_from_thread(self.refresh_panels)
def action_enroll(self) -> None:
self.push_screen(Enroll(), self.enrolled)
def enrolled(self, answer: tuple[str, str] | None) -> None:
if answer is None:
return
code, portal = answer
self.run_enroll(code, portal)
@work(thread=True)
def run_enroll(self, code: str, portal: str) -> None:
"""`fluksio enroll`, as its own process for the same reason serve is.
A running engine picks the configuration up on its own; this screen
only has to report what the command said.
"""
done = subprocess.run( # noqa: S603
[
sys.executable,
"-m",
"fluksio.cli",
"enroll",
code,
"--portal",
portal,
"--data-dir",
str(self.data_dir),
],
capture_output=True,
text=True,
)
for line in (done.stdout + done.stderr).splitlines():
self.call_from_thread(self.note, line)
self.push_screen(Enroll(self.data_dir))
def run_tui(args: argparse.Namespace) -> int:
@@ -524,4 +733,5 @@ def run_tui(args: argparse.Namespace) -> int:
if pid is not None:
print(f"The engine is still running: pid {pid} at {url or 'its port'}.")
print(f" fluksio serve reattaches to it; kill {pid} stops it.")
print(f" Its log: {app.log_path()}")
return 0