Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bound what the API accepts, and close the holes the audit found

Browse Source 
**SQLite is the database, and now says so.** `metric_minute` and every run
table are written with `sqlalchemy.dialects.sqlite.insert(...)
.on_conflict_do_update` and with `max(a, b)`, neither of which another
dialect has — so pointing `DATABASE_URL` at Postgres migrated cleanly,
served, logged in, and then lost every observability flush into the
collector's hold buffer and failed every run. It refuses at startup
instead. (The Postgres in the compose stack is Umami's; the engine's own
database has been a file beside the flows since 2026-08-21.)

**Every integer query parameter is bounded.** The caps were written as
`min(limit, 500)`, which a negative walks straight through — `?limit=-1`
compiles to `LIMIT -1` and SQLite returns the whole table. Ten signatures,
now `Query(ge=…, le=…)`. `hours=0` still means an hour, which
`_window_hours` was already deliberate about.

**Exports are capped at 10 000 runs** and say so with `X-Truncated`. The
filters bounded a sensible request and nothing bounded an unfiltered one,
which read every row into memory before a byte was streamed. `_series`
resolves cached curves in two queries rather than a `Run` lookup and a
`RunMetric` query per restored node — a comparison of twenty runs was
calling that twenty times over.

**`PUT /artifacts` has a size limit** (`MAX_ARTIFACT_BYTES`, 2 GiB, 0 to
disable), checked against `Content-Length` and again against the stream for
a chunked body, and its writes moved off the event loop.

**`/observability/timeseries` takes `since`/`until`**, the same window
`/runs` and `/events` take, capped at 2000 points — `hours=720&bucket_s=60`
was 43 200 of them in one array. It is also what a dragged chart needs to
re-fetch at its own resolution rather than magnifying buckets it has.

**Composite indexes** for the three list screens: `run(flow, created_at)`
and `(status, created_at)`, `flow_run(flow, started_at)`,
`engine_event(type, ts)`. Every index was single-column, so SQLite picked
one and sorted the rest by hand. Verified against a copy of a live database
(250k `flow_run` rows): the planner takes all four.

**Redis clients have socket timeouts.** A Redis that stops answering
without closing the connection hung the caller until the kernel gave up —
including `/utils/health/`, whose job is to notice.

**The panels file is written under one lock.** `save_panels` and
`unpair_panel` are both read-modify-write, and a save that read before an
unpair wrote put the old nonce back — silently un-revoking a screen that
had just been unpaired. The nonce carry-forward was written to make that
impossible; the gap between its read and its write is where it happened.

**Startup releases what it acquired.** Everything past `event_bus.bind`
registers how to close itself and the `finally` walks that list backwards;
a failure part-way through used to reach none of the shutdown steps and
leave the worker pool's subprocesses and every background task behind —
under `--reload`, once per bad edit. `modules.reconcile` moved into the
background: `uv` gets five minutes twice over, the healthcheck allows
eighty seconds, and the autoheal restarted the container before it could
finish installing.

`delete_run` takes SQLite's write lock up front (`core.db.writing`) rather
than upgrading a deferred transaction and losing to whichever flush
committed in between. `modules.sync` is serialised — two applies mutated
one venv at once. The proxied-call and stream dicts are bounded, and a
reused id cancels its predecessor instead of dropping the reference.

Security, found in passing and small enough to fix here:

- **`/secrets/` required only a signed-in user.** The names alone say what
  this installation talks to, and `PUT /{name}` takes any name, so any
  account could overwrite the credential a flow authenticates with.
  Superuser now — which `/search` already assumed and said so.
- **`POST /login/access-token` had no rate limit.** Argon2 is deliberately
  expensive and the route is unauthenticated and runs in the shared
  threadpool. Ten *failed* attempts per address per five minutes; a
  successful sign-in spends nothing.
- **a password reset link worked repeatedly for 48 hours.** The token now
  carries a digest of the password hash it was minted against, so it stops
  verifying once it has set one. No table of spent tokens needed.
- **enrolment accepted `http://`**, sending the claim code and then this
  installation's credential in clear. https, or a local address.
- the rate limiter read `request.client.host`, which behind Traefik is the
  proxy — so every per-address limit was one global bucket and one caller
  could lock out everyone. It reads the forwarded address, and its
  bucket table is capped rather than growing one key per address forever.
- SMTP has a timeout and sends after the response, so an unreachable mail
  host cannot pin a threadpool worker, and a reply's timing no longer says
  whether the address exists.

Test suite: engine-written rows are cleared between modules. A `FlowRun`
left `running` by one module turned up in another's query. Per-test
rollback is not available here — the module-scoped `client` runs the real
lifespan and its collector and run service write through sessions of their
own — so this bounds it where the writes come from. Three consecutive
green runs, orders randomised.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6hPWS6YEbT1P8LxhhFb2T
This commit is contained in:
stroblme
2026-08-29 20:40:05 +02:00
co-authored by Claude Opus 5
parent 1069247085
commit 57eace2226
24 changed files with 821 additions and 260 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/fluksio/main.py
+186 -144
View File
@@ -1,7 +1,8 @@
import asyncio
import contextlib
import inspect
import logging
from collections.abc import AsyncIterator, Coroutine
from collections.abc import AsyncIterator, Callable, Coroutine
from contextlib import AbstractAsyncContextManager, asynccontextmanager
from typing import Any
@@ -129,109 +130,32 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
# pays a version check for it and nothing else.
await run_in_threadpool(prepare, db_engine)
event_bus.bind(asyncio.get_running_loop())
# Node code is user code, and `print` is how it says things.
logs.install()
init_secrets(settings.SECRETS_FILE, settings.SECRET_KEY)
# Connectors register their node types before any flow is built with them.
load_plugins()
alerts = AlertManager(event_bus, config=read_alerts_config())
execution = ExecutionService(
queue=_work_queue(),
max_workers=settings.FLOW_MAX_WORKERS,
events=event_bus,
max_cascades=settings.FLOW_MAX_CASCADES,
)
store = FlowStore(settings.FLOWS_DIR)
# The packages node code imports, before anything tries to import them.
await run_in_threadpool(modules.reconcile, store)
# Beside the flows rather than in them: an artifact is what a run produced,
# not something anyone wrote, so it has no business in the git repository.
artifacts = ArtifactStore(settings.FLOWS_DIR.parent / "artifacts")
app.state.artifact_store = artifacts
accountant = ResourceAccountant(cpus=settings.FLOW_CPUS, gpus=settings.FLOW_GPUS)
app.state.resources = accountant
# Every machine a node could run on: this one, and whatever attaches.
placer = Placer(local=accountant, events=event_bus)
# Where more machines can be asked for when nothing attached will do.
placer.provisioners = load_provisioners(settings.PROVISIONERS_FILE, event_bus)
app.state.placer = placer
pool = PythonWorkerPool(
python=modules.venv_python(),
size=settings.FLOW_MAX_WORKERS,
events=event_bus,
# A worker in this container writes to the store directly; a remote one
# is given a URL instead. Node code calls the same two functions.
env={
ARTIFACT_DIR_ENV: str(artifacts.root),
# Every slot can be busy at once, so a worker left to size its own
# thread pool to the machine means as many processes as there are
# slots, each believing it has the whole of it. A node that says
# what it needs overrides this; one that says nothing gets a share.
**fair_share_env(accountant.cpus, settings.FLOW_MAX_WORKERS),
},
)
pool.start()
app.state.worker_pool = pool
# Assigned rather than passed both ways: the hub tells the placer when a
# machine comes or goes, and the placer needs the hub to know what is there.
worker_hub = RemoteWorkerHub(on_change=placer.wake)
placer.hub = worker_hub
app.state.worker_hub = worker_hub
controller = FlowController(
store=store,
state=_state_backend(),
events=event_bus,
max_workers=settings.FLOW_MAX_WORKERS,
fastapi_app=app,
execution=execution,
alerts=alerts,
workers=pool,
remote=worker_hub,
resources=accountant,
placer=placer,
)
app.state.flow_controller = controller
# A "dashboard" alert channel puts its alert into the graph. Bound here
# rather than passed in: the manager is built before the controller is.
alerts.publish = lambda name, value: controller.publish_message(
name, value, ValueSource(kind="api", id="alerts", label="Alerts")
)
dashboards = DashboardStore(controller.store)
app.state.dashboard_store = dashboards
controller.dashboards = dashboards
# Charts need a deeper series than the default; tell the engine
# before it starts recording.
controller.set_history_limits(dashboards.history_requirements())
# Runs read from a stream of their own: a burst of sweep runs must not
# stand between the automations and their work, and a run that takes an
# hour must not be judged by the cascade reaper's timings.
run_service = RunService(
controller=controller,
queue=_work_queue("run"),
state_factory=_run_state,
artifacts=artifacts,
parallel=settings.FLOW_MAX_RUNS,
)
app.state.run_service = run_service
# Said out loud because it is the only way to tell that a settings file was
# read at all — the numbers are what someone raising them is looking for.
logger.info(
"Engine limits: workers=%s cascades=%s runs=%s",
settings.FLOW_MAX_WORKERS,
execution.max_cascades,
run_service.parallel,
)
watchdog = LoopWatchdog(event_bus)
app.state.watchdog = watchdog
# Everything acquired below registers how to release it, and the `finally`
# at the end walks the list backwards. It used to be a fixed block of
# shutdown steps *after* every acquisition — so a failure part-way through
# startup reached none of them and left the worker pool's subprocesses and
# every background task behind. Under `--reload` that is once per bad edit.
started: list[Callable[[], Any]] = []
def _background(coro: Coroutine[Any, Any, None], name: str) -> asyncio.Task[None]:
"""Start a long-lived task that says something if it ever stops.
async def _release() -> None:
for close in reversed(started):
try:
result = close()
if inspect.isawaitable(result):
await result
except Exception:
logger.exception("Error while shutting down")
Each of these loops catches its own exceptions *inside* the loop, so
one raised anywhere else simply ended the task — an engine that went
on serving with no metrics, no alerts or no artifact sweep and nothing
anywhere saying so.
def _background(
coro: Coroutine[Any, Any, None], name: str, once: bool = False
) -> asyncio.Task[None]:
"""Start a background task that says something if it ever stops.
The loops catch their own exceptions *inside* the loop, so one raised
anywhere else simply ended the task — an engine that went on serving
with no metrics, no alerts or no artifact sweep and nothing anywhere
saying so. ``once`` is for a task that is meant to finish.
"""
task = asyncio.create_task(coro, name=name)
@@ -240,7 +164,8 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
return
exc = done.exception()
if exc is None:
logger.warning("Background task '%s' stopped on its own", name)
if not once:
logger.warning("Background task '%s' stopped on its own", name)
return
logger.error("Background task '%s' died: %s", name, exc, exc_info=exc)
event_bus.publish(
@@ -248,56 +173,173 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
)
task.add_done_callback(_finished)
started.append(task.cancel)
return task
watchdog_task = _background(watchdog.run(), "loop-watchdog")
alerts_task = _background(alerts.run(), "alert-manager")
metrics_task = _background(MetricsCollector(event_bus).run(), "metrics-collector")
gc_task = _background(_sweep_artifacts(artifacts, controller), "artifact-gc")
await controller.start()
run_service.start()
# Optional, and off unless someone enrolled this installation: the
# connector dials the portal, nothing dials in.
cloud_task: asyncio.Task[None] | None = None
app.state.cloud_connector = None
app.state.cloud_task = None
from fluksio.cloud import connector as cloud_connector
if cloud_config.exists():
cloud_connector.start(app)
cloud_task = app.state.cloud_task
# Watched whether or not one exists now: enrolling from the CLI writes the
# config from another process entirely, and an engine already serving
# should pick it up rather than need restarting.
enrol_task = _background(
cloud_connector.watch_enrolment(app), "cloud-enrolment-watch"
)
# The startup itself is inside the try, so a step that raises part-way
# through still reaches the release below.
try:
# Node code is user code, and `print` is how it says things.
logs.install()
init_secrets(settings.SECRETS_FILE, settings.SECRET_KEY)
# Connectors register their node types before any flow is built with them.
load_plugins()
alerts = AlertManager(event_bus, config=read_alerts_config())
execution = ExecutionService(
queue=_work_queue(),
max_workers=settings.FLOW_MAX_WORKERS,
events=event_bus,
max_cascades=settings.FLOW_MAX_CASCADES,
)
store = FlowStore(settings.FLOWS_DIR)
# The packages node code imports. In the background, because a first
# install of anything substantial takes minutes and `uv` is given five
# of them twice over — held here, the container never answers its
# healthcheck, autoheal restarts it, and it never finishes installing at
# all. A node whose import is not there yet fails and is retried, which
# is recoverable; a restart loop is not.
_background(
run_in_threadpool(modules.reconcile, store), "module-reconcile", once=True
)
# Beside the flows rather than in them: an artifact is what a run produced,
# not something anyone wrote, so it has no business in the git repository.
artifacts = ArtifactStore(settings.FLOWS_DIR.parent / "artifacts")
app.state.artifact_store = artifacts
accountant = ResourceAccountant(
cpus=settings.FLOW_CPUS, gpus=settings.FLOW_GPUS
)
app.state.resources = accountant
# Every machine a node could run on: this one, and whatever attaches.
placer = Placer(local=accountant, events=event_bus)
# Where more machines can be asked for when nothing attached will do.
placer.provisioners = load_provisioners(settings.PROVISIONERS_FILE, event_bus)
app.state.placer = placer
pool = PythonWorkerPool(
python=modules.venv_python(),
size=settings.FLOW_MAX_WORKERS,
events=event_bus,
# A worker in this container writes to the store directly; a remote one
# is given a URL instead. Node code calls the same two functions.
env={
ARTIFACT_DIR_ENV: str(artifacts.root),
# Every slot can be busy at once, so a worker left to size its own
# thread pool to the machine means as many processes as there are
# slots, each believing it has the whole of it. A node that says
# what it needs overrides this; one that says nothing gets a share.
**fair_share_env(accountant.cpus, settings.FLOW_MAX_WORKERS),
},
)
pool.start()
# On a thread: stopping a worker waits up to five seconds on each child,
# and with a pool per declared environment that is a shutdown the event
# loop should not be holding.
started.append(lambda: run_in_threadpool(pool.stop))
app.state.worker_pool = pool
# Assigned rather than passed both ways: the hub tells the placer when a
# machine comes or goes, and the placer needs the hub to know what is there.
worker_hub = RemoteWorkerHub(on_change=placer.wake)
placer.hub = worker_hub
app.state.worker_hub = worker_hub
controller = FlowController(
store=store,
state=_state_backend(),
events=event_bus,
max_workers=settings.FLOW_MAX_WORKERS,
fastapi_app=app,
execution=execution,
alerts=alerts,
workers=pool,
remote=worker_hub,
resources=accountant,
placer=placer,
)
app.state.flow_controller = controller
# A "dashboard" alert channel puts its alert into the graph. Bound here
# rather than passed in: the manager is built before the controller is.
alerts.publish = lambda name, value: controller.publish_message(
name, value, ValueSource(kind="api", id="alerts", label="Alerts")
)
dashboards = DashboardStore(controller.store)
app.state.dashboard_store = dashboards
controller.dashboards = dashboards
# Charts need a deeper series than the default; tell the engine
# before it starts recording.
controller.set_history_limits(dashboards.history_requirements())
# Runs read from a stream of their own: a burst of sweep runs must not
# stand between the automations and their work, and a run that takes an
# hour must not be judged by the cascade reaper's timings.
run_service = RunService(
controller=controller,
queue=_work_queue("run"),
state_factory=_run_state,
artifacts=artifacts,
parallel=settings.FLOW_MAX_RUNS,
)
app.state.run_service = run_service
# Said out loud because it is the only way to tell that a settings file was
# read at all — the numbers are what someone raising them is looking for.
logger.info(
"Engine limits: workers=%s cascades=%s runs=%s",
settings.FLOW_MAX_WORKERS,
execution.max_cascades,
run_service.parallel,
)
watchdog = LoopWatchdog(event_bus)
app.state.watchdog = watchdog
_background(watchdog.run(), "loop-watchdog")
_background(alerts.run(), "alert-manager")
_background(MetricsCollector(event_bus).run(), "metrics-collector")
_background(_sweep_artifacts(artifacts, controller), "artifact-gc")
await controller.start()
started.append(controller.stop)
run_service.start()
started.append(lambda: run_in_threadpool(run_service.stop))
# A machine asked for and not yet arrived would hold an allocation nobody
# is going to use.
started.append(
lambda: asyncio.gather(
*(
run_in_threadpool(provisioner.shutdown)
for provisioner in placer.provisioners
)
)
)
started.append(lambda: run_in_threadpool(close_shared_client))
# Optional, and off unless someone enrolled this installation: the
# connector dials the portal, nothing dials in.
cloud_task: asyncio.Task[None] | None = None
app.state.cloud_connector = None
app.state.cloud_task = None
from fluksio.cloud import connector as cloud_connector
if cloud_config.exists():
cloud_connector.start(app)
cloud_task = app.state.cloud_task
def _stop_cloud() -> None:
# Re-read from app.state: enrolling at runtime replaces this.
running = getattr(app.state, "cloud_task", None) or cloud_task
if running is not None:
running.cancel()
started.append(_stop_cloud)
# Watched whether or not one exists now: enrolling from the CLI writes the
# config from another process entirely, and an engine already serving
# should pick it up rather than need restarting.
_background(cloud_connector.watch_enrolment(app), "cloud-enrolment-watch")
# A mounted sub-app gets no lifespan of its own, so the MCP session
# manager is entered here; without it every /mcp request fails.
async with _mcp_sessions():
yield
except Exception:
# A startup that fails part-way lands here too, because everything
# since `event_bus.bind` registered its own release.
logger.exception("The engine could not start")
raise
finally:
watchdog_task.cancel()
alerts_task.cancel()
metrics_task.cancel()
gc_task.cancel()
enrol_task.cancel()
# Re-read from app.state: enrolling at runtime replaces this.
running_cloud = getattr(app.state, "cloud_task", None) or cloud_task
if running_cloud is not None:
running_cloud.cancel()
await run_in_threadpool(run_service.stop)
await controller.stop()
# On a thread, like the two above: stopping a worker waits up to five
# seconds on each child, and with a pool per declared environment that
# is a shutdown the event loop should not be holding.
await run_in_threadpool(pool.stop)
# A machine asked for and not yet arrived would hold an allocation
# nobody is going to use.
for provisioner in placer.provisioners:
await run_in_threadpool(provisioner.shutdown)
await run_in_threadpool(close_shared_client)
await _release()
if settings.MCP_ENABLED:
from fluksio.mcp.http import aclose