Keep the engine's state in SQLite, not Postgres

One process owns this database — the image has run a single uvicorn
worker for that reason since the four-engines bug — so a file beside the
flows is the honest shape for it, and it is what lets `fluksio serve`
need no infrastructure at all. Live values, node execution and the work
queue never came here anyway; what does is a rollup a minute at a time,
a row per cascade and the run history, and WAL keeps the readers going
while that one writer works.

DATA_DIR is now the one setting that moves everything an installation
keeps; the rest derive from it and the images still spell theirs out.
The schema is prepared in-process at startup, so the prestart service is
gone, and the ten Postgres-only revisions collapse into one portable
baseline.

Three things only worked because psycopg was casting for us: a token's
subject arriving as a string where the column is a UUID, `greatest`, and
`date_bin`. The timestamps needed a column type of their own — SQLite
stores no offset, and a naive datetime read back either raises against an
aware `now` or serialises as local time.

Postgres stays in the stack only for Umami, behind the analytics profile.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-21 22:19:45 +02:00
co-authored by Claude Opus 5
parent 2c369ac75f
commit 961a8f881d
51 changed files with 841 additions and 1089 deletions
+20 -20
View File
@@ -13,7 +13,7 @@ from typing import Any, Literal
from fastapi import APIRouter, Depends, Request
from fastapi.concurrency import run_in_threadpool
from pydantic import BaseModel
from sqlalchemy import ColumnElement, DateTime, Interval, cast, func, literal
from sqlalchemy import ColumnElement, Integer, cast, func
from sqlalchemy import select as sa_select
from sqlmodel import col, select
@@ -31,9 +31,15 @@ router = APIRouter(
#: How many slices a per-flow sparkline is folded into.
SPARK_SLICES = 60
#: The origin fixed-stride slots are aligned to, which is the alignment the
#: fold used to get from ``stamp - stamp % bucket_s``.
EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc)
def _epoch(column: Any) -> ColumnElement[int]:
"""Seconds since 1970, as an integer the database can bin on.
Binning is integer arithmetic on this rather than a dialect's own date
function: buckets are whole minutes, so the cast is exact, and the
expression is the same everywhere.
"""
return cast(func.extract("epoch", column), Integer)
class HealthSummary(BaseModel):
@@ -203,16 +209,10 @@ def read_timeseries(
) -> Any:
"""Executions, errors and timings over time, summed across nodes."""
hours = _window_hours(hours)
# Postgres does the fold: a week of minute rows per node used to cross the
# wire on every poll, and only the slices need to. The casts are load
# bearing — date_bin() is overloaded on timestamp and timestamptz, and an
# untyped bind parameter leaves the call ambiguous.
stride = timedelta(seconds=max(60, bucket_s))
slot = func.date_bin(
cast(literal(stride), Interval),
col(MetricBucket.bucket),
cast(literal(EPOCH), DateTime(timezone=True)),
).label("slot")
# The database does the fold: a week of minute rows per node used to cross
# the wire on every poll, and only the slices need to.
stride = max(60, bucket_s)
slot = (_epoch(col(MetricBucket.bucket)) // stride * stride).label("slot")
statement = sa_select(
slot,
func.sum(col(MetricBucket.executions)).label("executions"),
@@ -230,7 +230,7 @@ def read_timeseries(
return [
SeriesPoint(
ts=row.slot.timestamp(),
ts=float(row.slot),
executions=int(row.executions),
errors=int(row.errors),
messages=int(row.messages),
@@ -253,10 +253,10 @@ def read_flow_rollups(session: SessionDep, hours: int = 24) -> Any:
# Binned to the sparkline slice rather than the minute, so a flow costs at
# most SPARK_SLICES rows however long the window is. The slice is the
# window over SPARK_SLICES, which for whole hours is whole minutes.
slot = func.date_bin(
cast(literal(timedelta(minutes=hours)), Interval),
col(MetricBucket.bucket),
cast(literal(since), DateTime(timezone=True)),
stride = hours * 60
origin = int(start)
slot = (
(_epoch(col(MetricBucket.bucket)) - origin) // stride * stride + origin
).label("slot")
statement = (
sa_select(
@@ -298,7 +298,7 @@ def read_flow_rollups(session: SessionDep, hours: int = 24) -> Any:
# clamp holds the bucket landing exactly on the far edge in range.
index = min(
SPARK_SLICES - 1,
max(0, round((row.slot.timestamp() - start) / window * SPARK_SLICES)),
max(0, round((float(row.slot) - start) / window * SPARK_SLICES)),
)
entry["spark"][index] += row.executions