Keep the engine's state in SQLite, not Postgres

One process owns this database — the image has run a single uvicorn
worker for that reason since the four-engines bug — so a file beside the
flows is the honest shape for it, and it is what lets `fluksio serve`
need no infrastructure at all. Live values, node execution and the work
queue never came here anyway; what does is a rollup a minute at a time,
a row per cascade and the run history, and WAL keeps the readers going
while that one writer works.

DATA_DIR is now the one setting that moves everything an installation
keeps; the rest derive from it and the images still spell theirs out.
The schema is prepared in-process at startup, so the prestart service is
gone, and the ten Postgres-only revisions collapse into one portable
baseline.

Three things only worked because psycopg was casting for us: a token's
subject arriving as a string where the column is a UUID, `greatest`, and
`date_bin`. The timestamps needed a column type of their own — SQLite
stores no offset, and a naive datetime read back either raises against an
aware `now` or serialises as local time.

Postgres stays in the stack only for Umami, behind the analytics profile.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-21 22:19:45 +02:00
co-authored by Claude Opus 5
parent f8abd91fc0
commit 73eeec29b1
51 changed files with 841 additions and 1089 deletions
+31 -27
View File
@@ -19,12 +19,14 @@ default only (`export DOMAIN ?= …`), so `make dev DOMAIN=…` overrides it and
reaches the recipes.
**App only.** `cd app && make dev` starts this stack on its own Traefik with all host ports
published, in the foreground. `make dev-utils` starts just db, adminer, proxy, mailcatcher
and prestart — the useful half when the backend runs on the host.
published, in the foreground. `make dev-utils` starts just proxy and mailcatcher — the
useful half when the backend runs on the host.
**No Docker.** `make install` once, then `make dev-backend` (FastAPI on :8000) and
`make dev-frontend` (Vite on :5173) in two terminals. A local stack's CORS list already
contains `http://localhost:5173`, so a host Vite server can talk to a containerised API.
`make dev-frontend` (Vite on :5173) in two terminals. The backend needs nothing running: its
database is SQLite in `DATA_DIR` (`backend/flow-data` by default), created on first start.
A local stack's CORS list already contains `http://localhost:5173`, so a host Vite server
can talk to a containerised API.
Nothing invokes `docker compose` bare, and neither should you: both stacks keep their
compose files in a `docker/` directory, so without `-p fluksio-app` they collide in a
@@ -48,9 +50,9 @@ already being taken would stop the whole stack from starting.
| http://api.localhost/docs | OpenAPI / Swagger UI |
| http://localhost | marketing site (the `index` stack) |
Adminer and mailcatcher have no route here, so use the containers directly:
`docker exec -it fluksio-db psql -U postgres -d app` for the database,
`docker logs -f fluksio-app-mailcatcher-1` for captured mail.
Mailcatcher has no route here, so use the container directly:
`docker logs -f fluksio-app-mailcatcher-1` for captured mail. The database is a file on the
data volume: `docker exec -it fluksio-api sqlite3 /data/fluksio.db`.
Standalone (`cd app && make dev`) publishes the full set:
@@ -60,9 +62,7 @@ Standalone (`cd app && make dev`) publishes the full set:
| http://localhost:5173 | SPA (the nginx image, not the Vite dev server) |
| http://localhost:8000 | backend API |
| http://localhost:8090 | Traefik dashboard |
| http://127.0.0.1:8080 | adminer |
| http://localhost:1080 | mailcatcher web UI (SMTP on 1025) |
| localhost:5432 | Postgres |
## Hot reload, and what it misses
@@ -94,35 +94,39 @@ port, on purpose — nothing outside the stack needs them.
- **mailcatcher** — SMTP on 1025, web UI on 1080; the backend is pointed at it, so no
development mail leaves the machine.
`adminer` and `redis` come from `compose.yml` and exist in production too; dev only differs
in that adminer gets a host port. Without `REDIS_HOST` the flow engine keeps state in
memory instead of Redis.
`redis` comes from `compose.yml` and exists in production too. Without `REDIS_HOST` the flow
engine keeps state in memory instead of Redis. `db` and `adminer` are still there but only
start under the `analytics` profile — they are Umami's Postgres now, not the engine's.
## Configuration and secrets
The root `.env` is the source of truth. `scripts/setup.sh` creates `app/.env` from
`app/.env.example` and keeps the shared keys — domain, secret key, database password,
superuser — in step with it. Only the `.env.example` files are tracked; `make secrets`
`app/.env.example` and keeps the shared keys — domain, secret key, superuser — in step with
it. Only the `.env.example` files are tracked; `make secrets`
reprints the generated values. A release that adds a key needs `scripts/setup.sh --secrets`
before `make update`, so the key exists before the stack is rebuilt.
The backend settings load `../.env` relative to `backend/` through pydantic-settings, which
is why an exported environment variable always outranks the file — how CI points the suite
at a Docker-assigned Postgres port.
The backend settings load `../.env` relative to `backend/` through pydantic-settings
`FLUKSIO_ENV_FILE` points that elsewhere, which is what an installed `fluksio` does. An
exported environment variable always outranks the file, which is how the suite pins its own
configuration.
`DATA_DIR` is where this installation keeps everything: the SQLite database, the flow git
repository, `secrets.enc`, `oauth-key.pem`, `cloud.json`, artifacts and the user venv. Each
path derives from it and can still be set on its own — the compose stack spells all of them
out against `/data`.
## Tests
`make test` is `test-backend` plus `test-frontend`.
**Backend.** `cd backend && uv run bash scripts/tests-start.sh` waits for Postgres, then
runs pytest under coverage; the HTML report lands in `backend/htmlcov`. Postgres is the only
service needed — mail is patched out in `tests/`, and the flow engine falls back to
`MemoryState` while `REDIS_HOST` is empty. `tests/__init__.py` pins the suite's own
environment before anything imports the settings: database `app_test` (created and dropped
per session, so a run never touches development data), plus `ENVIRONMENT=local` and
`DOMAIN=localhost`, so a checkout configured for a deployment cannot drag that deployment's
configuration into the run. `make test-backend` resolves `POSTGRES_SERVER` from the running
`fluksio-db` container's address, because the integrated stack publishes no 5432.
**Backend.** `cd backend && uv run bash scripts/test.sh` runs pytest under coverage; the
HTML report lands in `backend/htmlcov`. Nothing has to be running — mail is patched out in
`tests/`, and the flow engine falls back to `MemoryState` while `REDIS_HOST` is empty.
`tests/__init__.py` pins the suite's own environment before anything imports the settings: a
SQLite file in a temp directory (deleted per session, so a run never touches development
data), its own superuser, and `ENVIRONMENT=local` with `DOMAIN=localhost`, so a checkout
configured for a deployment cannot drag that deployment's configuration into the run.
**Frontend (e2e).** `make test-frontend` runs the suite inside the pinned
`mcr.microsoft.com/playwright:v<version>-noble` image (version read from
@@ -168,7 +172,7 @@ contract between the two frontends, and that target is the only thing checking i
## CI
`.gitea/workflows/` runs four jobs, each writing `.env` from `.env.example` first:
`test-backend.yml` (pytest against a compose Postgres), `playwright.yml` (the e2e suite in
`test-backend.yml` (pytest, plus a check that both wheels build), `playwright.yml` (the e2e suite in
two shards, one compose project each, the browser loading the SPA from the nginx `frontend`
service), `pre-commit.yml` (the hooks above, plus strict mypy, which they do not cover) and
`test-compose.yml` (a smoke test that the production images actually come up).