stroblmeandClaude Opus 5
961a8f881d
One process owns this database — the image has run a single uvicorn worker for that reason since the four-engines bug — so a file beside the flows is the honest shape for it, and it is what lets `fluksio serve` need no infrastructure at all. Live values, node execution and the work queue never came here anyway; what does is a rollup a minute at a time, a row per cascade and the run history, and WAL keeps the readers going while that one writer works. DATA_DIR is now the one setting that moves everything an installation keeps; the rest derive from it and the images still spell theirs out. The schema is prepared in-process at startup, so the prestart service is gone, and the ten Postgres-only revisions collapse into one portable baseline. Three things only worked because psycopg was casting for us: a token's subject arriving as a string where the column is a UUID, `greatest`, and `date_bin`. The timestamps needed a column type of their own — SQLite stores no offset, and a naive datetime read back either raises against an aware `now` or serialises as local time. Postgres stays in the stack only for Umami, behind the analytics profile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
179 lines
9.7 KiB
Markdown
179 lines
9.7 KiB
Markdown
# Development
|
|
|
|
How to run, test and lint the `app` stack. The workspace root owns the rest:
|
|
[`README.md`](../README.md) for the quick start, [`DEPLOY.md`](../DEPLOY.md) for
|
|
production, [`DESIGN-GUIDELINES.md`](../DESIGN-GUIDELINES.md) before any UI work.
|
|
[`ROADMAP.md`](ROADMAP.md) is what is planned, [`NOTEPAD.md`](NOTEPAD.md) what is deferred.
|
|
|
|
## Running it
|
|
|
|
**Integrated — the normal way.** From the workspace root, once: `make init` (submodules,
|
|
secrets, the per-stack `.env` files, the shared external `proxy` docker network). Then
|
|
`make dev` brings both stacks up detached behind a single Traefik — which is what lets the
|
|
SPA, the API and the marketing site answer on one port. `make status` lists the containers,
|
|
`make down` stops everything.
|
|
|
|
The hostname comes from `DOMAIN` in `app/.env`, which `scripts/setup.sh` copies out of the
|
|
root `.env` — change it there and re-run `make init`. `app/Makefile` takes that value as a
|
|
default only (`export DOMAIN ?= …`), so `make dev DOMAIN=…` overrides it and the override
|
|
reaches the recipes.
|
|
|
|
**App only.** `cd app && make dev` starts this stack on its own Traefik with all host ports
|
|
published, in the foreground. `make dev-utils` starts just proxy and mailcatcher — the
|
|
useful half when the backend runs on the host.
|
|
|
|
**No Docker.** `make install` once, then `make dev-backend` (FastAPI on :8000) and
|
|
`make dev-frontend` (Vite on :5173) in two terminals. The backend needs nothing running: its
|
|
database is SQLite in `DATA_DIR` (`backend/flow-data` by default), created on first start.
|
|
A local stack's CORS list already contains `http://localhost:5173`, so a host Vite server
|
|
can talk to a containerised API.
|
|
|
|
Nothing invokes `docker compose` bare, and neither should you: both stacks keep their
|
|
compose files in a `docker/` directory, so without `-p fluksio-app` they collide in a
|
|
project named after that directory, and `--env-file` is required because compose
|
|
interpolates `.env` before it reads `compose.yml`. The `COMPOSE` variable in `Makefile` has
|
|
both. The files layer as `compose.yml` (production) → `compose.dev.yml` (local Traefik,
|
|
published ports, hot reload, test helpers) → `compose.local.yml` (integrated). CI layers
|
|
`compose.ci.yml` instead of the last one; `compose.traefik.yml` is the production edge
|
|
proxy, deployed on its own.
|
|
|
|
## URLs
|
|
|
|
Integrated stack (root `make dev`). Traefik's port 80 is the **only** published port:
|
|
`compose.local.yml` drops the rest, since behind the proxy they add nothing and one of them
|
|
already being taken would stop the whole stack from starting.
|
|
|
|
| URL | Service |
|
|
|---|---|
|
|
| http://app.localhost | dashboard SPA |
|
|
| http://api.localhost | backend API |
|
|
| http://api.localhost/docs | OpenAPI / Swagger UI |
|
|
| http://localhost | marketing site (the `index` stack) |
|
|
|
|
Mailcatcher has no route here, so use the container directly:
|
|
`docker logs -f fluksio-app-mailcatcher-1` for captured mail. The database is a file on the
|
|
data volume: `docker exec -it fluksio-api sqlite3 /data/fluksio.db`.
|
|
|
|
Standalone (`cd app && make dev`) publishes the full set:
|
|
|
|
| URL | Service |
|
|
|---|---|
|
|
| http://app.localhost, http://api.localhost | this stack's own Traefik on :80 |
|
|
| http://localhost:5173 | SPA (the nginx image, not the Vite dev server) |
|
|
| http://localhost:8000 | backend API |
|
|
| http://localhost:8090 | Traefik dashboard |
|
|
| http://localhost:1080 | mailcatcher web UI (SMTP on 1025) |
|
|
|
|
## Hot reload, and what it misses
|
|
|
|
`compose.dev.yml` bind-mounts `backend/fluksio` into the api container and runs uvicorn with
|
|
`--reload`, so a backend edit is live. The `develop.watch` block next to it syncs the same
|
|
directory, but only under `docker compose watch`, which no target runs — the detached
|
|
`up -d` flow never triggers it. The mount is what does the work, so an api container created
|
|
before the mount existed keeps serving the source baked into its image and has to be
|
|
recreated once.
|
|
|
|
The frontend is built into an nginx image, so a UI change needs an explicit rebuild, from
|
|
`app/`:
|
|
|
|
```sh
|
|
docker compose -p fluksio-app --env-file "$PWD/.env" \
|
|
-f docker/compose.yml -f docker/compose.dev.yml -f docker/compose.local.yml \
|
|
up --build -d frontend
|
|
```
|
|
|
|
## Services the dev stack adds
|
|
|
|
`compose.dev.yml` brings up a broker and a time-series database of the stack's own, so the
|
|
mqtt and influx node types are testable without external hardware. Neither publishes a host
|
|
port, on purpose — nothing outside the stack needs them.
|
|
|
|
- **mosquitto** — anonymous MQTT. Point a node at `broker_host: mosquitto`, port 1883.
|
|
- **influxdb 2.7** — `http://influxdb:8086`, org and bucket `fluksio`, token
|
|
`fluksio-dev-token`. No volume, so `down -v` starts it over.
|
|
- **mailcatcher** — SMTP on 1025, web UI on 1080; the backend is pointed at it, so no
|
|
development mail leaves the machine.
|
|
|
|
`redis` comes from `compose.yml` and exists in production too. Without `REDIS_HOST` the flow
|
|
engine keeps state in memory instead of Redis. `db` and `adminer` are still there but only
|
|
start under the `analytics` profile — they are Umami's Postgres now, not the engine's.
|
|
|
|
## Configuration and secrets
|
|
|
|
The root `.env` is the source of truth. `scripts/setup.sh` creates `app/.env` from
|
|
`app/.env.example` and keeps the shared keys — domain, secret key, superuser — in step with
|
|
it. Only the `.env.example` files are tracked; `make secrets`
|
|
reprints the generated values. A release that adds a key needs `scripts/setup.sh --secrets`
|
|
before `make update`, so the key exists before the stack is rebuilt.
|
|
|
|
The backend settings load `../.env` relative to `backend/` through pydantic-settings —
|
|
`FLUKSIO_ENV_FILE` points that elsewhere, which is what an installed `fluksio` does. An
|
|
exported environment variable always outranks the file, which is how the suite pins its own
|
|
configuration.
|
|
|
|
`DATA_DIR` is where this installation keeps everything: the SQLite database, the flow git
|
|
repository, `secrets.enc`, `oauth-key.pem`, `cloud.json`, artifacts and the user venv. Each
|
|
path derives from it and can still be set on its own — the compose stack spells all of them
|
|
out against `/data`.
|
|
|
|
## Tests
|
|
|
|
`make test` is `test-backend` plus `test-frontend`.
|
|
|
|
**Backend.** `cd backend && uv run bash scripts/test.sh` runs pytest under coverage; the
|
|
HTML report lands in `backend/htmlcov`. Nothing has to be running — mail is patched out in
|
|
`tests/`, and the flow engine falls back to `MemoryState` while `REDIS_HOST` is empty.
|
|
`tests/__init__.py` pins the suite's own environment before anything imports the settings: a
|
|
SQLite file in a temp directory (deleted per session, so a run never touches development
|
|
data), its own superuser, and `ENVIRONMENT=local` with `DOMAIN=localhost`, so a checkout
|
|
configured for a deployment cannot drag that deployment's configuration into the run.
|
|
|
|
**Frontend (e2e).** `make test-frontend` runs the suite inside the pinned
|
|
`mcr.microsoft.com/playwright:v<version>-noble` image (version read from
|
|
`frontend/package.json`) on the `proxy` network, against a stack that must already be up. It
|
|
maps `app.$(DOMAIN)` and `api.$(DOMAIN)` onto the Traefik container's address twice, with
|
|
`--add-host` and with `HOST_RESOLVER_RULES`, because Chromium pins `*.localhost` to loopback
|
|
whatever `/etc/hosts` says. Narrow a run with
|
|
`make test-frontend PLAYWRIGHT_ARGS="--grep flows"`.
|
|
|
|
The specs are `frontend/tests/*.spec.ts`; `playwright.config.ts` defines `setup` →
|
|
`chromium` and `mobile`. `auth.setup.ts` logs in once into `playwright/.auth/user.json`,
|
|
which both browser projects reuse; `mobile` runs `mobile.spec.ts` only, on a Pixel 5 at
|
|
393px.
|
|
|
|
Both origins come from `PLAYWRIGHT_BASE_URL` and `PLAYWRIGHT_API_URL`, deliberately never
|
|
from `VITE_API_URL` — that one belongs to the app build and in a deployment checkout it
|
|
names the deployment, which would mean driving a browser at the local stack while sending
|
|
teardown `DELETE`s to the live instance. `tests/guard.ts` runs first and refuses the whole
|
|
run when either origin resolves outside loopback or the private ranges, because the suite
|
|
creates and deletes flows, dashboards and users. `PLAYWRIGHT_ALLOW_PUBLIC=1` overrides it.
|
|
|
|
**Visual check.** From the root, `make verify` logs in and screenshots both themes into
|
|
`app/frontend/screenshots/{light,dark}/`. It is an alias for `make verify-docker`: the run
|
|
always happens in the Playwright container, which pins the browser by reading
|
|
`frontend/package.json`. Running Playwright on the host is unsupported — the workspace pins a
|
|
browser revision whose install is incomplete. The domain comes from `.env`, and
|
|
`make verify DOMAIN=example.com` still wins.
|
|
|
|
## Lint, hooks and the generated client
|
|
|
|
`make lint` runs ruff, mypy and biome, and only reports — it is what the pre-commit hook and
|
|
CI run, so neither rewrites the tree. `make format-frontend` is the writing half, applying
|
|
biome's fixes. `make hooks` at the root installs the pre-commit hooks in both stacks;
|
|
`.pre-commit-config.yaml` holds them. One is worth knowing about:
|
|
`generate-frontend-sdk` regenerates the SPA's API client from the backend's OpenAPI schema
|
|
whenever anything under `backend/` changes, so a route change reaches the frontend without
|
|
being asked. `make generate-client` forces it.
|
|
|
|
Run the root's `make design-check` before pushing anything that touches the design tokens,
|
|
`lib/motion.ts`, `components.json` or the font asset — the design system is a duplication
|
|
contract between the two frontends, and that target is the only thing checking it.
|
|
|
|
## CI
|
|
|
|
`.gitea/workflows/` runs four jobs, each writing `.env` from `.env.example` first:
|
|
`test-backend.yml` (pytest, plus a check that both wheels build), `playwright.yml` (the e2e suite in
|
|
two shards, one compose project each, the browser loading the SPA from the nginx `frontend`
|
|
service), `pre-commit.yml` (the hooks above, plus strict mypy, which they do not cover) and
|
|
`test-compose.yml` (a smoke test that the production images actually come up).
|