stroblme
4925472f91
The queue shared the `pipeline:` prefix with flow state, so `RedisState.clear()` could DEL the queue stream and `keys()` enumerated queue entries — only callers filtering `__`-prefixed names kept it safe. It moves to `queue:` without a migration: whatever is in flight at the upgrade is dropped once, documented in DEPLOY.md rather than papered over. Alongside it: `pool_pre_ping`, so a connection idle across a Postgres restart costs a round trip instead of a failed request; the test suite pins ENVIRONMENT=local and DOMAIN=localhost itself rather than inheriting a deployment's .env; and `depth` leaves the queue stats, where it reported the capped journal length as if it were a backlog. ALERTS_FILE and PANELS_FILE now point at /data. They defaulted to a path on no volume, so alert routing and every wall-panel pairing were living in the container's writable layer and vanishing on each rebuild. Carrying the existing files across is a manual step; DEPLOY.md has it. development.md was still the upstream template — compose.override.yml, localhost.tiangolo.com, `docker compose watch` as the dev flow — and said nothing about the Playwright suite. Rewritten against what the Makefiles actually do. deployment.md was template text too, duplicating the root DEPLOY.md, and is gone.
172 lines
9.2 KiB
Markdown
172 lines
9.2 KiB
Markdown
# Development
|
|
|
|
How to run, test and lint the `app` stack. The workspace root owns the rest:
|
|
[`README.md`](../README.md) for the quick start, [`DEPLOY.md`](../DEPLOY.md) for
|
|
production, [`DESIGN-GUIDELINES.md`](../DESIGN-GUIDELINES.md) before any UI work.
|
|
[`ROADMAP.md`](ROADMAP.md) is what is planned, [`NOTEPAD.md`](NOTEPAD.md) what is deferred.
|
|
|
|
## Running it
|
|
|
|
**Integrated — the normal way.** From the workspace root, once: `make init` (submodules,
|
|
secrets, the per-stack `.env` files, the shared external `proxy` docker network). Then
|
|
`make dev` brings both stacks up detached behind a single Traefik — which is what lets the
|
|
SPA, the API and the marketing site answer on one port. `make status` lists the containers,
|
|
`make down` stops everything.
|
|
|
|
The hostname comes from `DOMAIN` in `app/.env`, which `scripts/setup.sh` copies out of the
|
|
root `.env` — change it there and re-run `make init`. `make dev DOMAIN=…` at the root does
|
|
*not* reach this stack: `app/Makefile` assigns `DOMAIN` from `app/.env`, and a makefile
|
|
assignment overrides an inherited environment variable.
|
|
|
|
**App only.** `cd app && make dev` starts this stack on its own Traefik with all host ports
|
|
published, in the foreground. `make dev-utils` starts just db, adminer, proxy, mailcatcher
|
|
and prestart — the useful half when the backend runs on the host.
|
|
|
|
**No Docker.** `make install` once, then `make dev-backend` (FastAPI on :8000) and
|
|
`make dev-frontend` (Vite on :5173) in two terminals. A local stack's CORS list already
|
|
contains `http://localhost:5173`, so a host Vite server can talk to a containerised API.
|
|
|
|
Nothing invokes `docker compose` bare, and neither should you: both stacks keep their
|
|
compose files in a `docker/` directory, so without `-p fluksio-app` they collide in a
|
|
project named after that directory, and `--env-file` is required because compose
|
|
interpolates `.env` before it reads `compose.yml`. The `COMPOSE` variable in `Makefile` has
|
|
both. The files layer as `compose.yml` (production) → `compose.dev.yml` (local Traefik,
|
|
published ports, hot reload, test helpers) → `compose.local.yml` (integrated). CI layers
|
|
`compose.ci.yml` instead of the last one; `compose.traefik.yml` is the production edge
|
|
proxy, deployed on its own.
|
|
|
|
## URLs
|
|
|
|
Integrated stack (root `make dev`). Traefik's port 80 is the **only** published port:
|
|
`compose.local.yml` drops the rest, since behind the proxy they add nothing and one of them
|
|
already being taken would stop the whole stack from starting.
|
|
|
|
| URL | Service |
|
|
|---|---|
|
|
| http://app.localhost | dashboard SPA |
|
|
| http://api.localhost | backend API |
|
|
| http://api.localhost/docs | OpenAPI / Swagger UI |
|
|
| http://localhost | marketing site (the `index` stack) |
|
|
|
|
Adminer and mailcatcher have no route here, so use the containers directly:
|
|
`docker exec -it fluksio-db psql -U postgres -d app` for the database,
|
|
`docker logs -f fluksio-app-mailcatcher-1` for captured mail.
|
|
|
|
Standalone (`cd app && make dev`) publishes the full set:
|
|
|
|
| URL | Service |
|
|
|---|---|
|
|
| http://app.localhost, http://api.localhost | this stack's own Traefik on :80 |
|
|
| http://localhost:5173 | SPA (the nginx image, not the Vite dev server) |
|
|
| http://localhost:8000 | backend API |
|
|
| http://localhost:8090 | Traefik dashboard |
|
|
| http://127.0.0.1:8080 | adminer |
|
|
| http://localhost:1080 | mailcatcher web UI (SMTP on 1025) |
|
|
| localhost:5432 | Postgres |
|
|
|
|
## Hot reload, and what it misses
|
|
|
|
`compose.dev.yml` bind-mounts `backend/app` into the api container and runs uvicorn with
|
|
`--reload`, so a backend edit is live. The `develop.watch` block next to it syncs the same
|
|
directory, but only under `docker compose watch`, which no target runs — the detached
|
|
`up -d` flow never triggers it. The mount is what does the work, so an api container created
|
|
before the mount existed keeps serving the source baked into its image and has to be
|
|
recreated once.
|
|
|
|
The frontend is built into an nginx image, so a UI change needs an explicit rebuild, from
|
|
`app/`:
|
|
|
|
```sh
|
|
docker compose -p fluksio-app --env-file "$PWD/.env" \
|
|
-f docker/compose.yml -f docker/compose.dev.yml -f docker/compose.local.yml \
|
|
up --build -d frontend
|
|
```
|
|
|
|
## Services the dev stack adds
|
|
|
|
`compose.dev.yml` brings up a broker and a time-series database of the stack's own, so the
|
|
mqtt and influx node types are testable without external hardware. Neither publishes a host
|
|
port, on purpose — nothing outside the stack needs them.
|
|
|
|
- **mosquitto** — anonymous MQTT. Point a node at `broker_host: mosquitto`, port 1883.
|
|
- **influxdb 2.7** — `http://influxdb:8086`, org and bucket `fluksio`, token
|
|
`fluksio-dev-token`. No volume, so `down -v` starts it over.
|
|
- **mailcatcher** — SMTP on 1025, web UI on 1080; the backend is pointed at it, so no
|
|
development mail leaves the machine.
|
|
|
|
`adminer` and `redis` come from `compose.yml` and exist in production too; dev only differs
|
|
in that adminer gets a host port. Without `REDIS_HOST` the flow engine keeps state in
|
|
memory instead of Redis.
|
|
|
|
## Configuration and secrets
|
|
|
|
The root `.env` is the source of truth. `scripts/setup.sh` creates `app/.env` from
|
|
`app/.env.example` and keeps the shared keys — domain, secret key, database password,
|
|
superuser — in step with it. Only the `.env.example` files are tracked; `make secrets`
|
|
reprints the generated values. A release that adds a key needs `scripts/setup.sh --secrets`
|
|
before `make update`, so the key exists before the stack is rebuilt.
|
|
|
|
The backend settings load `../.env` relative to `backend/` through pydantic-settings, which
|
|
is why an exported environment variable always outranks the file — how CI points the suite
|
|
at a Docker-assigned Postgres port.
|
|
|
|
## Tests
|
|
|
|
`make test` is `test-backend` plus `test-frontend`.
|
|
|
|
**Backend.** `cd backend && uv run bash scripts/tests-start.sh` waits for Postgres, then
|
|
runs pytest under coverage; the HTML report lands in `backend/htmlcov`. Postgres is the only
|
|
service needed — mail is patched out in `tests/`, and the flow engine falls back to
|
|
`MemoryState` while `REDIS_HOST` is empty. `tests/__init__.py` pins the suite's own
|
|
environment before anything imports the settings: database `app_test` (created and dropped
|
|
per session, so a run never touches development data), plus `ENVIRONMENT=local` and
|
|
`DOMAIN=localhost`, so a checkout configured for a deployment cannot drag that deployment's
|
|
configuration into the run. `make test-backend` resolves `POSTGRES_SERVER` from the running
|
|
`fluksio-db` container's address, because the integrated stack publishes no 5432.
|
|
|
|
**Frontend (e2e).** `make test-frontend` runs the suite inside the pinned
|
|
`mcr.microsoft.com/playwright:v<version>-noble` image (version read from
|
|
`frontend/package.json`) on the `proxy` network, against a stack that must already be up. It
|
|
maps `app.$(DOMAIN)` and `api.$(DOMAIN)` onto the Traefik container's address twice, with
|
|
`--add-host` and with `HOST_RESOLVER_RULES`, because Chromium pins `*.localhost` to loopback
|
|
whatever `/etc/hosts` says. Narrow a run with
|
|
`make test-frontend PLAYWRIGHT_ARGS="--grep flows"`.
|
|
|
|
The specs are `frontend/tests/*.spec.ts`; `playwright.config.ts` defines `setup` →
|
|
`chromium` and `mobile`. `auth.setup.ts` logs in once into `playwright/.auth/user.json`,
|
|
which both browser projects reuse; `mobile` runs `mobile.spec.ts` only, on a Pixel 5 at
|
|
393px.
|
|
|
|
Both origins come from `PLAYWRIGHT_BASE_URL` and `PLAYWRIGHT_API_URL`, deliberately never
|
|
from `VITE_API_URL` — that one belongs to the app build and in a deployment checkout it
|
|
names the deployment, which would mean driving a browser at the local stack while sending
|
|
teardown `DELETE`s to the live instance. `tests/guard.ts` runs first and refuses the whole
|
|
run when either origin resolves outside loopback or the private ranges, because the suite
|
|
creates and deletes flows, dashboards and users. `PLAYWRIGHT_ALLOW_PUBLIC=1` overrides it.
|
|
|
|
**Visual check.** From the root, `make verify` logs in and screenshots both themes into
|
|
`app/frontend/screenshots/{light,dark}/`; `make verify-docker` does the same from the
|
|
Playwright container when the host lacks the browser libraries. Both default to `localhost`
|
|
rather than reading `.env`, so on a checkout configured for a deployment name the domain the
|
|
stack actually serves: `make verify DOMAIN=example.com`.
|
|
|
|
## Lint, hooks and the generated client
|
|
|
|
`make lint` runs ruff, mypy and biome. `make hooks` at the root installs the pre-commit
|
|
hooks in both stacks; `.pre-commit-config.yaml` holds them. One is worth knowing about:
|
|
`generate-frontend-sdk` regenerates the SPA's API client from the backend's OpenAPI schema
|
|
whenever anything under `backend/` changes, so a route change reaches the frontend without
|
|
being asked. `make generate-client` forces it.
|
|
|
|
Run the root's `make design-check` before pushing anything that touches the design tokens,
|
|
`lib/motion.ts`, `components.json` or the font asset — the design system is a duplication
|
|
contract between the two frontends, and that target is the only thing checking it.
|
|
|
|
## CI
|
|
|
|
`.gitea/workflows/` runs four jobs, each writing `.env` from `.env.example` first:
|
|
`test-backend.yml` (pytest against a compose Postgres), `playwright.yml` (the e2e suite in
|
|
two shards, one compose project each, the browser loading the SPA from the nginx `frontend`
|
|
service), `pre-commit.yml` (the hooks above, plus strict mypy, which they do not cover) and
|
|
`test-compose.yml` (a smoke test that the production images actually come up).
|