Make documentation standalone and generalize examples
This commit is contained in:
Melvin Strobl
+118
-153
@@ -1,192 +1,157 @@
|
||||
# Development
|
||||
|
||||
How to run, test and lint the `app` stack.
|
||||
Run all commands from this repository's root unless a step says otherwise.
|
||||
|
||||
## Running it
|
||||
## Run the app
|
||||
|
||||
**Integrated — the normal way.** From the workspace root, once: `make init` (submodules,
|
||||
secrets, the per-stack `.env` files, the shared external `proxy` docker network). Then
|
||||
`make dev` brings both stacks up detached behind a single Traefik — which is what lets the
|
||||
app and the marketing site answer on one port. The app is one container
|
||||
(`backend/Dockerfile`): the engine, its API and the built dashboard on port 8000. `make status` lists the containers,
|
||||
`make down` stops everything.
|
||||
### Docker development stack
|
||||
|
||||
The hostname comes from `DOMAIN` in `app/.env`, which `scripts/setup.sh` copies out of the
|
||||
root `.env` — change it there and re-run `make init`. `app/Makefile` takes that value as a
|
||||
default only (`export DOMAIN ?= …`), so `make dev DOMAIN=…` overrides it and the override
|
||||
reaches the recipes.
|
||||
Install Docker with Compose v2, Make and OpenSSL, then run:
|
||||
|
||||
**App only.** In a bare clone of this repo, `make init` once (`.env` from `.env.example`
|
||||
with generated secrets, the `proxy` docker network, prints the admin login), then
|
||||
`make dev-local` serves the stack detached at `http://app.localhost` behind Traefik on
|
||||
port 80. `make dev` starts this stack on its own Traefik with all host ports
|
||||
published, in the foreground. `make dev-utils` starts just proxy and mailcatcher — the
|
||||
useful half when the backend runs on the host.
|
||||
```sh
|
||||
make init # create .env, generate secrets and create the proxy network
|
||||
make dev-local # start the app and docs behind the local proxy
|
||||
make down # stop the stack
|
||||
```
|
||||
|
||||
**Single container.** `docker compose up -d` in this repo's root uses `compose.yml`, the
|
||||
setup the getting-started guide gives: the image on `localhost:8080` with its own Redis, no
|
||||
Traefik, no `.env`. It is a separate project from the stack below and shares nothing
|
||||
with it.
|
||||
`make init` prints the admin login. The app image contains the engine, API and
|
||||
built dashboard on port 8000. The proxy exposes it at
|
||||
[app.localhost](http://app.localhost), with the API under `/api/v1`.
|
||||
[api.localhost](http://api.localhost) reaches the same container, and
|
||||
[docs.localhost](http://docs.localhost) serves the documentation.
|
||||
|
||||
**No Docker.** `make install` once, then `make dev-backend` (FastAPI on :8000) and
|
||||
`make dev-frontend` (Vite on :5173) in two terminals. The backend needs nothing running: its
|
||||
database is SQLite in `DATA_DIR` (`backend/flow-data` by default), created on first start.
|
||||
A local stack's CORS list already contains `http://localhost:5173`, so a host Vite server
|
||||
can talk to a containerised API.
|
||||
Development targets default to `DOMAIN=localhost`, independently of `.env`.
|
||||
Use `make dev-local DOMAIN=example.test` to choose another development domain.
|
||||
`make dev-lan APP_PORT=8080` also publishes the app on the machine's LAN address
|
||||
for devices that cannot resolve the proxy hostname.
|
||||
|
||||
Nothing invokes `docker compose` bare, and neither should you: both stacks keep their
|
||||
compose files in a `docker/` directory, so without `-p fluksio-app` they collide in a
|
||||
project named after that directory, and `--env-file` is required because compose
|
||||
interpolates `.env` before it reads `compose.yml`. The `COMPOSE` variable in `Makefile` has
|
||||
both. The files layer as `compose.yml` (production) → `compose.dev.yml` (local Traefik,
|
||||
published ports, hot reload, test helpers) → `compose.local.yml` (integrated). CI layers
|
||||
`compose.ci.yml` instead of the last one; `compose.traefik.yml` is the production edge
|
||||
proxy, deployed on its own.
|
||||
|
||||
## URLs
|
||||
|
||||
Integrated stack (root `make dev`). Traefik's port 80 is the **only** published port:
|
||||
`compose.local.yml` drops the rest, since behind the proxy they add nothing and one of them
|
||||
already being taken would stop the whole stack from starting.
|
||||
`make dev` runs in the foreground with additional published ports:
|
||||
|
||||
| URL | Service |
|
||||
|---|---|
|
||||
| http://app.localhost | dashboard, with the API under `/api/v1` |
|
||||
| http://api.localhost | the same container, for callers configured with the API host |
|
||||
| http://app.localhost/docs | OpenAPI / Swagger UI |
|
||||
| http://localhost | marketing site (the `index` stack) |
|
||||
|
||||
Mailcatcher has no route here, so use the container directly:
|
||||
`docker logs -f fluksio-app-mailcatcher-1` for captured mail. The database is a file on the
|
||||
data volume: `docker exec -it fluksio-api sqlite3 /data/fluksio.db`.
|
||||
|
||||
Standalone (`cd app && make dev`) publishes the full set:
|
||||
|
||||
| URL | Service |
|
||||
|---|---|
|
||||
| http://app.localhost, http://api.localhost | this stack's own Traefik on :80 |
|
||||
| http://localhost:8000 | dashboard and API |
|
||||
| http://app.localhost, http://api.localhost | app through Traefik |
|
||||
| http://localhost:8000 | dashboard and API directly |
|
||||
| http://localhost:8090 | Traefik dashboard |
|
||||
| http://localhost:1080 | mailcatcher web UI (SMTP on 1025) |
|
||||
| http://localhost:1080 | captured mail; SMTP on port 1025 |
|
||||
|
||||
## Hot reload, and what it misses
|
||||
For `make dev-local`, read captured mail with
|
||||
`docker logs -f fluksio-app-mailcatcher-1`. The proxy is the only published
|
||||
port in that setup.
|
||||
|
||||
`compose.dev.yml` bind-mounts `backend/fluksio` and `worker/fluksio_worker` into the api
|
||||
container and runs uvicorn with `--reload` over both, so a backend edit is live and a worker
|
||||
edit restarts the engine along with its local workers. The `develop.watch` block next to it
|
||||
syncs the same directories, but only under `docker compose watch`, which no target runs — the detached
|
||||
`up -d` flow never triggers it. The mount is what does the work, so an api container created
|
||||
before the mount existed keeps serving the source baked into its image and has to be
|
||||
recreated once.
|
||||
### Single-container setup
|
||||
|
||||
The dashboard is built into the app image, so a UI change needs a rebuild:
|
||||
`make rebuild-frontend` from `app/`. Only the image's frontend stage runs again, and the
|
||||
container is recreated with the domain and overlays the running one was started with.
|
||||
`docker compose up -d` uses the top-level `compose.yml`. It serves the app on
|
||||
[localhost:8080](http://localhost:8080), includes Redis, and needs no `.env` or
|
||||
proxy. See the [facility guide](docs/getting-started/facility-automation.md).
|
||||
This setup has separate data from the development stack.
|
||||
|
||||
## Services the dev stack adds
|
||||
For the operator stack under `docker/`, use the Makefile targets. They pass
|
||||
the project name and env-file path explicitly. Its files layer as
|
||||
`compose.yml` → `compose.dev.yml` → `compose.local.yml`; CI uses
|
||||
`compose.ci.yml` as its final overlay. `compose.traefik.yml` supplies a
|
||||
separate production edge proxy.
|
||||
|
||||
`compose.dev.yml` brings up a broker and a time-series database of the stack's own, so the
|
||||
mqtt and influx node types are testable without external hardware. Neither publishes a host
|
||||
port, on purpose — nothing outside the stack needs them.
|
||||
### Without Docker
|
||||
|
||||
- **mosquitto** — anonymous MQTT. Point a node at `broker_host: mosquitto`, port 1883.
|
||||
- **influxdb 2.7** — `http://influxdb:8086`, org and bucket `fluksio`, token
|
||||
`fluksio-dev-token`. No volume, so `down -v` starts it over.
|
||||
- **mailcatcher** — SMTP on 1025, web UI on 1080; the backend is pointed at it, so no
|
||||
development mail leaves the machine.
|
||||
Install uv and Bun, then run `make install`. Use two terminals:
|
||||
|
||||
`redis` comes from `compose.yml` and exists in production too. Without `REDIS_HOST` the flow
|
||||
engine keeps state in memory instead of Redis. `db` and `adminer` are still there but only
|
||||
start under the `analytics` profile — they are Umami's Postgres now, not the engine's.
|
||||
```sh
|
||||
make dev-backend # FastAPI on :8000 with hot reload
|
||||
make dev-frontend # Vite on :5173
|
||||
```
|
||||
|
||||
The backend creates SQLite in `DATA_DIR` (`backend/flow-data` by default).
|
||||
No database service is required. `make dev-utils` starts just the proxy and
|
||||
mailcatcher if you need them. The Docker development API also accepts the
|
||||
Vite origin, so a host Vite process can use it.
|
||||
|
||||
## Reloading changes
|
||||
|
||||
`docker/compose.dev.yml` mounts `backend/fluksio` and `worker/fluksio_worker`
|
||||
into the API container. Uvicorn watches both, so changes restart the engine
|
||||
and local workers. The adjacent `develop.watch` settings apply only to
|
||||
`docker compose watch`; the Makefile's detached targets use the mounts.
|
||||
|
||||
The dashboard is built into the image. After a UI change, run:
|
||||
|
||||
```sh
|
||||
make rebuild-frontend
|
||||
```
|
||||
|
||||
This preserves the running stack's domain, overlays and LAN port.
|
||||
|
||||
## Development services
|
||||
|
||||
The development Compose file includes services for trying integrations:
|
||||
|
||||
- **Mosquitto:** anonymous MQTT at `mosquitto:1883` inside the stack.
|
||||
- **InfluxDB:** `http://influxdb:8086`, organization and bucket `fluksio`,
|
||||
development token `fluksio-dev-token`.
|
||||
- **Mailcatcher:** captures outbound mail locally.
|
||||
|
||||
MQTT and InfluxDB have no published host ports. Redis persists flow state and
|
||||
queued work. Without `REDIS_HOST`, a source-run engine uses memory instead.
|
||||
Postgres and Adminer start only with the optional `analytics` profile.
|
||||
|
||||
## Configuration and secrets
|
||||
|
||||
The root `.env` is the source of truth. `scripts/setup.sh` creates `app/.env` from
|
||||
`app/.env.example` and keeps the shared keys — domain, secret key, superuser — in step with
|
||||
it. Only the `.env.example` files are tracked; `make secrets`
|
||||
reprints the generated values. A release that adds a key needs `scripts/setup.sh --secrets`
|
||||
before `make update`, so the key exists before the stack is rebuilt.
|
||||
`make init` copies `.env.example` to `.env` and generates initial secrets.
|
||||
Edit `.env` directly and keep it out of git. After updating the repository,
|
||||
compare it with `.env.example` for new settings. Generate additional secrets
|
||||
with `openssl rand -hex 32`.
|
||||
|
||||
The backend settings load `../.env` relative to `backend/` through pydantic-settings —
|
||||
`FLUKSIO_ENV_FILE` points that elsewhere, which is what an installed `fluksio` does. An
|
||||
exported environment variable always outranks the file, which is how the suite pins its own
|
||||
configuration.
|
||||
Backend settings load this repository's `.env` when run from `backend/`.
|
||||
`FLUKSIO_ENV_FILE` selects another file, and exported variables override it.
|
||||
The installed CLI uses the selected instance's data directory instead.
|
||||
|
||||
`DATA_DIR` is where this instance keeps everything: the SQLite database, the flow git
|
||||
repository, `secrets.enc`, `oauth-key.pem`, `cloud.json`, artifacts and the user venv. Each
|
||||
path derives from it and can still be set on its own — the compose stack spells all of them
|
||||
out against `/data`.
|
||||
`DATA_DIR` holds the database, flow repository, credentials, artifacts and
|
||||
managed node environment. Individual paths can be overridden. Docker stores
|
||||
them under `/data`; see [Configuration](docs/reference/configuration.md).
|
||||
|
||||
## Tests
|
||||
|
||||
`make test` is `test-backend` plus `test-frontend`.
|
||||
`make test` runs backend tests, frontend helper checks and Playwright tests.
|
||||
|
||||
**Backend.** `cd backend && uv run bash scripts/test.sh` runs pytest under coverage; the
|
||||
HTML report lands in `backend/htmlcov`. Nothing has to be running — mail is patched out in
|
||||
`tests/`, and the flow engine falls back to `MemoryState` while `REDIS_HOST` is empty.
|
||||
`tests/__init__.py` pins the suite's own environment before anything imports the settings: a
|
||||
SQLite file in a temp directory (deleted per session, so a run never touches development
|
||||
data), its own superuser, and `ENVIRONMENT=local`, so a checkout
|
||||
configured for a deployment cannot drag that deployment's configuration into the run.
|
||||
- `make test-backend` runs pytest with coverage. Its temporary SQLite database
|
||||
and test settings need no running services and leave development data alone.
|
||||
The HTML coverage report is in `backend/htmlcov/`.
|
||||
- `make test-checks` runs the frontend's `*.check.ts` helper scripts.
|
||||
- `make test-frontend` starts an isolated `fluksio-test` stack behind the local
|
||||
Traefik, then runs Playwright in a pinned container. Start `make dev-local`
|
||||
or `make dev-lan` first. The test stack stays up for inspection at
|
||||
`app.test.localhost`; `make down` stops it.
|
||||
- `make test-webkit` runs the drag tests in WebKit against the same test stack.
|
||||
|
||||
**Frontend (e2e).** `make test-frontend` runs the suite against a stack of its own, because
|
||||
the suite creates and deletes flows, dashboards and users. `make test-stack`, which it runs
|
||||
first, starts or updates the compose project `fluksio-test` (`docker/compose.test.yml` over the
|
||||
local stack's files), with its own containers, volumes and image tags. The local Traefik serves
|
||||
it on `app.test.localhost` and `api.test.localhost`, so `make dev` has to be up, in either
|
||||
mode: the test stack never layers `compose.lan.yml` and publishes no port. It stays running
|
||||
after a run for a look at a failure, `make down` stops it, and its volumes are expendable
|
||||
(`docker compose -p fluksio-test down -v` starts it over).
|
||||
|
||||
The suite runs inside the pinned `mcr.microsoft.com/playwright:v<version>-noble` image
|
||||
(version read from `frontend/package.json`) on the `proxy` network. It maps both names onto
|
||||
the Traefik container's address twice, with `--add-host` and with `HOST_RESOLVER_RULES`,
|
||||
because Chromium pins `*.localhost` to loopback whatever `/etc/hosts` says. Narrow a run with
|
||||
Narrow a browser run with
|
||||
`make test-frontend PLAYWRIGHT_ARGS="--grep flows"`.
|
||||
The browser version comes from `frontend/package.json`. Container resolver
|
||||
rules route `*.localhost` to Traefik, including in Chromium and WebKit.
|
||||
|
||||
`make test-webkit` runs the drag specs in WebKit against the same stack. WebKit takes
|
||||
`*.localhost` to loopback through GLib and ignores `/etc/hosts`, so that container shares
|
||||
Traefik's network namespace, where loopback is Traefik. Playwright's Linux WebKit reads a
|
||||
`writing-mode: vertical-lr` range across, so a vertical slider's geometry is not asserted there.
|
||||
Specs live in `frontend/tests/`. Authentication is saved once, then reused by
|
||||
the desktop and mobile projects. The suite takes its origin from
|
||||
`PLAYWRIGHT_BASE_URL`, and optionally `PLAYWRIGHT_API_URL`, independently of
|
||||
`.env`. Its guard refuses public addresses because tests create and delete
|
||||
flows, dashboards and users.
|
||||
|
||||
The specs are `frontend/tests/*.spec.ts`; `playwright.config.ts` defines `setup` →
|
||||
`chromium` and `mobile`, plus `webkit` when `PLAYWRIGHT_WEBKIT=1` is set. `auth.setup.ts` logs
|
||||
in once into `playwright/.auth/user.json`, which the browser projects reuse; `mobile` runs
|
||||
`mobile.spec.ts` only, on a Pixel 5 at 393px.
|
||||
For manual visual checks, `frontend/scripts/capture-screenshots.mjs` captures
|
||||
both themes at desktop and phone sizes. Set `APP_URL`, `FIRST_SUPERUSER` and
|
||||
`FIRST_SUPERUSER_PASSWORD` in the environment, then run it from `frontend/`
|
||||
with Bun in a Playwright-capable environment. `SCREENSHOT_DIR` selects the
|
||||
output directory. `WEBSITE_URL` optionally adds screenshots of another site.
|
||||
|
||||
The origin comes from `PLAYWRIGHT_BASE_URL`, with the API on it unless `PLAYWRIGHT_API_URL`
|
||||
names another, deliberately never from `.env`. In a deployment checkout that file names the
|
||||
deployment, which would mean driving a browser at the local stack while sending teardown
|
||||
`DELETE`s to the live instance. `tests/guard.ts` runs first and refuses the whole
|
||||
run when either origin resolves outside loopback or the private ranges, because the suite
|
||||
creates and deletes flows, dashboards and users. `PLAYWRIGHT_ALLOW_PUBLIC=1` overrides it.
|
||||
## Lint and documentation
|
||||
|
||||
**Visual check.** From the root, `make verify` logs in and screenshots both themes into
|
||||
`app/frontend/screenshots/{light,dark}/`. It is an alias for `make verify-docker`: the run
|
||||
always happens in the Playwright container, which pins the browser by reading
|
||||
`frontend/package.json`. Running Playwright on the host is unsupported — the workspace pins a
|
||||
browser revision whose install is incomplete. The domain comes off the running stack (the
|
||||
app container's Traefik rule), so the shots are of whatever `make dev` last started.
|
||||
`make lint` checks Python with Ruff and mypy, and the frontend with Biome.
|
||||
`make format-frontend` applies frontend formatting. Install the repository's
|
||||
commit hooks with `uv run --project backend prek install` after `make install`.
|
||||
The backend hook also regenerates the frontend API client; run
|
||||
`make generate-client` to request that explicitly.
|
||||
|
||||
## Lint, hooks and the generated client
|
||||
|
||||
`make lint` runs ruff, mypy and biome, and only reports — it is what the pre-commit hook and
|
||||
CI run, so neither rewrites the tree. `make format-frontend` is the writing half, applying
|
||||
biome's fixes. `make hooks` at the root installs the pre-commit hooks in both stacks;
|
||||
`.pre-commit-config.yaml` holds them. One is worth knowing about:
|
||||
`generate-frontend-sdk` regenerates the SPA's API client from the backend's OpenAPI schema
|
||||
whenever anything under `backend/` changes, so a route change reaches the frontend without
|
||||
being asked. `make generate-client` forces it.
|
||||
|
||||
Run the root's `make design-check` before pushing anything that touches the design tokens,
|
||||
`lib/motion.ts`, `components.json` or the font asset — the design system is a duplication
|
||||
contract between the two frontends, and that target is the only thing checking it.
|
||||
Read [the design reference](DESIGN-GUIDELINES.md) before UI work.
|
||||
`make docs` builds the documentation, and `make docs-serve` opens a preview.
|
||||
|
||||
## CI
|
||||
|
||||
`.gitea/workflows/` runs four jobs:
|
||||
`test-backend.yml` (pytest on Python 3.12 and 3.14, plus a check that both wheels build), `playwright.yml` (the e2e suite in
|
||||
two shards, one compose project each, the browser loading the SPA from the `backend`
|
||||
service), `pre-commit.yml` (the hooks above, plus strict mypy, which they do not cover) and
|
||||
`test-compose.yml` (a smoke test that the production image actually comes up). All but
|
||||
`test-backend.yml` write `.env` from `.env.example` first. The backend suite needs none.
|
||||
Workflows in `.gitea/workflows/` check backend tests and wheel builds,
|
||||
Playwright, lint, documentation, and a Compose image smoke test. They use
|
||||
this repository's files and `.env.example`; no parent workspace is required.
|
||||
|
||||
Reference in New Issue
Block a user