Make documentation standalone and generalize examples

This commit is contained in:
Melvin Strobl
2026-10-25 14:43:20 +01:00
parent 13bf0f76d7
commit bdb18b99da
57 changed files with 553 additions and 481 deletions
+4 -5
View File
@@ -1,6 +1,5 @@
# Template for the app stack's .env. The workspace root's scripts/setup.sh copies # Template for the app stack. `make init` copies this to .env and generates
# this file to .env on first run and then keeps the shared keys in sync with the # initial secrets. Keep real credentials in .env, which is ignored by git.
# root .env. Secrets stay as `changethis` here and are generated locally.
# Domain Traefik routes on and acquires TLS certificates for. # Domain Traefik routes on and acquires TLS certificates for.
DOMAIN=localhost DOMAIN=localhost
@@ -19,7 +18,7 @@ STACK_NAME=fluksio-app
# only matter to a dashboard served from elsewhere, such as the Vite server. # only matter to a dashboard served from elsewhere, such as the Vite server.
BACKEND_CORS_ORIGINS="http://localhost,http://localhost:5173,http://app.localhost,https://localhost,https://localhost:5173" BACKEND_CORS_ORIGINS="http://localhost,http://localhost:5173,http://app.localhost,https://localhost,https://localhost:5173"
SECRET_KEY=changethis SECRET_KEY=changethis
FIRST_SUPERUSER=admin@fluksio.com FIRST_SUPERUSER=admin@example.com
FIRST_SUPERUSER_PASSWORD=changethis FIRST_SUPERUSER_PASSWORD=changethis
# Emails # Emails
@@ -65,7 +64,7 @@ MCP_ENABLED=true
# under Admin. # under Admin.
SIGNUP_ENABLED=false SIGNUP_ENABLED=false
# Local time for schedules; set from the root .env by scripts/setup.sh. # Timezone for schedules. Set this to the timezone your flows should use.
TZ=UTC TZ=UTC
# A camera's live view between networks: STUN and TURN servers, comma # A camera's live view between networks: STUN and TURN servers, comma
+1 -2
View File
@@ -1,7 +1,6 @@
name: pre-commit name: pre-commit
# Gitea Actions. `uses:` is resolved against the instance's action mirror, the # Gitea Actions resolves `owner/action@tag` through its configured action source.
# same convention the sibling N3XD repositories use — plain `owner/action@tag`.
on: on:
push: push:
branches: branches:
+1 -4
View File
@@ -1,9 +1,6 @@
name: Compose Smoke Test name: Compose Smoke Test
# Lives in this repository rather than the workspace root: the stack it starts # Uses only this repository's Compose files and .env.example.
# is entirely app-local (docker/compose*.yml + .env.example). A root workflow
# would additionally need a token to check out the private submodules and would
# have to run scripts/setup.sh to generate secrets, for the same coverage.
on: on:
push: push:
branches: branches:
+2 -2
View File
@@ -6,7 +6,7 @@ node_modules/
/blob-report/ /blob-report/
/playwright/.cache/ /playwright/.cache/
# Generated locally by the workspace root's scripts/setup.sh; never committed. # Local configuration and credentials; never committed.
.env .env
secrets/ secrets/
@@ -20,7 +20,7 @@ __pycache__/
*.pem *.pem
*.key *.key
# Output of `make verify` (root Makefile). # Screenshots generated by the frontend capture scripts.
/frontend/screenshots/ /frontend/screenshots/
*.tsbuildinfo *.tsbuildinfo
+198
View File
@@ -0,0 +1,198 @@
# Fluksio design guidelines
UI reference for the app, its dashboard editor and documentation site.
Paths below are relative to this repository.
## Design language
Use neutral white and near-black surfaces with a slate-blue primary accent.
Reserve terracotta for one secondary action per view. `--card` is the main
surface. Separate surfaces with borders and elevation. Controls are pills;
cards and panels use a 16px radius.
## Colour tokens
| Token | Light | Dark | Use |
|---|---|---|---|
| `--background` | `#ffffff` | `#0a0a0a` | page |
| `--foreground` | `#333232` | `#f5f5f5` | body text |
| `--card` / `--popover` | `#ffffff` | `#1a1a1a` | dominant surface; dark lifts above the page |
| `--card-foreground` / `--popover-foreground` | `#333232` | `#f5f5f5` | text on those surfaces |
| `--primary` | `#4a7189` | `#7ba3b8` | primary fills, active state, focus ring |
| `--primary-foreground` | `#ffffff` | `#0a0a0a` | text on `--primary` |
| `--primary-nested` | `#152128` | `#345160` | a reading drawn inside a `--primary` fill |
| `--brand-secondary` | `#de8f6e` | `#e5a184` | the one secondary call-to-action per view |
| `--brand-secondary-foreground` | `#333232` | `#0a0a0a` | text on `--brand-secondary` |
| `--secondary` / `--muted` | `#f2f2f2` | `#232323` | recessed surfaces |
| `--secondary-foreground` | `#333232` | `#f5f5f5` | text on those |
| `--muted-foreground` | `#6b6b6b` | `#a3a3a3` | secondary text, section headers |
| `--accent` | `#f5f5f5` | `#262626` | hover / selected surface |
| `--accent-foreground` | `#333232` | `#f5f5f5` | text on `--accent` |
| `--destructive` | `#b94d4d` | `#cf6666` | destructive actions, failing nodes |
| `--status-success` | `#5e8b6d` | `#87b596` | a node's last run succeeded |
| `--status-warning` | `#b45309` | `#e8a33d` | a node that works, but degraded, such as a device that is offline |
| `--chart-1` … `--chart-8` | see [Chart palette](#chart-palette) | | chart series, in slot order |
| `--border` / `--input` | `#e4e4e4` | `#2a2a2a` | hairlines, field borders |
| `--ring` | `#4a7189` | `#7ba3b8` | focus ring; always equals `--primary` |
| `--sidebar*` | mirrors card / accent / primary | | sidebar surface set |
The identity blue `#59849b` stays in the wordmark. Use `--primary` for controls
so text on primary fills meets contrast requirements. `--primary-nested` is
for stacked readings inside a primary fill, separated by a gutter.
- Use tokens in UI components. Custom SVG glyphs may use `currentColor`.
- Keep `--sidebar-accent` equal to `--accent`.
- Use one `--brand-secondary` action per view.
- Name every status in text. Running uses `--primary`, success
`--status-success`, failure `--destructive`, and degraded
`--status-warning`. Idle has no indicator.
- Home's brain graph uses primary discs as its identity. Its outer ring uses
`--foreground` for runnable nodes and terracotta for flow faults. Failed
nodes keep a terracotta disc until acknowledged. Persistent faults keep
their labels visible.
### Chart palette
| Slot | Hue | Light | Dark |
|---|---|---|---|
| 1 | `--primary` | `#4a7189` | `#7ba3b8` |
| 2 | orange | `#e26f00` | `#d26600` |
| 3 | violet | `#6f49ca` | `#8d71e6` |
| 4 | green | `#00721b` | `#55c15e` |
| 5 | orchid | `#c66ae1` | `#dd8bf6` |
| 6 | ochre | `#966a00` | `#ae7c00` |
| 7 | teal | `#00a093` | `#00c0b1` |
| 8 | magenta | `#b32474` | `#bf4483` |
Take colours in slot order. Line charts stop at five series and show a legend
when there is more than one. Reserve red, sage green and amber for status.
The terminal UI uses the terminal's named ANSI colours instead.
## The dashboard is exempt
User-defined dashboards can choose `fluksio`, `material` or `glass` under
`frontend/src/components/Dashboard/ui/`. The default `fluksio` look follows
these guidelines. Other looks may change surfaces and motion.
Behavior, keyboard handling and accessibility stay in `ui/core/`.
A dashboard palette overrides tokens on its own canvas, with colours assigned
by position. It must preserve contrast, named status, 44px touch targets and
reduced-motion behavior. Fault and success colours keep their meanings.
Widgets use container queries to follow their tile size.
Dashboard motion presets live in `ui/core/motion.ts`; the app shell uses
`frontend/src/lib/motion.ts`.
## Typography
Use the self-hosted Inter variable font at
`frontend/public/fonts/inter-latin-variable.woff2` for body text, headings and
wordmarks. `.font-display` selects it for display text. Use the system
monospace stack through `font-mono` for code, message names and payloads.
Use Tailwind's size scale. Arbitrary `text-[…]` sizes are banned and checked
by the `no-arbitrary-font-size` hook.
Section headers use:
```text
text-xs font-medium uppercase tracking-[0.5px] text-muted-foreground
```
## Radius
| Utility | Value | Use |
|---|---|---|
| `rounded-full` | pill | buttons, inputs, toggles, chips, badges |
| `rounded-sm` | 8px | checkboxes, small chips, nested items |
| `rounded-md` | 12px | menus, popovers, tooltips, textareas |
| `rounded-lg` | 16px | cards, panels, dialogs, alerts, sidebar |
| `rounded-xl` | 28px | large containers |
## Elevation
Use `shadow-e1` for resting cards, `shadow-e2` for floating surfaces, and
`shadow-e3` for modals. Avoid arbitrary or stock shadow utilities on surfaces.
Borders use `border border-border`, or `border-input` on fields.
## Motion
Use the `motion` package with presets from `frontend/src/lib/motion.ts`.
Keep JavaScript timings aligned with the CSS easing and duration tokens.
- Easing is `emphasized` for entering or expressive movement, `standard` for
neutral transitions.
- Durations are 150, 200 and 300 ms. One-shot activity feedback uses 500 ms.
- Variants are `fadeIn`, `scaleIn`, `slideUp`, `overlayFade` and `listStagger`.
- Wrap the app in `<MotionConfig reducedMotion="user">`.
## Microcopy
Keep captions short. Put explanations in `InfoTip`, `PanelSection.help` or
`FieldLabel.help`. A tooltip holds one or two sentences; longer instructions
belong in the docs. Help must work by tap as well as hover.
Visible supporting text describes state, such as an empty list, a stopped
flow or an error. A page may have one descriptive line below its heading.
Say what a control does. Keep design rationale in repository documentation.
Avoid em dashes in UI strings. Empty-value placeholders and structural
separators are allowed. The `no-em-dash` hook checks these rules, with
separators listed in `frontend/scripts/no-em-dash.allow.json`.
## Iconography
Use Lucide. Custom glyphs belong in `frontend/src/components/icons/` as 24×24
SVGs with no fill, `currentColor` strokes, weight 2 and round caps.
## Shells
The authenticated app uses a `bg-card` region with a floating translucent
sidebar. The flow editor floats its controls over the node canvas. Both use
`bg-card/80 backdrop-blur-md`, a token border and `shadow-e2`.
## Responsive
Use `md` at 768px as the mobile boundary. Keep `useIsMobile()` and Tailwind's
`md:` usage aligned, with stock breakpoints.
1. Keep the page within the viewport. Give wide tables and charts their own
`overflow-x-auto` region. Use `min-w-0` on flex and grid children when
content could stretch the page. `frontend/tests/mobile.spec.ts` checks this.
2. Cap floating controls with `max-w-[calc(100vw-2rem)]` and allow wrapping.
3. Use 44px touch targets below `md`, 32px above. Use `icon-touch`, `sm-touch`
or matching responsive utilities.
4. Turn floating panels into full-screen sheets on phones with the shared
`Flow/SidePanel`. Gate canvas side insets behind `md:`.
5. Provide tap or visible alternatives to hover actions.
6. Keep dragging, placing and resizing on desktop. Flow layout is automatic,
left-to-right on desktop and top-to-bottom on phones. Rectangular dashboard
grids stack to one column on phones.
Use `grid gap-* sm:grid-cols-2 lg:grid-cols-N` for card grids and
`p-4 sm:p-6 md:p-8` for page gutters.
## Overlay surfaces & content chips
Floating surfaces use `bg-card/80 backdrop-blur-md`, a token border and
`shadow-e2`. Use token alpha for translucency; panel opacity also fades text.
Content chips are transparent at rest and fill on hover or selection.
Use `hover:bg-accent/50` where hover and selection need distinct states.
Segmented controls use one bordered pill with no dividers, transparent
segments and `bg-accent` for the selected segment.
## UI configuration
`frontend/components.json` uses shadcn style `new-york`, CSS variables and
Lucide icons. Components under `frontend/src/components/ui/` are owned by
this repository. Update token definitions, component usage and this reference
together when changing the design.
## UX
- Keep common actions easy to reach without context menus.
- Apply theme through the `.dark` class on `<html>` and the pre-paint theme
script so the first frame uses the chosen theme.
- A dashboard subtree can use `.theme-light` or `.theme-dark` independently.
Menus and tooltips rendered outside it must carry that theme class too.
+8 -42
View File
@@ -1,45 +1,11 @@
# Design system (Fluksio) # Design system
`app` (dashboard SPA + API) and `index` (marketing site) are **separate Read [DESIGN-GUIDELINES.md](DESIGN-GUIDELINES.md) before changing the UI.
repos** that share one visual design. The full, canonical spec — palette, It covers tokens, typography, components, motion and responsive behavior.
radius, elevation, motion, component map, and shell patterns — lives in the
parent repo's [`DESIGN-GUIDELINES.md`](../DESIGN-GUIDELINES.md). Read it before
any UI work.
## Sync rule (read before touching tokens) The frontend owns its CSS tokens in `frontend/src/index.css`, motion presets
in `frontend/src/lib/motion.ts`, and shadcn configuration in
`frontend/components.json`.
The `@theme inline`, `:root`, and `.theme-dark` blocks in `app/frontend/src/index.css` User-defined dashboards have their own appearance and motion settings.
and `index/src/index.css` **must be kept byte-identical**, and See [Dashboards and devices](docs/interface/dashboards.md) for that contract.
`frontend/src/lib/motion.ts` is duplicated verbatim. Any palette / radius /
elevation / motion-token change is made in *both* repos in the same change.
There is intentionally no shared npm package — the duplicated blocks are the
sync point. Components (`src/components/ui/*`) are copied per repo (the shadcn
model) and may diverge; only the tokens and motion presets must match.
Both repos share `components.json`: style `new-york`, `cssVariables: true`,
icon library `lucide`.
Verify with `make design-check` from the workspace root.
## Shell
The authenticated area is a `bg-card` region (`SidebarProvider`) with a floating
translucent `AppSidebar` (`variant="floating"`, `bg-card/80 backdrop-blur-md`).
The flow editor floats its chrome over a full-bleed node canvas using the same
frosted-surface model. See DESIGN-GUIDELINES.md → Shells and → Overlay surfaces
& content chips.
## Mobile
One breakpoint, `md` (768px), and a phone inspects rather than arranges: no
dragging, no placing, no resizing. The rules that keep it that way — and the
"nothing scrolls horizontally" check that enforces them — are in
DESIGN-GUIDELINES.md → Responsive.
## Dashboards
A user-defined dashboard deliberately steps outside the shared design system:
it has two complete component sets of its own (`glass` and `material`) and
takes its colours from the document rather than from the tokens. See
DESIGN-GUIDELINES.md → The dashboard is exempt. The shells, the flow editor and
every admin screen are unaffected.
+5 -7
View File
@@ -1,15 +1,13 @@
# ─── Fluksio app Makefile ─── # ─── Fluksio app Makefile ───
# Convenience targets for development, testing, linting, and deployment. # Convenience targets for development, testing, linting, and deployment.
# The workspace root delegates to these (see ../Makefile).
.PHONY: init env-check dev-utils dev dev-local dev-lan rebuild-frontend up down update install dev-backend dev-frontend \ .PHONY: init env-check dev-utils dev dev-local dev-lan rebuild-frontend up down update install dev-backend dev-frontend \
generate-client sync-example test test-backend test-checks test-stack test-frontend test-webkit soak \ generate-client sync-example test test-backend test-checks test-stack test-frontend test-webkit soak \
bench-startup bench-engine lint lint-backend lint-frontend format-frontend umami build docs docs-serve clean help bench-startup bench-engine lint lint-backend lint-frontend format-frontend umami build docs docs-serve clean help
COMPOSE_ROOT := $(CURDIR) COMPOSE_ROOT := $(CURDIR)
# Explicit project name keeps this stack isolated from the sibling website # An explicit project name avoids collisions with other Compose stacks
# stack (otherwise both default to "docker", the directory their compose # whose configuration also lives in a directory named "docker".
# files live in).
COMPOSE_PROJECT := fluksio-app COMPOSE_PROJECT := fluksio-app
# Compose interpolation needs the project-local .env before reading compose.yml. # Compose interpolation needs the project-local .env before reading compose.yml.
COMPOSE := docker compose -p $(COMPOSE_PROJECT) --env-file $(COMPOSE_ROOT)/.env COMPOSE := docker compose -p $(COMPOSE_PROJECT) --env-file $(COMPOSE_ROOT)/.env
@@ -77,7 +75,7 @@ dev-utils: ## Start only the utility containers (proxy, mailcatcher)
dev: ## Start the full dev stack (includes a local Traefik proxy) dev: ## Start the full dev stack (includes a local Traefik proxy)
$(COMPOSE_DEV) up --build $(COMPOSE_DEV) up --build
dev-local: ## Start the local stack at http://app.localhost (also run by the root `make dev`) dev-local: ## Start the local stack at http://app.localhost
DOMAIN=$${DOMAIN:-localhost} ENVIRONMENT=$${ENVIRONMENT:-local} \ DOMAIN=$${DOMAIN:-localhost} ENVIRONMENT=$${ENVIRONMENT:-local} \
$(COMPOSE_LOCAL) up --build -d --remove-orphans proxy backend docs mailcatcher $(COMPOSE_LOCAL) up --build -d --remove-orphans proxy backend docs mailcatcher
@@ -122,7 +120,7 @@ umami: ## Provision + start the optional Umami site-analytics service (idempote
# start when its session-signing secret is unset or still the placeholder. # start when its session-signing secret is unset or still the placeholder.
@secret=$$(grep -E '^UMAMI_APP_SECRET=' $(COMPOSE_ROOT)/.env 2>/dev/null | head -1 | cut -d= -f2-); \ @secret=$$(grep -E '^UMAMI_APP_SECRET=' $(COMPOSE_ROOT)/.env 2>/dev/null | head -1 | cut -d= -f2-); \
if [ -z "$$secret" ] || [ "$$secret" = "changethis" ]; then \ if [ -z "$$secret" ] || [ "$$secret" = "changethis" ]; then \
echo "ERROR: UMAMI_APP_SECRET is empty or 'changethis' in .env — run the workspace root's scripts/setup.sh (or set it: openssl rand -hex 32) before 'make umami'." >&2; \ echo "ERROR: UMAMI_APP_SECRET is empty or 'changethis' in .env — set it in .env with 'openssl rand -hex 32' before 'make umami'." >&2; \
exit 1; \ exit 1; \
fi fi
# --no-recreate: reuse an already-running db instead of recreating it under # --no-recreate: reuse an already-running db instead of recreating it under
@@ -135,7 +133,7 @@ umami: ## Provision + start the optional Umami site-analytics service (idempote
pw=$$(grep -E '^UMAMI_DB_PASSWORD=' $(COMPOSE_ROOT)/.env 2>/dev/null | head -1 | cut -d= -f2-); \ pw=$$(grep -E '^UMAMI_DB_PASSWORD=' $(COMPOSE_ROOT)/.env 2>/dev/null | head -1 | cut -d= -f2-); \
db=$$(grep -E '^UMAMI_DB=' $(COMPOSE_ROOT)/.env 2>/dev/null | head -1 | cut -d= -f2-); db=$${db:-umami}; \ db=$$(grep -E '^UMAMI_DB=' $(COMPOSE_ROOT)/.env 2>/dev/null | head -1 | cut -d= -f2-); db=$${db:-umami}; \
if [ -z "$$pw" ] || [ "$$pw" = "changethis" ]; then \ if [ -z "$$pw" ] || [ "$$pw" = "changethis" ]; then \
echo "ERROR: UMAMI_DB_PASSWORD is empty or 'changethis' in .env — run the workspace root's scripts/setup.sh before 'make umami'." >&2; \ echo "ERROR: UMAMI_DB_PASSWORD is empty or 'changethis' in .env — set it in .env with 'openssl rand -hex 32' before 'make umami'." >&2; \
exit 1; \ exit 1; \
fi; \ fi; \
$(COMPOSE_PROD) exec -T db psql -v ON_ERROR_STOP=1 -U "$$pguser" -d postgres \ $(COMPOSE_PROD) exec -T db psql -v ON_ERROR_STOP=1 -U "$$pguser" -d postgres \
+17 -15
View File
@@ -1,11 +1,11 @@
# Fluksio App # Fluksio App
Fluksio is a node-based automation software that brings trust and reliability to your flow. Connect a few nodes, give their messages names, and let Fluksio handle the
It just works and looks good. running. Use live flows for automation and batch runs for experiments, with
dashboards to see what happened.
The core of Fluksio: a node-based, test-driven automation software built to scale. This This repository holds the FastAPI backend, flow engine and dashboard SPA.
repo holds the FastAPI backend, the flow engine, and the dashboard SPA, built into one The Docker image serves all three on one port.
image that serves all three on one port.
## Layout ## Layout
@@ -23,30 +23,32 @@ scripts/ generate-client.sh, test.sh
## Install without Docker ## Install without Docker
Get started quickly by running Start an engine with Python 3.12 or newer on Linux or macOS:
```sh ```sh
pip install fluksio pip install fluksio
fluksio serve fluksio serve
``` ```
Then, head over to [fluksio.com](https://fluksio.com), sign up and add a new instance. For the optional browser portal, sign up at [fluksio.com](https://fluksio.com)
Using the code provided, run and add an instance. Use the code it gives you:
```sh ```sh
fluksio enroll <code> fluksio enroll <code>
``` ```
and you're ready to rock. Your engine also works without enrollment. A machine that only runs nodes
for another engine needs just the worker package:
Fluksio is distributed at it's heart.
A machine that should only *run nodes* for an engine elsewhere installs less:
```sh ```sh
pip install fluksio-worker pip install fluksio-worker
FLUKSIO_WORKER_TOKEN=… fluksio-worker --url wss://api.fluksio.com/api/v1/workers/attach --labels gpu FLUKSIO_WORKER_TOKEN='<worker-token>' fluksio-worker \
--url wss://fluksio.example.com/api/v1/workers/attach --labels gpu
``` ```
Replace the example address with your engine's URL and create a token as
described in [Remote workers](docs/code/workers.md).
## Getting started ## Getting started
Needs Docker with Compose v2, on x86-64 or arm64 such as a Raspberry Pi 4 or 5. Needs Docker with Compose v2, on x86-64 or arm64 such as a Raspberry Pi 4 or 5.
@@ -82,7 +84,8 @@ make lint # ruff + mypy + biome
make generate-client # regenerate the frontend SDK from the OpenAPI schema make generate-client # regenerate the frontend SDK from the OpenAPI schema
``` ```
`make help` lists every target. `make help` lists every target. See [Development](development.md) for stack
options and checks.
## Documentation ## Documentation
@@ -96,7 +99,6 @@ make docs # static build into ./site
It is served at `docs.${DOMAIN}` by the `docs` service in `docker/compose.yml`, It is served at `docs.${DOMAIN}` by the `docs` service in `docker/compose.yml`,
and `.gitea/workflows/docs.yml` builds it with `--strict` on every push. and `.gitea/workflows/docs.yml` builds it with `--strict` on every push.
## License ## License
Copyright (C) 2026 Melvin Strobl - GNU Affero General Public License v3.0 or Copyright (C) 2026 Melvin Strobl - GNU Affero General Public License v3.0 or
+1 -1
View File
@@ -66,7 +66,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
$(for e in $(echo "$EXTRAS" | tr , ' '); do printf -- '--extra %s ' "$e"; done) $(for e in $(echo "$EXTRAS" | tr , ' '); do printf -- '--extra %s ' "$e"; done)
# Connectors are ordinary installed packages found through the # Connectors are ordinary installed packages found through the
# `fluksio.node_types` entry point. `make connectors` builds them into here; # `fluksio.node_types` entry point. Place built connector wheels here;
# an image with none is the normal case. # an image with none is the normal case.
COPY ./backend/connector-wheels /tmp/connector-wheels COPY ./backend/connector-wheels /tmp/connector-wheels
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
+33 -34
View File
@@ -1,60 +1,59 @@
# Fluksio # Fluksio
Fluksio is a node-based automation software that brings trust and reliability to your flow. Turn Python functions into connected nodes, run experiments, and keep their
It just works and looks good. metrics and artifacts together. Start an engine with Python 3.12 or newer
Get started by running on Linux or macOS:
```sh ```sh
pip install fluksio pip install fluksio
fluksio serve fluksio serve
``` ```
and you're ready to go. The first start prints an admin password and opens a terminal dashboard.
For a browser dashboard, connect the instance to the optional
[portal](https://docs.fluksio.com/interface/portal/).
## For data science ## For data science
You can turn your existing data science project into a flow by decorating your functions with `@node` ... Save this small experiment as `study.py`. No dataset or GPU needed:
```python ```python
# myresearch/train.py import math
import fluksio
from fluksio import Port, node from fluksio import Flow, Port, node
@node( @node(
requires=["dataset", Port("lr", "float")], requires=[Port("lr", "float")],
provides=[Port("loss", "float", stream=True), Port("weights", "artifact")], provides=[Port("loss", "float", stream=True), Port("final_loss", "float")],
worker="gpu", worker_policy="prefer",
) )
def fit(dataset, lr, epochs=25): def fit(lr, epochs=25):
for epoch in range(epochs): for epoch in range(epochs):
loss = step(...) loss = math.exp(-lr * (epoch + 1))
yield {"loss": loss} # published as it happens, kept as a series yield {"loss": loss}
return {"weights": fluksio.save_artifact("weights.pt")} return {"final_loss": loss}
train = Flow(
"train",
nodes=[fit],
inputs=[Port("lr", "float", initial=0.01)],
outputs=["final_loss"],
)
``` ```
... and passing them to a `Flow`: Matching message names connect the nodes and determine their order. After
syncing the project, start a run:
```python
# myresearch/pipeline.py
from fluksio import Flow, Port
from myresearch.data import prepare
from myresearch.evaluate import evaluate
from myresearch.train import fit
train = Flow("train", nodes=[prepare, fit, evaluate],
inputs=[Port("lr", "float", initial=0.01)], outputs=["score"])
```
Fluksio will automatically infer the order of nodes based on the inputs and outputs you defined.
When everything is set, you can launch your first run as follows:
```sh ```sh
fluksio sync study
fluksio run train --lr 0.05 --wait fluksio run train --lr 0.05 --wait
``` ```
Checkout our [documentation](https://docs.fluksio.com/getting-started/data-science/) for more infos. Each yield records a point on the loss curve. Replace the toy calculation
with your own training code when you're ready. For artifacts and comparisons, try the
[data science guide](https://docs.fluksio.com/getting-started/data-science/).
## Some other features ## More to explore
- **Flows**: typed messages between nodes, wired by name, edited on a canvas - **Flows**: typed messages between nodes, wired by name, edited on a canvas
or declared in code. Every change is a commit in a git repository you own. or declared in code. Every change is a commit in a git repository you own.
@@ -65,8 +64,8 @@ Checkout our [documentation](https://docs.fluksio.com/getting-started/data-scien
- **Remote workers**: `pip install fluksio-worker` on the GPU box; it dials - **Remote workers**: `pip install fluksio-worker` on the GPU box; it dials
*out* over one websocket, so nothing there has to be reachable. *out* over one websocket, so nothing there has to be reachable.
Fluksio can also be used for facility automation. For sensors, devices and live automation, try the
Visit us on [fluksio.com](https://fluksio.com) or go straight to our [documentation](https://docs.fluksio.com). [facility automation guide](https://docs.fluksio.com/getting-started/facility-automation/).
## License ## License
+2 -3
View File
@@ -1,3 +1,2 @@
Connector wheels built by `make connectors` at the workspace root land here and Place connector wheels here before building the backend image. They are
are installed into the image. Kept empty in git: a connector is an installed installed into the engine environment. This directory stays empty in git.
package, not part of the app.
+1 -1
View File
@@ -38,7 +38,7 @@ router = APIRouter(prefix="/cloud", tags=["cloud"])
class EnrollBody(BaseModel): class EnrollBody(BaseModel):
#: Where the portal's API lives, e.g. https://hub.fluksio.com #: Where the portal's API lives, e.g. https://portal.example.com
portal_url: str = Field(min_length=1, max_length=255) portal_url: str = Field(min_length=1, max_length=255)
claim_code: str = Field(min_length=1, max_length=32) claim_code: str = Field(min_length=1, max_length=32)
+1 -1
View File
@@ -50,7 +50,7 @@ def socket_token(headers: Mapping[str, str], ticket: str = "", token: str = "")
`token` is the query parameter clients sent before the other two existed. `token` is the query parameter clients sent before the other two existed.
It is read for one more release so an engine can be upgraded before its It is read for one more release so an engine can be upgraded before its
workers; remove it after that (see NOTEPAD). workers. New clients should use the Authorization header.
""" """
header = headers.get("authorization") or "" header = headers.get("authorization") or ""
if header.lower().startswith("bearer "): if header.lower().startswith("bearer "):
+2 -5
View File
@@ -205,11 +205,8 @@ class ExecutionService:
def _sleep_until_due(self) -> bool: def _sleep_until_due(self) -> bool:
"""Wait for the soonest deadline, the housekeeping cap, or a new one. """Wait for the soonest deadline, the housekeeping cap, or a new one.
A fixed poll here made every delayed item late by 0-1000ms whatever the Sleeping until the deadline avoids adding a fixed polling delay to
load — on a rollershutter driven for a measured 26 seconds, 2-4% of its timed work. New deadlines wake the timer so earlier work is not missed.
travel every time, accumulating in the position its node believes it is
at. Sleeping to the deadline instead leaves a wake-up and a promotion,
which is milliseconds.
""" """
self._timer_wake.clear() self._timer_wake.clear()
try: try:
+1 -1
View File
@@ -150,7 +150,7 @@ class WebSocketNode(Node):
"A WebSocket node needs either inputs (to send) or outputs (to listen)" "A WebSocket node needs either inputs (to send) or outputs (to listen)"
) )
# ponytail: one direction per node, as mqtt and http. Inputs and # ponytail: one direction per node, as mqtt and http. Inputs and
# outputs over one socket is the NOTEPAD follow-up. # outputs over one socket is not supported yet.
self.mode = ( self.mode = (
WebSocketNode.Mode.SENDER if requires else WebSocketNode.Mode.LISTENER WebSocketNode.Mode.SENDER if requires else WebSocketNode.Mode.LISTENER
) )
+5 -11
View File
@@ -79,17 +79,12 @@ _NUMBERS = {w: n for n, w in enumerate(_ONES)} | {
def _plain(command: str) -> str: def _plain(command: str) -> str:
"""The command as the model was trained on it: lower case, no sentence marks. """Normalize case and sentence punctuation to match the training format."""
A transcript's "Turn on the kitchen light." reads less sure than the same
words plain, 0.80 against 0.99 with every area shown.
"""
return re.sub(r"[.,!?;:]+(?=\s|$)", "", command).strip().lower() return re.sub(r"[.,!?;:]+(?=\s|$)", "", command).strip().lower()
def _aliased(prompt: str, aliases: dict[str, str]) -> str: def _aliased(prompt: str, aliases: dict[str, str]) -> str:
"""``prompt`` with the house's words in wren's, whole words in any case: """Replace whole-word aliases with target names, ignoring case."""
"the traverse" as "the living room light", "page too" as "patio"."""
if not aliases: if not aliases:
return prompt return prompt
words = {k.lower(): v for k, v in aliases.items()} words = {k.lower(): v for k, v in aliases.items()}
@@ -103,8 +98,7 @@ def _aliased(prompt: str, aliases: dict[str, str]) -> str:
def _digits(command: str) -> str: def _digits(command: str) -> str:
"""Number words as digits: "in five minutes" kept its delay only as "in 5 """Convert number words to digits, as expected by the model."""
minutes", and "twenty two degrees" read as 21."""
tens, ones = "|".join(_TENS), "|".join(_ONES[1:10]) tens, ones = "|".join(_TENS), "|".join(_ONES[1:10])
command = re.sub( command = re.sub(
rf"\b({tens})[ -]({ones})\b", rf"\b({tens})[ -]({ones})\b",
@@ -266,8 +260,8 @@ class WrenNode(Node):
) )
aliases: dict[str, str] = Field( aliases: dict[str, str] = Field(
default_factory=dict, default_factory=dict,
description="Your own words, and words heard wrong, in the model's: " description="Map your own names to the model's target names: "
'{"traverse": "living room light", "page too": "patio"}.', '{"desk lamp": "office light"}.',
) )
examples: list[dict[str, Any]] = Field( examples: list[dict[str, Any]] = Field(
default_factory=list, default_factory=list,
+3 -7
View File
@@ -6,11 +6,8 @@ listens any more. The supervisor restarts those tasks with a growing delay and
stands a flow down when one keeps failing, because a flow crash-looping every stands a flow down when one keeps failing, because a flow crash-looping every
second is worse than a flow that is visibly stopped. second is worse than a flow that is visibly stopped.
Standing down is a rest, not a verdict. It used to be the end of the task: the Quarantined tasks retry after a longer cooldown with a fresh restart budget.
flow stayed dead until somebody published it or the engine restarted. On a This lets unattended flows recover from extended connection failures.
house that runs unattended that turns a broker rebooting at four in the morning
into a hole in the record until someone notices — so the rest expires, the
budget comes back, and the task tries again on a much longer clock.
Deliberately a plain task registry rather than a `TaskGroup`: a group cancels Deliberately a plain task registry rather than a `TaskGroup`: a group cancels
its siblings when one member fails, which is the opposite of what supervision its siblings when one member fails, which is the opposite of what supervision
@@ -38,8 +35,7 @@ BACKOFF = settings.FLOW_RESTART_BACKOFF
# Per task rather than per flow: a flow whose subscriber and its keepalive # Per task rather than per flow: a flow whose subscriber and its keepalive
# publisher both go down with the same broker spends one shared budget twice as # publisher both go down with the same broker spends one shared budget twice as
# fast as it was sized for, and gives up before the backoff above has reached # fast as it was sized for, and gives up before the backoff above has reached
# its longest step. That is what took the house's inverter readings out for # its longest step.
# nearly five hours — six crashes across two tasks inside 41 seconds.
FAILURE_BUDGET = 5 FAILURE_BUDGET = 5
FAILURE_WINDOW = 300.0 FAILURE_WINDOW = 300.0
# How long a task sits out before its budget is handed back. Minutes rather # How long a task sits out before its budget is handed back. Minutes rather
+2 -3
View File
@@ -21,9 +21,8 @@ from urllib.parse import urlsplit, urlunsplit
#: the engine's own supervision backs off with. #: the engine's own supervision backs off with.
BACKOFF = (1.0, 5.0, 30.0, 60.0) BACKOFF = (1.0, 5.0, 30.0, 60.0)
#: How long a read waits before looking at `stop`. The socket is otherwise #: How long a read waits before checking `stop`, so an idle connection does
#: silent for as long as the house is, and a dashboard that is closing should #: not delay closing the dashboard.
#: not wait for the next event to notice.
POLL_S = 1.0 POLL_S = 1.0
+8 -12
View File
@@ -6,11 +6,11 @@ arriving at a subscriber and reaching everything downstream of it. It builds a
pipeline and an execution service in this process, so there is no HTTP, no pipeline and an execution service in this process, so there is no HTTP, no
container and no browser between the measurement and the engine. container and no browser between the measurement and the engine.
Three scenarios, each answering a different question: Four scenarios, each answering a different question:
``throttle`` ``throttle``
What does a rate-limited port cost? A source publishing every 15s behind What does a rate-limited port cost? A source publishes through relays
seven relays, driven at the rate the house's inverter actually sends. while incoming readings exceed its publication rate.
Reports cascades enqueued, node executions and bus events per injection — Reports cascades enqueued, node executions and bus events per injection —
the numbers, not the wall clock, are the point. the numbers, not the wall clock, are the point.
@@ -119,7 +119,7 @@ def node(
def relay( def relay(
node_id: str, src: str, dst: str, interval: float, counters: Counters node_id: str, src: str, dst: str, interval: float, counters: Counters
) -> Node: ) -> Node:
"""A node that republishes what it read — the shape the house is full of. """A relay node that republishes its input.
``src`` and ``dst`` are bare port names: the pipeline qualifies them with ``src`` and ``dst`` are bare port names: the pipeline qualifies them with
the flow, and the node function only ever sees ports. the flow, and the node function only ever sees ports.
@@ -238,9 +238,8 @@ def percentile(values: list[float], p: float) -> float:
def scenario_throttle(args: argparse.Namespace) -> dict[str, Any]: def scenario_throttle(args: argparse.Namespace) -> dict[str, Any]:
"""A rate-limited source behind relays, driven far faster than its window. """A rate-limited source behind relays, driven far faster than its window.
The house's shape: an inverter sending ~200 readings a minute into seven Counts the work performed when rapid inputs are held by a publication
relay nodes, each told to publish every 15s. What should cost seven interval. Relay count and interval come from the benchmark arguments.
publishes a minute is what this counts.
""" """
counters = Counters() counters = Counters()
source = node( source = node(
@@ -285,11 +284,8 @@ def scenario_throttle(args: argparse.Namespace) -> dict[str, Any]:
def scenario_relay(args: argparse.Namespace) -> dict[str, Any]: def scenario_relay(args: argparse.Namespace) -> dict[str, Any]:
"""An *unlimited* source into limited relays, and one node reading them all. """An *unlimited* source into limited relays, and one node reading them all.
The other half of the house's shape, and the one holding it at the source Each input starts a cascade, but rate-limited relay outputs should wake
cannot help with: the readings really do arrive, so a cascade is owed — the consumer only when they publish a new value.
but the relays hold their outputs back, and what reads them has nothing
new to read. That node published 619 messages a minute off inputs that
changed six times.
""" """
counters = Counters() counters = Counters()
source = node("source", lambda params: None, provides=[spec("raw")]) source = node("source", lambda params: None, provides=[spec("raw")])
+3 -9
View File
@@ -1,13 +1,7 @@
"""How long it takes to get a run started, which is the thing Kedro is slow at. """Measure submission and completion latency against a running engine.
A pipeline framework that boots the project per run pays that cost every time: Uses a no-op flow to isolate scheduling overhead. Optionally compare it with
``kedro run`` on a pipeline that does nothing takes about a second, and a sweep an empty Kedro pipeline on the same machine. Run with the dev stack up::
of five hundred configs therefore spends ten minutes doing nothing. Fluksio
answers that by not booting anything — the engine is already up and its workers
already have the code loaded, so submitting is one request.
This measures that claim against a running stack, so it can be checked rather
than asserted. Run it with the dev stack up::
make bench-startup make bench-startup
make bench-startup BENCH_ARGS="--runs 50 --kedro ../path/to/kedro/project" make bench-startup BENCH_ARGS="--runs 50 --kedro ../path/to/kedro/project"
+2 -3
View File
@@ -12,9 +12,8 @@ the work queue — and then takes pieces of the stack away to see what survives.
The redis scenario, which `all` includes, needs `--force`: the Redis it stops The redis scenario, which `all` includes, needs `--force`: the Redis it stops
is the one the whole stack journals to, not a copy of its own. is the one the whole stack journals to, not a copy of its own.
Every docker verb goes through :func:`docker`, which refuses any container that Every Docker command goes through :func:`docker`, which limits operations to
is not part of this project's compose stack. The machine this runs on hosts containers in this project's Compose stack.
unrelated services; nothing here may ever reach them.
The engine's semantic is latest-value-wins with cascades running concurrently, The engine's semantic is latest-value-wins with cascades running concurrently,
so counter equality is not a promise and is not asserted. What is asserted is so counter equality is not a promise and is not asserted. What is asserted is
@@ -1,6 +1,6 @@
"""Who may see a device's camera: superusers only, or every member too. """Who may see a device's camera: superusers only, or every member too.
openpanel ROADMAP C4. ``viewers`` on a device covers its ``image`` and ``viewers`` on a device covers its ``image`` and
``video`` messages, their bytes, frames on the socket and its live view. A ``video`` messages, their bytes, frames on the socket and its live view. A
device credential is bounded by its dashboards instead. device credential is bounded by its dashboards instead.
""" """
+1 -1
View File
@@ -1,6 +1,6 @@
"""Cameras on a device: frames over its socket, demand, and what a screen fetches. """Cameras on a device: frames over its socket, demand, and what a screen fetches.
openpanel ROADMAP C1. A device reports a ``camera`` port that sends only while A device reports a ``camera`` port that sends only while
somebody wants it. Demand is what the viewer sockets want plus what running somebody wants it. Demand is what the viewer sockets want plus what running
flows read, mapped onto the device's on-demand ports and sent to its socket. flows read, mapped onto the device's on-demand ports and sent to its socket.
""" """
+1 -1
View File
@@ -1,6 +1,6 @@
"""Live view: WebRTC signalling relayed between a viewer's socket and a device's. """Live view: WebRTC signalling relayed between a viewer's socket and a device's.
openpanel ROADMAP L1. The server routes by session, holds a device to two The server routes by session, holds a device to two
viewers, ends a session when either socket goes or no answer comes, and hands viewers, ends a session when either socket goes or no answer comes, and hands
out TURN credentials in coturn's REST scheme. out TURN credentials in coturn's REST scheme.
""" """
+1 -7
View File
@@ -1,10 +1,4 @@
"""What the house port needed from the built-in nodes. """Device messaging with MQTT filters, wrapped values and per-message delays."""
Three additions, each with a device behind it: an MQTT filter that actually
routes what it subscribed to, a value lifted out of the object a device wraps
it in, and a wait that differs per message because a shutter takes longer to
come up than to go down.
"""
from fluksio.flow.messages import DType, MessageSpec from fluksio.flow.messages import DType, MessageSpec
from fluksio.flow.nodes import MqttNode from fluksio.flow.nodes import MqttNode
+4 -9
View File
@@ -211,11 +211,8 @@ def test_only_advisory_codes_are_flagged_advisory():
def test_a_producer_that_cannot_run_does_not_strand_its_consumer(): def test_a_producer_that_cannot_run_does_not_strand_its_consumer():
"""One wave, two producers of one consumer, and one of them never ready. """One wave, two producers of one consumer, and one of them never ready.
The house's shape: a battery reading wakes the boiler arbiter and the The ready producer must reach its consumer even when a sibling is
watering-pump trigger in the same wave, and the DMX encoder reads both. The waiting for an input. The consumer can use that sibling's stored value.
pump waits on a pulse that only exists at 02:00, so it is never ready — and
passing over it used to leave the encoder unreachable, which put every
boiler and plug command into its message and no further.
""" """
ran = [] ran = []
@@ -268,10 +265,8 @@ def test_a_producer_that_cannot_run_does_not_strand_its_consumer():
def test_a_quiet_producer_does_not_veto_a_noisy_sibling(): def test_a_quiet_producer_does_not_veto_a_noisy_sibling():
"""Two report-by-exception producers, one consumer, and only one with news. """Two report-by-exception producers, one consumer, and only one with news.
The house reads both boilers into one DMX encoder through `rbe` nodes. The A producer that publishes nothing must not block a shared consumer
kitchen boiler is unchanged almost always, so it returns nothing — and from processing its sibling's new value.
while that held the encoder's dependency count up, the main boiler's
command reached its message and got no further.
""" """
ran = [] ran = []
+5 -5
View File
@@ -50,11 +50,11 @@ def test_a_service_comes_back_and_its_url_follows(monkeypatch):
async def scenario() -> None: async def scenario() -> None:
agent = Agent() agent = Agent()
agent.worker = RemoteWorker( agent.worker = RemoteWorker(
name="amcomp", name="gpu-worker",
labels=["gpu"], labels=["gpu"],
send=agent.send_json, send=agent.send_json,
loop=asyncio.get_running_loop(), loop=asyncio.get_running_loop(),
host="10.0.0.5", host="192.0.2.5",
) )
hub = RemoteWorkerHub() hub = RemoteWorkerHub()
node = ServiceNode( node = ServiceNode(
@@ -78,7 +78,7 @@ def test_a_service_comes_back_and_its_url_follows(monkeypatch):
hub.attach(agent.worker) hub.attach(agent.worker)
await until(lambda: bool(agent.serves())) await until(lambda: bool(agent.serves()))
await until(lambda: url() == "http://10.0.0.5:8110") await until(lambda: url() == "http://192.0.2.5:8110")
first = agent.serves()[0] first = agent.serves()[0]
assert first["argv"] == ["serve", "--port", "8110"] assert first["argv"] == ["serve", "--port", "8110"]
assert health[-1] == "ok" assert health[-1] == "ok"
@@ -88,7 +88,7 @@ def test_a_service_comes_back_and_its_url_follows(monkeypatch):
await until(lambda: url() == "") await until(lambda: url() == "")
assert "degraded" in health[health.index("ok") :] assert "degraded" in health[health.index("ok") :]
await until(lambda: len(agent.serves()) == 2) await until(lambda: len(agent.serves()) == 2)
await until(lambda: url() == "http://10.0.0.5:8110") await until(lambda: url() == "http://192.0.2.5:8110")
# Stopping the flow ends the call on the worker, and empties the url. # Stopping the flow ends the call on the worker, and empties the url.
await node.stop() await node.stop()
@@ -110,7 +110,7 @@ def test_a_service_that_never_gets_healthy_reads_down(monkeypatch):
async def scenario() -> None: async def scenario() -> None:
agent = Agent() agent = Agent()
agent.worker = RemoteWorker( agent.worker = RemoteWorker(
name="amcomp", name="gpu-worker",
labels=["gpu"], labels=["gpu"],
send=agent.send_json, send=agent.send_json,
loop=asyncio.get_running_loop(), loop=asyncio.get_running_loop(),
+1 -5
View File
@@ -116,11 +116,7 @@ def test_one_flows_failure_does_not_stand_another_down():
def test_two_tasks_on_one_flow_do_not_spend_each_others_budget(): def test_two_tasks_on_one_flow_do_not_spend_each_others_budget():
"""The house's power flow: an MQTT subscriber and its keepalive publisher. """A subscriber and publisher each get a full restart budget."""
Both die with the same broker. Sharing one budget meant the flow was stood
down after half the restarts either task was allowed on its own.
"""
tries = {"sub": 0, "pub": 0} tries = {"sub": 0, "pub": 0}
def crasher(which: str): def crasher(which: str):
+118 -153
View File
@@ -1,192 +1,157 @@
# Development # Development
How to run, test and lint the `app` stack. Run all commands from this repository's root unless a step says otherwise.
## Running it ## Run the app
**Integrated — the normal way.** From the workspace root, once: `make init` (submodules, ### Docker development stack
secrets, the per-stack `.env` files, the shared external `proxy` docker network). Then
`make dev` brings both stacks up detached behind a single Traefik — which is what lets the
app and the marketing site answer on one port. The app is one container
(`backend/Dockerfile`): the engine, its API and the built dashboard on port 8000. `make status` lists the containers,
`make down` stops everything.
The hostname comes from `DOMAIN` in `app/.env`, which `scripts/setup.sh` copies out of the Install Docker with Compose v2, Make and OpenSSL, then run:
root `.env` — change it there and re-run `make init`. `app/Makefile` takes that value as a
default only (`export DOMAIN ?= …`), so `make dev DOMAIN=…` overrides it and the override
reaches the recipes.
**App only.** In a bare clone of this repo, `make init` once (`.env` from `.env.example` ```sh
with generated secrets, the `proxy` docker network, prints the admin login), then make init # create .env, generate secrets and create the proxy network
`make dev-local` serves the stack detached at `http://app.localhost` behind Traefik on make dev-local # start the app and docs behind the local proxy
port 80. `make dev` starts this stack on its own Traefik with all host ports make down # stop the stack
published, in the foreground. `make dev-utils` starts just proxy and mailcatcher — the ```
useful half when the backend runs on the host.
**Single container.** `docker compose up -d` in this repo's root uses `compose.yml`, the `make init` prints the admin login. The app image contains the engine, API and
setup the getting-started guide gives: the image on `localhost:8080` with its own Redis, no built dashboard on port 8000. The proxy exposes it at
Traefik, no `.env`. It is a separate project from the stack below and shares nothing [app.localhost](http://app.localhost), with the API under `/api/v1`.
with it. [api.localhost](http://api.localhost) reaches the same container, and
[docs.localhost](http://docs.localhost) serves the documentation.
**No Docker.** `make install` once, then `make dev-backend` (FastAPI on :8000) and Development targets default to `DOMAIN=localhost`, independently of `.env`.
`make dev-frontend` (Vite on :5173) in two terminals. The backend needs nothing running: its Use `make dev-local DOMAIN=example.test` to choose another development domain.
database is SQLite in `DATA_DIR` (`backend/flow-data` by default), created on first start. `make dev-lan APP_PORT=8080` also publishes the app on the machine's LAN address
A local stack's CORS list already contains `http://localhost:5173`, so a host Vite server for devices that cannot resolve the proxy hostname.
can talk to a containerised API.
Nothing invokes `docker compose` bare, and neither should you: both stacks keep their `make dev` runs in the foreground with additional published ports:
compose files in a `docker/` directory, so without `-p fluksio-app` they collide in a
project named after that directory, and `--env-file` is required because compose
interpolates `.env` before it reads `compose.yml`. The `COMPOSE` variable in `Makefile` has
both. The files layer as `compose.yml` (production) → `compose.dev.yml` (local Traefik,
published ports, hot reload, test helpers) → `compose.local.yml` (integrated). CI layers
`compose.ci.yml` instead of the last one; `compose.traefik.yml` is the production edge
proxy, deployed on its own.
## URLs
Integrated stack (root `make dev`). Traefik's port 80 is the **only** published port:
`compose.local.yml` drops the rest, since behind the proxy they add nothing and one of them
already being taken would stop the whole stack from starting.
| URL | Service | | URL | Service |
|---|---| |---|---|
| http://app.localhost | dashboard, with the API under `/api/v1` | | http://app.localhost, http://api.localhost | app through Traefik |
| http://api.localhost | the same container, for callers configured with the API host | | http://localhost:8000 | dashboard and API directly |
| http://app.localhost/docs | OpenAPI / Swagger UI |
| http://localhost | marketing site (the `index` stack) |
Mailcatcher has no route here, so use the container directly:
`docker logs -f fluksio-app-mailcatcher-1` for captured mail. The database is a file on the
data volume: `docker exec -it fluksio-api sqlite3 /data/fluksio.db`.
Standalone (`cd app && make dev`) publishes the full set:
| URL | Service |
|---|---|
| http://app.localhost, http://api.localhost | this stack's own Traefik on :80 |
| http://localhost:8000 | dashboard and API |
| http://localhost:8090 | Traefik dashboard | | http://localhost:8090 | Traefik dashboard |
| http://localhost:1080 | mailcatcher web UI (SMTP on 1025) | | http://localhost:1080 | captured mail; SMTP on port 1025 |
## Hot reload, and what it misses For `make dev-local`, read captured mail with
`docker logs -f fluksio-app-mailcatcher-1`. The proxy is the only published
port in that setup.
`compose.dev.yml` bind-mounts `backend/fluksio` and `worker/fluksio_worker` into the api ### Single-container setup
container and runs uvicorn with `--reload` over both, so a backend edit is live and a worker
edit restarts the engine along with its local workers. The `develop.watch` block next to it
syncs the same directories, but only under `docker compose watch`, which no target runs — the detached
`up -d` flow never triggers it. The mount is what does the work, so an api container created
before the mount existed keeps serving the source baked into its image and has to be
recreated once.
The dashboard is built into the app image, so a UI change needs a rebuild: `docker compose up -d` uses the top-level `compose.yml`. It serves the app on
`make rebuild-frontend` from `app/`. Only the image's frontend stage runs again, and the [localhost:8080](http://localhost:8080), includes Redis, and needs no `.env` or
container is recreated with the domain and overlays the running one was started with. proxy. See the [facility guide](docs/getting-started/facility-automation.md).
This setup has separate data from the development stack.
## Services the dev stack adds For the operator stack under `docker/`, use the Makefile targets. They pass
the project name and env-file path explicitly. Its files layer as
`compose.yml` → `compose.dev.yml` → `compose.local.yml`; CI uses
`compose.ci.yml` as its final overlay. `compose.traefik.yml` supplies a
separate production edge proxy.
`compose.dev.yml` brings up a broker and a time-series database of the stack's own, so the ### Without Docker
mqtt and influx node types are testable without external hardware. Neither publishes a host
port, on purpose — nothing outside the stack needs them.
- **mosquitto** — anonymous MQTT. Point a node at `broker_host: mosquitto`, port 1883. Install uv and Bun, then run `make install`. Use two terminals:
- **influxdb 2.7** — `http://influxdb:8086`, org and bucket `fluksio`, token
`fluksio-dev-token`. No volume, so `down -v` starts it over.
- **mailcatcher** — SMTP on 1025, web UI on 1080; the backend is pointed at it, so no
development mail leaves the machine.
`redis` comes from `compose.yml` and exists in production too. Without `REDIS_HOST` the flow ```sh
engine keeps state in memory instead of Redis. `db` and `adminer` are still there but only make dev-backend # FastAPI on :8000 with hot reload
start under the `analytics` profile — they are Umami's Postgres now, not the engine's. make dev-frontend # Vite on :5173
```
The backend creates SQLite in `DATA_DIR` (`backend/flow-data` by default).
No database service is required. `make dev-utils` starts just the proxy and
mailcatcher if you need them. The Docker development API also accepts the
Vite origin, so a host Vite process can use it.
## Reloading changes
`docker/compose.dev.yml` mounts `backend/fluksio` and `worker/fluksio_worker`
into the API container. Uvicorn watches both, so changes restart the engine
and local workers. The adjacent `develop.watch` settings apply only to
`docker compose watch`; the Makefile's detached targets use the mounts.
The dashboard is built into the image. After a UI change, run:
```sh
make rebuild-frontend
```
This preserves the running stack's domain, overlays and LAN port.
## Development services
The development Compose file includes services for trying integrations:
- **Mosquitto:** anonymous MQTT at `mosquitto:1883` inside the stack.
- **InfluxDB:** `http://influxdb:8086`, organization and bucket `fluksio`,
development token `fluksio-dev-token`.
- **Mailcatcher:** captures outbound mail locally.
MQTT and InfluxDB have no published host ports. Redis persists flow state and
queued work. Without `REDIS_HOST`, a source-run engine uses memory instead.
Postgres and Adminer start only with the optional `analytics` profile.
## Configuration and secrets ## Configuration and secrets
The root `.env` is the source of truth. `scripts/setup.sh` creates `app/.env` from `make init` copies `.env.example` to `.env` and generates initial secrets.
`app/.env.example` and keeps the shared keys — domain, secret key, superuser — in step with Edit `.env` directly and keep it out of git. After updating the repository,
it. Only the `.env.example` files are tracked; `make secrets` compare it with `.env.example` for new settings. Generate additional secrets
reprints the generated values. A release that adds a key needs `scripts/setup.sh --secrets` with `openssl rand -hex 32`.
before `make update`, so the key exists before the stack is rebuilt.
The backend settings load `../.env` relative to `backend/` through pydantic-settings — Backend settings load this repository's `.env` when run from `backend/`.
`FLUKSIO_ENV_FILE` points that elsewhere, which is what an installed `fluksio` does. An `FLUKSIO_ENV_FILE` selects another file, and exported variables override it.
exported environment variable always outranks the file, which is how the suite pins its own The installed CLI uses the selected instance's data directory instead.
configuration.
`DATA_DIR` is where this instance keeps everything: the SQLite database, the flow git `DATA_DIR` holds the database, flow repository, credentials, artifacts and
repository, `secrets.enc`, `oauth-key.pem`, `cloud.json`, artifacts and the user venv. Each managed node environment. Individual paths can be overridden. Docker stores
path derives from it and can still be set on its own — the compose stack spells all of them them under `/data`; see [Configuration](docs/reference/configuration.md).
out against `/data`.
## Tests ## Tests
`make test` is `test-backend` plus `test-frontend`. `make test` runs backend tests, frontend helper checks and Playwright tests.
**Backend.** `cd backend && uv run bash scripts/test.sh` runs pytest under coverage; the - `make test-backend` runs pytest with coverage. Its temporary SQLite database
HTML report lands in `backend/htmlcov`. Nothing has to be running — mail is patched out in and test settings need no running services and leave development data alone.
`tests/`, and the flow engine falls back to `MemoryState` while `REDIS_HOST` is empty. The HTML coverage report is in `backend/htmlcov/`.
`tests/__init__.py` pins the suite's own environment before anything imports the settings: a - `make test-checks` runs the frontend's `*.check.ts` helper scripts.
SQLite file in a temp directory (deleted per session, so a run never touches development - `make test-frontend` starts an isolated `fluksio-test` stack behind the local
data), its own superuser, and `ENVIRONMENT=local`, so a checkout Traefik, then runs Playwright in a pinned container. Start `make dev-local`
configured for a deployment cannot drag that deployment's configuration into the run. or `make dev-lan` first. The test stack stays up for inspection at
`app.test.localhost`; `make down` stops it.
- `make test-webkit` runs the drag tests in WebKit against the same test stack.
**Frontend (e2e).** `make test-frontend` runs the suite against a stack of its own, because Narrow a browser run with
the suite creates and deletes flows, dashboards and users. `make test-stack`, which it runs
first, starts or updates the compose project `fluksio-test` (`docker/compose.test.yml` over the
local stack's files), with its own containers, volumes and image tags. The local Traefik serves
it on `app.test.localhost` and `api.test.localhost`, so `make dev` has to be up, in either
mode: the test stack never layers `compose.lan.yml` and publishes no port. It stays running
after a run for a look at a failure, `make down` stops it, and its volumes are expendable
(`docker compose -p fluksio-test down -v` starts it over).
The suite runs inside the pinned `mcr.microsoft.com/playwright:v<version>-noble` image
(version read from `frontend/package.json`) on the `proxy` network. It maps both names onto
the Traefik container's address twice, with `--add-host` and with `HOST_RESOLVER_RULES`,
because Chromium pins `*.localhost` to loopback whatever `/etc/hosts` says. Narrow a run with
`make test-frontend PLAYWRIGHT_ARGS="--grep flows"`. `make test-frontend PLAYWRIGHT_ARGS="--grep flows"`.
The browser version comes from `frontend/package.json`. Container resolver
rules route `*.localhost` to Traefik, including in Chromium and WebKit.
`make test-webkit` runs the drag specs in WebKit against the same stack. WebKit takes Specs live in `frontend/tests/`. Authentication is saved once, then reused by
`*.localhost` to loopback through GLib and ignores `/etc/hosts`, so that container shares the desktop and mobile projects. The suite takes its origin from
Traefik's network namespace, where loopback is Traefik. Playwright's Linux WebKit reads a `PLAYWRIGHT_BASE_URL`, and optionally `PLAYWRIGHT_API_URL`, independently of
`writing-mode: vertical-lr` range across, so a vertical slider's geometry is not asserted there. `.env`. Its guard refuses public addresses because tests create and delete
flows, dashboards and users.
The specs are `frontend/tests/*.spec.ts`; `playwright.config.ts` defines `setup` → For manual visual checks, `frontend/scripts/capture-screenshots.mjs` captures
`chromium` and `mobile`, plus `webkit` when `PLAYWRIGHT_WEBKIT=1` is set. `auth.setup.ts` logs both themes at desktop and phone sizes. Set `APP_URL`, `FIRST_SUPERUSER` and
in once into `playwright/.auth/user.json`, which the browser projects reuse; `mobile` runs `FIRST_SUPERUSER_PASSWORD` in the environment, then run it from `frontend/`
`mobile.spec.ts` only, on a Pixel 5 at 393px. with Bun in a Playwright-capable environment. `SCREENSHOT_DIR` selects the
output directory. `WEBSITE_URL` optionally adds screenshots of another site.
The origin comes from `PLAYWRIGHT_BASE_URL`, with the API on it unless `PLAYWRIGHT_API_URL` ## Lint and documentation
names another, deliberately never from `.env`. In a deployment checkout that file names the
deployment, which would mean driving a browser at the local stack while sending teardown
`DELETE`s to the live instance. `tests/guard.ts` runs first and refuses the whole
run when either origin resolves outside loopback or the private ranges, because the suite
creates and deletes flows, dashboards and users. `PLAYWRIGHT_ALLOW_PUBLIC=1` overrides it.
**Visual check.** From the root, `make verify` logs in and screenshots both themes into `make lint` checks Python with Ruff and mypy, and the frontend with Biome.
`app/frontend/screenshots/{light,dark}/`. It is an alias for `make verify-docker`: the run `make format-frontend` applies frontend formatting. Install the repository's
always happens in the Playwright container, which pins the browser by reading commit hooks with `uv run --project backend prek install` after `make install`.
`frontend/package.json`. Running Playwright on the host is unsupported — the workspace pins a The backend hook also regenerates the frontend API client; run
browser revision whose install is incomplete. The domain comes off the running stack (the `make generate-client` to request that explicitly.
app container's Traefik rule), so the shots are of whatever `make dev` last started.
## Lint, hooks and the generated client Read [the design reference](DESIGN-GUIDELINES.md) before UI work.
`make docs` builds the documentation, and `make docs-serve` opens a preview.
`make lint` runs ruff, mypy and biome, and only reports — it is what the pre-commit hook and
CI run, so neither rewrites the tree. `make format-frontend` is the writing half, applying
biome's fixes. `make hooks` at the root installs the pre-commit hooks in both stacks;
`.pre-commit-config.yaml` holds them. One is worth knowing about:
`generate-frontend-sdk` regenerates the SPA's API client from the backend's OpenAPI schema
whenever anything under `backend/` changes, so a route change reaches the frontend without
being asked. `make generate-client` forces it.
Run the root's `make design-check` before pushing anything that touches the design tokens,
`lib/motion.ts`, `components.json` or the font asset — the design system is a duplication
contract between the two frontends, and that target is the only thing checking it.
## CI ## CI
`.gitea/workflows/` runs four jobs: Workflows in `.gitea/workflows/` check backend tests and wheel builds,
`test-backend.yml` (pytest on Python 3.12 and 3.14, plus a check that both wheels build), `playwright.yml` (the e2e suite in Playwright, lint, documentation, and a Compose image smoke test. They use
two shards, one compose project each, the browser loading the SPA from the `backend` this repository's files and `.env.example`; no parent workspace is required.
service), `pre-commit.yml` (the hooks above, plus strict mypy, which they do not cover) and
`test-compose.yml` (a smoke test that the production image actually comes up). All but
`test-backend.yml` write `.env` from `.env.example` first. The backend suite needs none.
+4 -5
View File
@@ -1,7 +1,6 @@
# Integrated local stack — layered on top of compose.yml + compose.dev.yml by # Local proxy stack, layered over compose.yml + compose.dev.yml by
# the root `make dev`. Reattaches this stack's Traefik to the shared external # `make dev-local`. Traefik joins the external `proxy` network and serves
# `proxy` network so it also routes the website stack's containers, and serves # plain HTTP on *.${DOMAIN} (default *.localhost).
# everything over plain http on *.${DOMAIN} (default *.localhost).
# Everything here is reachable through Traefik on *.${DOMAIN}, so the host ports # Everything here is reachable through Traefik on *.${DOMAIN}, so the host ports
# compose.dev.yml publishes are dropped: they add nothing and make the whole # compose.dev.yml publishes are dropped: they add nothing and make the whole
@@ -34,6 +33,6 @@ services:
- BACKEND_CORS_ORIGINS=http://${DOMAIN:-localhost},http://app.${DOMAIN:-localhost},http://localhost:5173 - BACKEND_CORS_ORIGINS=http://${DOMAIN:-localhost},http://app.${DOMAIN:-localhost},http://localhost:5173
networks: networks:
# Shared with the website stack; created by the root scripts/setup.sh. # External reverse-proxy network. Created by `make init`.
proxy: proxy:
external: true external: true
+5 -6
View File
@@ -146,9 +146,8 @@ services:
# rebuild would silently stop every phone being notified: the key they # rebuild would silently stop every phone being notified: the key they
# subscribed against would be gone. # subscribed against would be gone.
- WEBPUSH_FILE=/data/webpush.json - WEBPUSH_FILE=/data/webpush.json
# Schedules are written in local time: "off at 02:00" means the house's # Cron schedules use this timezone. Set it to the timezone your flows
# two in the morning, not the container's. Unset, an image is UTC, and a # should follow; the image defaults to UTC.
# cron would be right twice a year.
- TZ=${TZ:-UTC} - TZ=${TZ:-UTC}
volumes: volumes:
@@ -248,7 +247,8 @@ services:
# process per port. Credentials are minted by the backend from the same # process per port. Credentials are minted by the backend from the same
# TURN_SECRET (coturn's REST scheme), so nothing is stored here. The secret # TURN_SECRET (coturn's REST scheme), so nothing is stored here. The secret
# goes in through a config file rather than the command line, which `ps` # goes in through a config file rather than the command line, which `ps`
# shows. See DEPLOY.md for ports, DNS and certificates. # shows. Configure ports with TURN_MIN_PORT and TURN_MAX_PORT and mount
# certificates under TURN_CERT_DIR below.
coturn: coturn:
logging: *logging logging: *logging
image: coturn/coturn:4.6.3 image: coturn/coturn:4.6.3
@@ -348,7 +348,6 @@ volumes:
app-flow-data: app-flow-data:
networks: networks:
# Shared with the website stack and whatever reverse proxy fronts them. # External reverse-proxy network. Created by `make init`.
# Created once by the workspace root's scripts/setup.sh.
proxy: proxy:
external: true external: true
+5 -5
View File
@@ -26,11 +26,11 @@ invalidates its access. MCP uses a separate signing key at
`OAUTH_PRIVATE_KEY_FILE`; rotating that key invalidates all MCP tokens. `OAUTH_PRIVATE_KEY_FILE`; rotating that key invalidates all MCP tokens.
An instance connected to a [portal](../interface/portal.md) also answers An instance connected to a [portal](../interface/portal.md) also answers
through it, at `https://hub.fluksio.com/i/<instance-id>/mcp` (the portal's through the portal's API address, for example
own `hub.` host for a custom one). The portal forwards the endpoint and its `https://portal.example.com/i/<instance-id>/mcp`. The portal forwards MCP and
OAuth, and the instance checks the agent's token as it does locally. The OAuth requests; the instance checks the agent's token as it does locally.
consent page opens in the instance as the portal serves it, so open the Open the instance from the portal first so the browser can show its consent
instance from the portal first if this browser has not recently. page.
## Available tools ## Available tools
+2 -2
View File
@@ -57,7 +57,7 @@ that instance's running engine; plain mode refuses to start a duplicate.
| `--admin-email ADDR` | `admin@example.com` | First admin account | | `--admin-email ADDR` | `admin@example.com` | First admin account |
| `--admin-password PW` | Generated | First admin password | | `--admin-password PW` | Generated | First admin password |
| `--enroll CODE` | Unset | Enroll before starting, if not already paired | | `--enroll CODE` | Unset | Enroll before starting, if not already paired |
| `--portal URL` | `https://hub.fluksio.com` | Portal used by `--enroll` | | `--portal URL` | Built-in portal address | Portal used by `--enroll` |
| `--max-runs N` | `4` | Concurrent batch runs | | `--max-runs N` | `4` | Concurrent batch runs |
| `--max-cascades N` | `4` | Concurrent live cascades | | `--max-cascades N` | `4` | Concurrent live cascades |
| `--max-workers N` | `4` | Local Python worker processes | | `--max-workers N` | `4` | Local Python worker processes |
@@ -106,7 +106,7 @@ from the instance's project directory, or select it explicitly.
| Option | Purpose | | Option | Purpose |
|---|---| |---|---|
| `--portal URL` | Use another portal instead of `https://hub.fluksio.com` | | `--portal URL` | Use another compatible portal |
| `--as EMAIL` | Choose the local superuser represented by portal sessions | | `--as EMAIL` | Choose the local superuser represented by portal sessions |
| `--data-dir PATH` | Select instance storage | | `--data-dir PATH` | Select instance storage |
| `--global` | Select `~/.fluksio` | | `--global` | Select `~/.fluksio` |
+7 -17
View File
@@ -132,9 +132,9 @@ storage; emitted frames alone are not recorded as run artifacts.
### Making clips playable on an LVGL panel ### Making clips playable on an LVGL panel
A wall panel that renders dashboards natively, such as an ESP32-P4 running A wall panel that renders dashboards natively, such as an ESP32-P4 running
openpanel, decodes only H.264 Constrained Baseline (in software, about 29 fps openpanel, decodes H.264 Constrained Baseline and MJPEG. Clips straight from
at 640×360) and MJPEG. It refuses clips straight from a phone or camera, which a phone or camera may use an unsupported profile or resolution, such as
are usually Main or High profile at 1080p or more. Re-encode those with a node Main or High profile at 1080p or more. Re-encode those with a node
that takes a `video` input port `clip` and has a `video` output port that takes a `video` input port `clip` and has a `video` output port
`panel_clip`: `panel_clip`:
@@ -288,19 +288,9 @@ downloads them again. Point `MODELS` at a directory under `/data` to keep them
on the volume. An [attached worker](workers.md) starts a process per call and on the volume. An [attached worker](workers.md) starts a process per call and
loads the model each time. loads the model each time.
What it costs, measured on a 2016 server CPU (Xeon E5-2683 v4): Models download on first use. Speed, memory use and recognition quality depend
on your hardware, model and audio. Try a short clip from your own microphone
| | Size | Memory | 3 s of speech, one core | All cores | before choosing a model; `tiny` is a useful first stop on a small CPU.
|---|---|---|---|---|
| Packages | 150 MB download, 460 MB installed | | | |
| Whisper `tiny` | 78 MB | 300 MB | 1.6 s | 0.7 s |
| Whisper `base` | 148 MB | 375 MB | 2.9 s | 1.2 s |
| piper `en_US-lessac-medium` | 63 MB | 250 MB | 0.6 s | 0.3 s |
Each model downloads once, on its node's first call. Loading it takes about a
second for Whisper and four seconds for piper. Memory is the worker's peak while
it holds the model. These figures come from piper's own speech fed back into
Whisper. A room microphone is harder, on `tiny` most of all.
## Printing ## Printing
@@ -391,7 +381,7 @@ access token; see [The HTTP API](api.md).
managed environment can also install your project from a Modules manifest: managed environment can also install your project from a Modules manifest:
```text ```text
-e /home/you/my-research -e /path/to/my-research
numpy>=2 numpy>=2
``` ```
+7 -6
View File
@@ -38,11 +38,12 @@ export FLUKSIO_WORKER_TOKEN='<token from the engine>'
fluksio-worker \ fluksio-worker \
--url wss://fluksio.example.com/api/v1/workers/attach \ --url wss://fluksio.example.com/api/v1/workers/attach \
--labels gpu,cuda12 \ --labels gpu,cuda12 \
--python /opt/torch-venv/bin/python --python /path/to/node-venv/bin/python
``` ```
Use your engine's hostname. `--python` selects the environment containing the Replace the address and interpreter path with your own. `--python` selects
node's dependencies, such as PyTorch or a device SDK. If you install the worker the environment containing the node's dependencies, such as PyTorch or a
device SDK. If you install the worker
in that environment, you can omit the option. An engine installation also in that environment, you can omit the option. An engine installation also
provides the same agent as `fluksio worker`. provides the same agent as `fluksio worker`.
@@ -55,7 +56,7 @@ provides the same agent as `fluksio worker`.
| `--python` | worker's interpreter | Python used for node code | | `--python` | worker's interpreter | Python used for node code |
| `--parallel` | `1` | concurrent calls | | `--parallel` | `1` | concurrent calls |
| `--artifact-url` | derived from `--url` | artifact endpoint, if hosted separately | | `--artifact-url` | derived from `--url` | artifact endpoint, if hosted separately |
| `--service-host` | address the engine sees | host in a service's url, when the engine reaches this machine elsewhere, such as through a tunnel | | `--service-host` | address the engine sees | reachable hostname or IP for services, when the connection address is unsuitable |
| `--cpus` | detected | advertised CPU count | | `--cpus` | detected | advertised CPU count |
| `--gpus` | detected | advertised GPU count; explicit `0` means none | | `--gpus` | detected | advertised GPU count; explicit `0` means none |
| `--ram-mb` | detected | advertised memory in MB | | `--ram-mb` | detected | advertised memory in MB |
@@ -183,7 +184,7 @@ in `$DATA_DIR/provisioners.json`, replacing the example addresses and paths:
[{ [{
"type": "slurm", "type": "slurm",
"name": "hpc", "name": "hpc",
"login": "me@login.cluster", "login": "user@login.cluster.example.com",
"ssh_key": "/secrets/hpc_ed25519", "ssh_key": "/secrets/hpc_ed25519",
"engine_url": "wss://fluksio.example.com/api/v1/workers/attach", "engine_url": "wss://fluksio.example.com/api/v1/workers/attach",
"max_idle_s": 300, "max_idle_s": 300,
@@ -195,7 +196,7 @@ in `$DATA_DIR/provisioners.json`, replacing the example addresses and paths:
"ram_mb": 65536, "ram_mb": 65536,
"labels": ["gpu"], "labels": ["gpu"],
"sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"], "sbatch": ["--partition=gpu", "--gres=gpu:1", "--time=04:00:00"],
"prerun": ["module load cuda/12", "source ~/venvs/fluksio/bin/activate"] "prerun": ["module load cuda", "source /path/to/node-venv/bin/activate"]
}] }]
}] }]
``` ```
+12 -10
View File
@@ -7,10 +7,11 @@ No hardware required for the first lap.
You need Git and Docker with Compose v2 on a 64-bit machine, x86 or ARM. A You need Git and Docker with Compose v2 on a 64-bit machine, x86 or ARM. A
Raspberry Pi 4 or 5 with a 64-bit OS works too. Clone the app repository and Raspberry Pi 4 or 5 with a 64-bit OS works too. Clone the app repository and
start its single container: start its single container. Replace `<repository-url>` with the app
repository's URL:
```sh ```sh
git clone https://git.stroblme.de/Fluksio/app.git fluksio git clone <repository-url> fluksio
cd fluksio cd fluksio
docker compose up -d docker compose up -d
docker compose logs fluksio docker compose logs fluksio
@@ -115,24 +116,25 @@ See [Dashboards and devices](../interface/dashboards.md) for widgets and pairing
a wall tablet. a wall tablet.
For longer-term history, add an **InfluxDB** node that consumes For longer-term history, add an **InfluxDB** node that consumes
`living_temperature`. It needs the `influx` extra. Configure your server URL, organization, bucket, token `living_temperature`. It needs the `influx` extra. Configure your database URL,
and `writes` mapping. Store the token under **Secrets** and select it in the organization, bucket, token and `writes` mapping. Store the token under
node's credential field. [Node types](../reference/node-types.md#influxdb) **Secrets** and select it in the node's credential field.
[Node types](../reference/node-types.md#influxdb)
has the write and query formats. has the write and query formats.
Fluksio does not run an InfluxDB server either. Use your own. Run InfluxDB separately, just like the MQTT broker.
## Run it long term ## Run it long term
The container from `compose.yml` is fit for long-term use. It restarts with The container from `compose.yml` is fit for long-term use. It restarts with
Docker, and `/data` holds everything it keeps: the database, the flows, Docker, and `/data` holds everything it keeps: the database, the flows,
secrets, artifacts and its Redis. Back up that volume. Compose names it secrets, artifacts and its Redis. Back up that volume. Compose prefixes its
`fluksio_data`. name with your project name.
The container speaks plain HTTP. For access from outside your network, put a The container speaks plain HTTP. For access from outside your network, put a
reverse proxy with HTTPS and WebSocket support in front of it, such as Caddy, reverse proxy with HTTPS and WebSocket support in front of it, such as Caddy,
Traefik or nginx, or [enroll with the portal](../interface/portal.md). A Traefik or nginx, or [enroll with the portal](../interface/portal.md).
Caddyfile for `fluksio.example.com` is one line: For example, a Caddyfile can proxy your app address:
```text ```text
fluksio.example.com { fluksio.example.com {
+2 -3
View File
@@ -1,8 +1,7 @@
# Pick your starting point # Pick your starting point
Choose the setup that fits your work. Pick the setup that fits your work. Both paths lead to the same engine, with
Both paths lead to the same Fluksio engine, with a first flow to try along the way. a first flow to try along the way.
This is really just about a decision what you're planning to do with Fluksio.
<div class="fluksio-lanes" markdown> <div class="fluksio-lanes" markdown>
+10 -10
View File
@@ -501,13 +501,13 @@ start and renames the file `panels.json.migrated`.
### A screen without a browser ### A screen without a browser
The openpanel firmware, kept in its own repository, draws dashboards as native Compatible device firmware can draw dashboards as native widgets without a
widgets on an ESP32-P4 wall panel (Waveshare ESP32-P4-WIFI6-Touch-LCD-7B, browser. For example, openpanel supports ESP32-P4 wall panels. Once its
1024×600). Once its config page points it at this instance (the device's link config page points it at this instance, it shows a pairing code; enter it as
works too, the device keeps only its address), it shows the pairing code itself; enter it as in step 5. It reads the same published in step 5. The device's link works too, since the device keeps only the
dashboards and live values and draws every widget type with two exceptions, instance address. It reads the same published dashboards and live values.
both shown as a placeholder: a website tile that embeds a page (image tiles Embedded website tiles and run charts show placeholders; image tiles work,
work), and a chart of runs, which a device credential may not read. Media tiles and device credentials cannot read runs. Media tiles
play MJPEG streams, pushed frames, images, and H.264 Baseline or MJPEG clips up play MJPEG streams, pushed frames, images, and H.264 Baseline or MJPEG clips up
to 640×360. Re-encode other clips with the [clip recipe](../code/nodes.md#making-clips-playable-on-an-lvgl-panel). to 640×360. Re-encode other clips with the [clip recipe](../code/nodes.md#making-clips-playable-on-an-lvgl-panel).
@@ -625,9 +625,9 @@ A camera device (openpanel on a NANO) reports these ports:
dashboard tile, the preview on the device's page, a screen through the dashboard tile, the preview on the device's page, a screen through the
portal), or a node of a running flow reads its message. Once its flow wires portal), or a node of a running flow reads its message. Once its flow wires
`camera`, the device's page shows a **Camera** preview. A flow that reads `camera`, the device's page shows a **Camera** preview. A flow that reads
`camera` keeps it sending all the time, about 300 GB a month through a cloud `camera` keeps it sending continuously. Traffic depends on resolution, frame
instance, so the Device node warns when a node in its flow reads a port that rate and image quality, so the Device node warns when a flow node reads an
sends on demand. Have flows read `snapshot` or `person` instead. To keep the on-demand port. For occasional updates, read `snapshot` or `person`. To keep the
picture of whoever came by, a node reading `snapshot` can hand it on to a picture of whoever came by, a node reading `snapshot` can hand it on to a
message of its own, which a media tile shows until the next one: message of its own, which a media tile shows until the next one:
+4 -3
View File
@@ -1,9 +1,10 @@
# The dashboard app # The dashboard app
Open your instance's app address, or select it at [fluksio.com](https://fluksio.com) Open your instance's app address, or select it at [fluksio.com](https://fluksio.com)
after enrollment. For the local Docker setup, use after enrollment. For the single-container setup, open
[app.localhost](http://app.localhost) and the admin login printed by [localhost:8080](http://localhost:8080) and use the admin login printed by
`make init`. See [Getting started](../getting-started/index.md) for setup. `docker compose logs fluksio`. See [Getting started](../getting-started/index.md)
for setup.
## Finding your way around ## Finding your way around
+5 -4
View File
@@ -18,7 +18,7 @@ API requests, responses and live updates through the connection.
fluksio enroll <code> fluksio enroll <code>
``` ```
The CLI uses `https://hub.fluksio.com` by default. If you started the engine The CLI uses the built-in portal address by default. If you started the engine
with `--data-dir` or `--global`, use the same option when enrolling. A running with `--data-dir` or `--global`, use the same option when enrolling. A running
engine picks up the connection automatically; otherwise start it with engine picks up the connection automatically; otherwise start it with
`fluksio serve`. `fluksio serve`.
@@ -65,12 +65,13 @@ removing them at the portal, or disconnecting, deletes them.
## Hooks and agents through the portal ## Hooks and agents through the portal
The portal also forwards a flow's hooks and the MCP endpoint, so they work The portal also forwards a flow's hooks and the MCP endpoint, so they work
without an inbound port: without an inbound port. Replace `portal.example.com` with your portal's API
address in these examples:
- A webhook or a served WebSocket: - A webhook or a served WebSocket:
`https://hub.fluksio.com/i/<instance-id>/hooks/<flow>/<url>/<secret>`, `https://portal.example.com/i/<instance-id>/hooks/<flow>/<url>/<secret>`,
with `wss://` for a socket. with `wss://` for a socket.
- An agent: `https://hub.fluksio.com/i/<instance-id>/mcp`, see - An agent: `https://portal.example.com/i/<instance-id>/mcp`, see
[Agents over MCP](../code/agents.md). [Agents over MCP](../code/agents.md).
Through the portal a hook needs a [secret](../reference/node-types.md#http). Through the portal a hook needs a [secret](../reference/node-types.md#http).
+2 -2
View File
@@ -105,8 +105,8 @@ comma separated. Empty, the default, is the minimal engine.
| `server` | all of the above | | `server` | all of the above |
A node whose extra is missing says so on the canvas. The same names work as A node whose extra is missing says so on the canvas. The same names work as
pip extras, such as `pip install 'fluksio[mqtt]'`. Connector packages built pip extras, such as `pip install 'fluksio[mqtt]'`. To include connectors, place
with `make connectors` are installed into the image too. their wheels in `backend/connector-wheels/` before building the image.
## Environment ## Environment
+1 -1
View File
@@ -377,7 +377,7 @@ name of its own, in the node panel's port name field or as
| `model` | empty | the trained model, an artifact digest; empty runs wren's demo model | | `model` | empty | the trained model, an artifact digest; empty runs wren's demo model |
| `min_confidence` | empty | the confidence a call needs; empty takes the model's own threshold | | `min_confidence` | empty | the confidence a call needs; empty takes the model's own threshold |
| `targets` | empty | by target port, what the model is told of it and how far it is trusted (below) | | `targets` | empty | by target port, what the model is told of it and how far it is trusted (below) |
| `aliases` | empty | your own words, or words the recogniser hears wrong, in the model's: `{"traverse": "living room light", "page too": "patio"}` | | `aliases` | empty | map your own names to the model's target names, such as `{"desk lamp": "office light"}` |
| `examples` | empty | for training only, commands to learn beyond the ones wren generates (below) | | `examples` | empty | for training only, commands to learn beyond the ones wren generates (below) |
| `tests` | empty | for training only, commands a new model is scored on (below) | | `tests` | empty | for training only, commands a new model is scored on (below) |
+1 -1
View File
@@ -3,7 +3,7 @@
* Follows the zensical custom color scheme approach * Follows the zensical custom color scheme approach
* (https://zensical.org/docs/setup/colors/#custom-color-schemes): override the * (https://zensical.org/docs/setup/colors/#custom-color-schemes): override the
* Material `--md-*` CSS variables per [data-md-color-scheme]. Colors mirror the * Material `--md-*` CSS variables per [data-md-color-scheme]. Colors mirror the
* design tokens in ../DESIGN.md → ../../DESIGN-GUIDELINES.md: neutral white / * design tokens in DESIGN-GUIDELINES.md at the repository root: neutral white /
* near-black surfaces carrying one slate-blue accent, with terracotta reserved * near-black surfaces carrying one slate-blue accent, with terracotta reserved
* for the single secondary affordance per view. The built-in `default` (light) * for the single secondary affordance per view. The built-in `default` (light)
* and `slate` (dark) schemes stay as the base so all of Material's derived * and `slate` (dark) schemes stay as the base so all of Material's derived
+8 -1
View File
@@ -1 +1,8 @@
# Fluksio Frontend # Fluksio frontend
React dashboard for the flow engine. From the repository root, run
`make install`, then `make dev-backend` and `make dev-frontend` in separate
terminals.
See [Development](../development.md) for Docker, tests and lint, and
[the design reference](../DESIGN-GUIDELINES.md) before changing the UI.
+1 -1
View File
@@ -1,5 +1,5 @@
/** /**
* The house panels, at the size of the screen they hang on. * Capture dashboards at their target screen size.
* *
* `/view/{name}` is the wall-panel route: no sidebar, no editor. Shooting it * `/view/{name}` is the wall-panel route: no sidebar, no editor. Shooting it
* at exactly the panel's pixels is the only way to see whether a tile fell off * at exactly the panel's pixels is the only way to see whether a tile fell off
+15 -15
View File
@@ -1,11 +1,10 @@
/** /**
* Visual verification for the integrated local stack (root `make verify`). * Capture the app at desktop and phone sizes in both themes.
* *
* Logs into the dashboard with the bootstrap superuser and captures the app * Logs into the dashboard with the bootstrap superuser and captures the app
* shell plus the website hero in both themes. This is the standard "did the * shell. Set WEBSITE_URL to also capture another site's landing page.
* change actually work" gate — look at the PNGs, do not just trust the build.
* *
* Env (all set by the root Makefile): * Env:
* APP_URL, WEBSITE_URL, FIRST_SUPERUSER, FIRST_SUPERUSER_PASSWORD * APP_URL, WEBSITE_URL, FIRST_SUPERUSER, FIRST_SUPERUSER_PASSWORD
* SCREENSHOT_DIR (default: ./screenshots) * SCREENSHOT_DIR (default: ./screenshots)
*/ */
@@ -13,14 +12,14 @@ import { mkdir } from "node:fs/promises"
import { chromium } from "@playwright/test" import { chromium } from "@playwright/test"
const APP_URL = process.env.APP_URL || "http://app.localhost" const APP_URL = process.env.APP_URL || "http://app.localhost"
const WEBSITE_URL = process.env.WEBSITE_URL || "http://localhost" const WEBSITE_URL = process.env.WEBSITE_URL
const EMAIL = process.env.FIRST_SUPERUSER const EMAIL = process.env.FIRST_SUPERUSER
const PASSWORD = process.env.FIRST_SUPERUSER_PASSWORD const PASSWORD = process.env.FIRST_SUPERUSER_PASSWORD
const OUT = process.env.SCREENSHOT_DIR || "screenshots" const OUT = process.env.SCREENSHOT_DIR || "screenshots"
if (!EMAIL || !PASSWORD) { if (!EMAIL || !PASSWORD) {
console.error( console.error(
"FIRST_SUPERUSER / FIRST_SUPERUSER_PASSWORD are unset — run from the root `make verify`.", "Set FIRST_SUPERUSER and FIRST_SUPERUSER_PASSWORD before running this script.",
) )
process.exit(1) process.exit(1)
} }
@@ -50,8 +49,8 @@ async function withTheme(browser, theme, viewport) {
} }
// Chromium pins *.localhost to loopback (RFC 6761), so /etc/hosts cannot point // Chromium pins *.localhost to loopback (RFC 6761), so /etc/hosts cannot point
// it at Traefik. The containerised run (root `make verify-docker`) passes the // it at Traefik. A containerised run can pass the mapping through
// mapping through here instead; empty on a host run. // HOST_RESOLVER_RULES instead.
const RESOLVER = process.env.HOST_RESOLVER_RULES const RESOLVER = process.env.HOST_RESOLVER_RULES
const browser = await chromium.launch( const browser = await chromium.launch(
RESOLVER ? { args: [`--host-resolver-rules=${RESOLVER}`] } : {}, RESOLVER ? { args: [`--host-resolver-rules=${RESOLVER}`] } : {},
@@ -64,11 +63,12 @@ for (const theme of ["light", "dark"]) {
const context = await withTheme(browser, theme, viewport) const context = await withTheme(browser, theme, viewport)
const page = await context.newPage() const page = await context.newPage()
await page.goto(`${WEBSITE_URL}/`, { waitUntil: "networkidle" }) if (WEBSITE_URL) {
// networkidle fires before the staggered entrance animations settle, which await page.goto(`${WEBSITE_URL}/`, { waitUntil: "networkidle" })
// would capture buttons mid-fade and make contrast look broken. // Wait for entrance animations before capturing the page.
await page.waitForTimeout(1500) await page.waitForTimeout(1500)
await page.screenshot({ path: `${dir}/website-hero.png` }) await page.screenshot({ path: `${dir}/website-hero.png` })
}
await page.goto(`${APP_URL}/login`, { waitUntil: "networkidle" }) await page.goto(`${APP_URL}/login`, { waitUntil: "networkidle" })
await page.screenshot({ path: `${dir}/app-login.png` }) await page.screenshot({ path: `${dir}/app-login.png` })
@@ -223,8 +223,8 @@ async function captureDevices(page, dir) {
* A media tile drawing what a camera published: the Demo's Media screen, * A media tile drawing what a camera published: the Demo's Media screen,
* whose house is drawn by a node as heat. * whose house is drawn by a node as heat.
* *
* Skipped unless the Demo is seeded (root `make seed-hosted-demo`), since it * Skipped unless the instance has the Demo's Media dashboard and frame source.
* is the one shot that needs a source of frames. The bytes arrive as * The bytes arrive as
* a blob — pushed down the socket, or fetched with the session's credential, * a blob — pushed down the socket, or fetched with the session's credential,
* which no `img` could carry on its own — so a `blob:` source is the proof the * which no `img` could carry on its own — so a `blob:` source is the proof the
* whole path ran rather than that a picture is merely present. * whole path ran rather than that a picture is merely present.
+4 -4
View File
@@ -1,10 +1,10 @@
import { SiGitea } from "react-icons/si" import { BookOpen } from "lucide-react"
const socialLinks = [ const socialLinks = [
{ {
icon: SiGitea, icon: BookOpen,
href: "https://git.stroblme.de/Fluksio", href: "https://docs.fluksio.com",
label: "Gitea", label: "Documentation",
}, },
] ]
@@ -8,7 +8,7 @@
* Raised from anywhere through `notify()` in `lib/notificationStore`, which is * Raised from anywhere through `notify()` in `lib/notificationStore`, which is
* where the semantics live; this only draws them. * where the semantics live; this only draws them.
* *
* Byte-identical with the portal's copy — see DESIGN-GUIDELINES.md § Sync rule. * See DESIGN-GUIDELINES.md at the repository root for notification styling.
*/ */
import { CircleAlert, CircleCheck, Info, TriangleAlert, X } from "lucide-react" import { CircleAlert, CircleCheck, Info, TriangleAlert, X } from "lucide-react"
@@ -2,10 +2,8 @@ import Avatar from "boring-avatars"
import useAuth from "@/hooks/useAuth" import useAuth from "@/hooks/useAuth"
// Shared avatar identity config, kept in step with the portal's copy // Email provides a stable seed for each account's avatar.
// (index/frontend/src/components/Common/UserAvatar.tsx) so one account renders // These are decorative identity colours, not design tokens.
// the same face in both shells. The seed is the email, the only stable per-user
// id both sides hold. These are decorative identity colours, not design tokens.
export const AVATAR_VARIANT = "marble" as const export const AVATAR_VARIANT = "marble" as const
export const AVATAR_COLORS = [ export const AVATAR_COLORS = [
"#EAAC0F", "#EAAC0F",
@@ -1,10 +1,8 @@
/** /**
* How each look moves. * How each look moves.
* *
* `lib/motion.ts` is the app's own set and is duplicated verbatim in the * Dashboard motion has its own presets, separate from the app shell's
* website repo (`make design-check` holds the two byte-identical), so a * `lib/motion.ts`. The two looks are meant to *feel* different: material settles quickly
* dashboard that moves differently needs its own table rather than an entry
* there. The two looks are meant to *feel* different: material settles quickly
* and squarely, glass carries a little overshoot. * and squarely, glass carries a little overshoot.
* *
* Physics only. What animates is each set's business; this is how it moves * Physics only. What animates is each set's business; this is how it moves
@@ -10,9 +10,8 @@
* *
* The result is written onto the canvas as the plain token variables the app * The result is written onto the canvas as the plain token variables the app
* already themes with (`--background`, `--card`, `--primary`, …), which is the * already themes with (`--background`, `--card`, `--primary`, …), which is the
* same mechanism `.theme-light` and `.theme-dark` use on a subtree. Nothing here touches * same mechanism `.theme-light` and `.theme-dark` use on a subtree. Palette
* `index.css`: that file's token blocks are byte-identical with the website's * overrides leave the app's base tokens in `index.css` unchanged.
* and `make design-check` holds them there.
* *
* Every value written out is a literal hex. uPlot draws on a canvas, and a * Every value written out is a literal hex. uPlot draws on a canvas, and a
* canvas takes neither `color-mix()` nor an unresolved custom property, so a * canvas takes neither `color-mix()` nor an unresolved custom property, so a
+3 -8
View File
@@ -9,12 +9,8 @@
)); ));
/* /*
* Self-hosted font, duplicated verbatim across the fluksio and website repos (a * Self-hosted Inter for body text and the `.font-display` class.
* sync point like `lib/motion.ts`; see the root DESIGN-GUIDELINES.md, * See DESIGN-GUIDELINES.md → Typography.
* Typography). Kept OUT of the byte-identical `@theme inline` / `:root` /
* `.theme-dark` blocks. Inter is used for both body (set on `body` in
* `@layer base`) and the display face applied via the `.font-display` class
* below.
*/ */
@font-face { @font-face {
font-family: "Inter"; font-family: "Inter";
@@ -95,8 +91,7 @@
* *
* `--primary` is #4a7189, not the identity blue #59849b: the latter measures * `--primary` is #4a7189, not the identity blue #59849b: the latter measures
* 4.04:1 against white, below AA for the label sitting on a bg-primary fill. * 4.04:1 against white, below AA for the label sitting on a bg-primary fill.
* #59849b remains the wordmark colour (index/src/assets/fluksio-*.svg). See the root * #59849b remains the wordmark colour. See DESIGN-GUIDELINES.md → Colour tokens.
* DESIGN-GUIDELINES.md → Colour tokens.
* *
* `--primary-nested` is that same hue and saturation a few steps deeper, for * `--primary-nested` is that same hue and saturation a few steps deeper, for
* a reading drawn inside a `--primary` fill — the bar widget's stacked * a reading drawn inside a `--primary` fill — the bar widget's stacked
+1 -3
View File
@@ -5,9 +5,7 @@
* across the app. Mirrors the `--ease-*` / `--duration-*` timing tokens defined * across the app. Mirrors the `--ease-*` / `--duration-*` timing tokens defined
* in `index.css`, so CSS transitions and JS animations stay in sync. * in `index.css`, so CSS transitions and JS animations stay in sync.
* *
* This file is duplicated verbatim in `app` and `index` (the shadcn model, * See DESIGN-GUIDELINES.md at the repository root for the motion rules.
* see ../../DESIGN.md / the root DESIGN-GUIDELINES.md). Keep the two copies
* identical; `make design-check` at the workspace root verifies it.
* *
* Reduced motion: wrap each app root in `<MotionConfig reducedMotion="user">` so * Reduced motion: wrap each app root in `<MotionConfig reducedMotion="user">` so
* transform/layout animations are disabled automatically when the OS requests it. * transform/layout animations are disabled automatically when the OS requests it.
+1 -1
View File
@@ -8,7 +8,7 @@
* Same hand-rolled external store as `connectionStore`, for the same reason: a * Same hand-rolled external store as `connectionStore`, for the same reason: a
* `useSyncExternalStore` snapshot is the whole requirement. * `useSyncExternalStore` snapshot is the whole requirement.
* *
* Byte-identical with the portal's copy — see DESIGN-GUIDELINES.md § Sync rule. * See DESIGN-GUIDELINES.md at the repository root for notification styling.
*/ */
export type NotificationSeverity = "error" | "warning" | "info" | "success" export type NotificationSeverity = "error" | "warning" | "info" | "success"
+1 -1
View File
@@ -57,7 +57,7 @@ test.beforeAll(async ({ browser }) => {
await api(page, `/messages/${flowName}.label`, { await api(page, `/messages/${flowName}.label`, {
method: "POST", method: "POST",
data: { data: {
value: "ubuntu-gpu-node-01.lab.internal (numpy 2.5.2, cuda 12.4)", value: "research-gpu-worker.example.test (numpy 2.5.2, cuda 12.4)",
}, },
}) })
await page.close() await page.close()
+12 -8
View File
@@ -10,22 +10,26 @@ pip install fluksio-worker
fluksio-worker \ fluksio-worker \
--url wss://api.example.com/api/v1/workers/attach \ --url wss://api.example.com/api/v1/workers/attach \
--labels gpu,cuda12 \ --labels gpu,cuda12 \
--python /opt/torch-venv/bin/python --python /path/to/node-venv/bin/python
``` ```
`--python` is the interpreter node code runs on, which is how this machine keeps Replace the address and interpreter path with your own. Create a token on the
its own wheels without the engine ever installing them. Mint a token from the engine with `POST /api/v1/workers/tokens` and set `FLUKSIO_WORKER_TOKEN` before
engine with `POST /api/v1/workers/tokens`; it carries this worker's name and starting the worker. See the [worker guide](https://docs.fluksio.com/code/workers/)
labels, and `--labels` may repeat those but not change them. for the commands.
It sends the token as an `Authorization` header, which an engine from before `--python` selects the environment holding your node dependencies. Omit it to
that does not read: upgrade the engine first, then its workers. use the worker's own interpreter. The token fixes the worker's name and labels;
`--labels` may repeat those labels but cannot change them.
Engine and worker must use compatible protocols. Upgrade the engine first,
then its workers.
A plain `ws://` address is refused unless the engine is on this machine, since A plain `ws://` address is refused unless the engine is on this machine, since
the token and node secrets would cross the network in clear; `--insecure` the token and node secrets would cross the network in clear; `--insecure`
allows it on a network you trust. allows it on a network you trust.
Linux and macOS. Requires Python 3.12 or newer on Linux or macOS.
## License ## License
+1 -2
View File
@@ -54,8 +54,7 @@ nav = [
{ "Payload types" = "reference/payload-types.md" }, { "Payload types" = "reference/payload-types.md" },
{ "Configuration" = "reference/configuration.md" }, { "Configuration" = "reference/configuration.md" },
] }, ] },
# The roadmap, the design guidelines and the architecture material stay in # Only docs/ is published. Contributor guidance stays at the repository root.
# the repositories — repo-only, intentionally not published.
] ]
# Drop the "Made with Zensical" generator notice from the footer. # Drop the "Made with Zensical" generator notice from the footer.