Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Publish the documentation site: docs.fluksio.com

Browse Source 
A zensical site under docs/, served by a new `docs` compose service behind
Traefik, built with --strict in CI. Same pattern the sibling n3xd workspace
uses.

Getting started splits the way the landing page does — one path is
`pip install fluksio` and a training script, the other is a Docker stack and
an afternoon in the browser — because the two audiences will not spend the same
amount of time. Everything after that is shared: the concepts, the web
interface (app and portal), the CLI and the API, and a reference for node types,
payload types and configuration.

The three flow guides move here from the docs submodule rather than being
copied, so there is one version of them.

Styling mirrors DESIGN-GUIDELINES.md: the app's token palette remapped onto
Material's variables in both schemes, Inter, the 16px panel radius, and the one
terracotta accent spent on the facility lane of the audience split.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M7Xv3cJEW5c8AXxn2hoojV
This commit is contained in:
stroblme
2026-08-22 05:55:34 +02:00
co-authored by Claude Opus 5
parent 8632d975e6
commit 11e032386b
34 changed files with 4167 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/getting-started/data-science.md
+323
View File
@@ -0,0 +1,323 @@
# Getting started: data science
You have a training script. It works. What you do not have is any reliable
answer to "what was the learning rate on the run that got 94%?", and the
`results_final_v3_ACTUAL.csv` in your home directory is not helping.
This page adds Fluksio to what you already have. It takes about five minutes,
installs one Python package, and does not ask you to restructure anything.
## Install
```sh
pip install fluksio
fluksio serve
```
That is the whole installation. No Docker, no database server, no ports to
open. The first run prints something like:
```text
Created the admin account admin@example.com
password: k3Qm-8vTpLdX
Shown once. Change it from the dashboard.
Fluksio 0.1.0 — data in /home/you/.fluksio
API http://127.0.0.1:8000/api/v1
No portal. Pair this installation with:
fluksio enroll <code> --portal https://hub.example.com
```
**Write that password down.** It is shown once and it is how you authenticate
from here on.
Everything the installation owns lives in `~/.fluksio`: a SQLite database, a
git repository holding your flows, the artifact store, and a virtual
environment your node code runs in. Move it with `--data-dir`, which is worth
doing on a cluster where `$HOME` is a network filesystem — SQLite's
write-ahead log does not work on NFS, and `fluksio serve` warns you when it
notices.
!!! tip "Keep it running"
The engine is meant to be resident. That is the whole reason submitting a
run costs about 15 ms instead of the second a project-bootstrapping
orchestrator spends before it does anything. Leave it in a `tmux` window,
or write a small `systemd --user` unit for it.
## Get a token
Everything below is the HTTP API. Grab a token once:
```sh
export FLUKSIO=http://127.0.0.1:8000/api/v1
export TOKEN=$(curl -s -X POST $FLUKSIO/login/access-token \
-d "username=admin@example.com&password=k3Qm-8vTpLdX" | jq -r .access_token)
```
While you are experimenting, the interactive schema at
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
## Tell it about your packages
Node code runs in `~/.fluksio/user-venv`, deliberately separate from the
environment Fluksio itself is installed in — so a pin of yours can never
collide with one of ours. That venv starts empty, so the first thing to do is
say what your script imports:
```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"requirements\": $(jq -Rs . < requirements.txt)}"
```
It is a pip manifest, installed with `uv pip sync`, and it is versioned
alongside your flows — so what a run imported is recorded with what it ran.
Adding a package takes effect immediately; nothing restarts.
??? note "Already have a venv you would rather not duplicate?"
Attach it as a worker instead of reinstalling into it. Mint a token, then
point the agent at your existing interpreter:
```sh
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"name": "laptop"}'
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
--token "$WORKER_TOKEN" --labels local --python "$(which python)"
```
Then mark the node `"device": "local"` and it runs on that interpreter. It
is the same mechanism that sends a node to a GPU box, and it is worth
knowing about early — see [Remote workers](../code/workers.md).
## Wrap your training script
A **flow** is a graph of nodes. A **batch flow** is one that runs on demand
from parameters to a result, which is what an experiment is. Your existing
script becomes the body of a node.
Say your script looks roughly like this:
```python
def train(lr, epochs):
model = build_model()
for epoch in range(epochs):
loss = step(model, lr)
print(f"epoch {epoch}: {loss}") # ← this is what you are losing
torch.save(model.state_dict(), "weights.pt")
return loss
```
Two changes turn it into a node:
```python
"""Fit the model. A generator, so numbers escape while it is still running."""
import fluksio
def process(lr, epochs):
model = build_model()
for epoch in range(int(epochs)):
loss = step(model, lr)
yield {"loss": loss} # ← published now, on the loss port
torch.save(model.state_dict(), "weights.pt")
return {
"weights": fluksio.save_artifact("weights.pt"),
"final_loss": loss,
}
```
`yield` replaces `print`. Each one publishes on the node's `loss` port the
instant it happens, and the run keeps every value as a series — which is why
Fluksio has no `log_metric()` call. A metric that leaves through a port is a
message like any other: a chart can bind to it, a downstream node can consume
it, and it shows up on the canvas. A metric that escapes through a logging
function is invisible to all three.
`fluksio.save_artifact` handles the things too big to be messages — a
checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a
small reference. Nothing changes about how you write the file.
!!! note "Where a `yield` cannot reach"
If the number comes from inside somebody else's callback — Keras, Lightning,
HuggingFace `Trainer` — `fluksio.emit(loss=...)` writes the same port the
same way:
```python
model.fit(callbacks=[LambdaCallback(
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
```
## Create the flow
There is no scaffolding command yet, so a flow is created by PUTting its
definition. That is a fifteen-line script you run once:
```python
"""Create the `train` flow. Run once; edit it in the canvas afterwards."""
import httpx
API = "http://127.0.0.1:8000/api/v1"
api = httpx.Client(base_url=API, timeout=60)
token = api.post(
"/login/access-token",
data={"username": "admin@example.com", "password": "k3Qm-8vTpLdX"},
).json()["access_token"]
api.headers["Authorization"] = f"Bearer {token}"
api.put("/flows/train", json={
"name": "train",
"title": "Model training",
# Batch: nothing is activated, nothing fires until a run asks.
"mode": "batch",
# Its inputs are the run's parameters, with the values a run gets when it
# names none.
"inputs": [
{"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01},
{"spec": {"name": "epochs", "dtype": "int"}, "initial": 50},
],
# What a run reports as its result.
"outputs": ["final_loss", "weights"],
"nodes": [{
"id": "train",
"type": "python",
"title": "Fit the model",
# An *idle* timeout once the node streams: this is how long it may go
# quiet, not how long it may run.
"timeout": 600,
"requires": [
{"name": "lr", "dtype": "float"},
{"name": "epochs", "dtype": "int"},
],
"provides": [
# `stream` says this port publishes repeatedly during one execution.
{"name": "loss", "dtype": "float", "stream": True},
{"name": "final_loss", "dtype": "float"},
{"name": "weights", "dtype": "artifact"},
],
}],
}).raise_for_status()
api.put("/flows/train/nodes/train/source",
json={"code": open("train_node.py").read()}).raise_for_status()
version = api.get("/flows/train", params={"draft": True}).json()["definition"]["version"]
api.post("/flows/train/publish", json={"version": version}).raise_for_status()
print("published")
```
Two things worth noticing. Ports are declared, not inferred — `process(lr,
epochs)` gets its arguments from the ports of the same name, and the types are
checked on every value. And saving writes a *draft*; `publish` is what the
engine picks up. That separation is what lets you edit a flow that is running.
## Run it
```sh
curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"params": {"lr": 0.003, "epochs": 200}, "seed": 7}'
```
It answers immediately with a queued run — training is measured in hours, so
nothing waits for it. A parameter you did not declare, or one of the wrong
type, is refused with a 422 before anything executes.
Then, whenever you like:
```sh
curl -s $FLUKSIO/runs/<id> -H "Authorization: Bearer $TOKEN" | jq
curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $TOKEN" | jq
```
The run carries its parameters, a digest of them, the seed, its result, how
long each node took, what it logged, and every artifact it produced. That is
the answer to "what was the learning rate on the run that got 94%?".
## Sweep it
A grid search and an ensemble are the same call — you build the list, Fluksio
runs them in parallel:
```sh
curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"runs": [
{"params": {"lr": 0.001}, "seed": 1},
{"params": {"lr": 0.003}, "seed": 1},
{"params": {"lr": 0.010}, "seed": 1}
]}'
```
They share a `group_id`, so the sweep is `GET /runs?group=…`. Running them
concurrently is safe because each run gets a state backend of its own — two
runs of one flow cannot overwrite each other's values.
Compare the curves in one call:
```sh
curl -s "$FLUKSIO/runs/series/compare?ids=$A,$B,$C&metric=train.loss" \
-H "Authorization: Bearer $TOKEN"
```
which answers in exactly the shape a chart widget draws.
## Small scripts you are just playing with
The same machinery, minus the ceremony. If what you want is "keep a record of
every version of this thing I keep tweaking", note that **your flows are
already a git repository**:
```sh
cd ~/.fluksio/flows
git log --oneline
```
Every save is a commit — the node source, the parameters, the graph. A run
records the commit it ran at, so `git show` on that hash is literally the code
that produced the number. You get the versioning without adopting anything.
For quick iteration, keep the flow small (one node is fine), keep the engine
running, and submit from wherever you are working:
```python
import httpx
run = httpx.post(f"{API}/runs/flows/train", json={"params": {"lr": lr}},
headers=auth).json()
```
A submit is around 15 ms, so calling that in a loop is a reasonable thing to do.
## Get a browser onto it
The pip install gives you the engine and the API, not a web interface — a
machine with no inbound route cannot serve one usefully anyway. To see the
canvas, the run history and live loss curves, pair the installation with a
portal, which serves the dashboard from its side:
```sh
fluksio enroll <claim-code> --portal https://hub.fluksio.com
```
Get the claim code from the portal under **Installations → Add installation**.
Nothing needs to be exposed: your machine dials out and holds the connection
open. See [Accounts and the portal](../interface/portal.md).
If you would rather stay entirely offline, run the dashboard SPA yourself from
the app's Docker image — see the [facility path](facility-automation.md), which
is the same stack.
## Where to go next
- [Runs: pipelines that finish](../concepts/runs.md) — the full picture:
artifacts, sweeps, durability, what happens when your engine dies mid-training
- [Writing node code](../code/nodes.md) — generators, settings, what a node may
and may not do
- [Remote workers](../code/workers.md) — send the training node to the GPU box
and keep the rest on your laptop
- [The flow editor](../interface/flow-editor.md) — once you have a portal, this
is where the graph gets easier to change than the script did
docs/getting-started/facility-automation.md
+272
View File
@@ -0,0 +1,272 @@
# Getting started: facility automation
You have a box in a cupboard, a handful of sensors that already publish
somewhere, and an ambition to make the house do something about them. This
page brings up a Fluksio instance you can leave running for years, then wires
the first sensor through to a dashboard.
Budget an afternoon. Most of it is the browser, which is the point: you will be
in this interface a lot, so it is worth learning it properly.
## What you are standing up
```text
sensors ──MQTT──▶ ┌──────────────┐ ──▶ InfluxDB (history)
│ Fluksio │
HTTP / webhooks ──▶│ flow engine │ ──▶ dashboards (what you look at)
│ │
schedules ────────▶└──────────────┘ ──▶ MQTT / HTTP (what you control)
│
└── workers on other boxes (optional)
```
One engine holds every flow. Nothing here is a plugin you install separately —
the broker client, the time-series writer, the dashboards and the alerting are
all part of the same process, editing the same graph.
## Prerequisites
- Docker and Compose v2 on the host
- A hostname you can point at it. `fluksio.local`, a subdomain, or just
`localhost` if you only ever reach it from that machine
- Optionally: an MQTT broker and an InfluxDB you already run. If not, the stack
can start both for you
## Bring up the stack
Clone the app repository and start it:
```sh
git clone https://git.stroblme.de/Fluksio/app.git ~/fluksio
cd ~/fluksio
cp .env.example .env
$EDITOR .env # DOMAIN, FIRST_SUPERUSER, ENVIRONMENT=production
make up
```
`.env` is the whole configuration. The four settings that matter on day one:
| Setting | What it does |
|---|---|
| `DOMAIN` | the hostname everything is served under; the SPA lands on `app.${DOMAIN}` and the API on `api.${DOMAIN}` |
| `FIRST_SUPERUSER` | the account you sign in with |
| `FIRST_SUPERUSER_PASSWORD` | leave it as `changethis` and one is generated for you |
| `ENVIRONMENT` | `production` closes the interactive API schema; `local` leaves it open |
Everything the installation owns — the database, your flows, secrets,
artifacts, the packages your node code imports — is on one Docker volume.
Backing that volume up is backing up the installation.
!!! tip "Reverse proxy"
The stack emits Traefik labels and ships a Traefik you can bring up
alongside it (`docker/compose.traefik.yml`). If you already run Nginx
Proxy Manager or Caddy, attach it to the `proxy` network instead and
forward `app.${DOMAIN}` → `fluksio-app:80` and `api.${DOMAIN}` →
`fluksio-api:8000`.
??? note "Even smaller: no Docker at all"
`pip install fluksio && fluksio serve` gives you the same engine with no
containers, keeping its data in `~/.fluksio`. What it does not give you is
the web interface, which the SPA container serves — so you would drive it
from the API, or [pair it with a portal](../interface/portal.md) that
serves the dashboard for you. Good for a Raspberry Pi that only runs flows;
less good as your main instance.
Open `http://app.${DOMAIN}` and sign in. You should be looking at Home: an
empty brain graph, a health summary, and a flow list with nothing in it.
## Your first flow
Go to **Flows → New flow** and call it `house`. You land on the canvas.
A flow is a set of nodes that talk to each other through **named messages**.
You do not draw wires: a node says which messages it needs and which it
produces, and the canvas draws the graph that follows from those names. That
sounds like a small difference and turns out to be a large one — renaming is
safe, fan-in is free, and two flows can share a value by naming it.
### Read a sensor
Press **Add node** (or ⌘K / Ctrl-K, which opens the command palette) and pick
**MQTT**. In its panel on the right:
- **Broker host** — your broker's hostname, `mosquitto` if you are using the
one the stack can start
- **Topic** — map each output to a topic:
`{"living_temperature": "zigbee2mqtt/living/temperature"}`
- **Provides** — add one output port named `living_temperature`, type `float`
That is a working node. Press **Publish** (⌘S) and the engine picks it up.
The canvas now draws your node with a live value on its output as soon as the
broker sends one. Click the wire to see the last payload and its history.
### Do something with it
Add a **Function** node. This is a Python node — the code editor opens in its
panel:
```python
def process(living_temperature, comfortable=21.0):
"""Ask for heat when the room is below the comfort point."""
return {"heat_wanted": living_temperature < comfortable}
```
Declare `living_temperature` as an input (type `float`) and `heat_wanted` as an
output (type `bool`). `comfortable` is not a port — it is a **setting**,
because it is a constant of this node rather than something the graph carries.
It shows up as a field in the node's Settings section.
The canvas now draws MQTT → your function, because the message names line up.
Nothing else was needed.
### Act on it
Add a second **MQTT** node, this time with `heat_wanted` as an *input*, and a
topic mapping to whatever your relay listens on. A node with inputs publishes;
a node with outputs subscribes.
Publish the flow. You have a thermostat.
!!! warning "Test before it touches a relay"
A flow can be **paused** (it holds messages instead of running them) and
**stepped** (release exactly one). Together with the run button — which
injects a value by hand — that is how you convince yourself the logic is
right before the contactor finds out. Both live on the dock at the bottom
of the canvas.
## Store the history
A live value is enough to control something and useless for answering "was last
February colder?". That is what the **InfluxDB** node is for.
Add one, and configure it to write the message you already have:
```json
{
"url": "http://influxdb:8086",
"token": {"$secret": "influx-token"},
"org": "home",
"bucket": "sensors",
"writes": {
"living_temperature": {
"measurement": "environment",
"field": "temp_c",
"tags": {"room": "living"}
}
}
}
```
Give it `living_temperature` as an input and every value that passes gets a
point.
Note the token. Credentials never sit in a flow: `{"$secret": "influx-token"}`
is a reference into an encrypted store, and the editor renders those fields as
a secret picker. Add the actual value once under **Secrets**. Flows are a git
repository you might well push somewhere — this is what keeps a password out of
it.
Reading back is the same node with `queries` instead of `writes`, or — for
anything a chart asks for — a pair of small Python nodes on either side that
build a Flux query and shape its rows. That indirection is deliberate: the
database node holds the connection and nothing else, so a dashboard widget
never learns which database answered it.
## Put it on a screen
**Dashboards → New dashboard**, then drag widgets onto the grid and bind each
one to a message. A gauge on `house.living_temperature`, a switch on
`house.heat_wanted`, a chart on the history.
Widgets are typed the same way ports are: a switch binds to a `bool`, a gauge
to a number, an agenda to a `list`. Bind it wrong and the editor says so rather
than drawing nothing.
Controls work in the other direction — a switch on a dashboard publishes the
message it is bound to, exactly as a node would. The canvas draws it as a
labelled endpoint feeding the nodes that read it, so a value never appears from
nowhere.
For a tablet on the wall, see [Dashboards and panels](../interface/dashboards.md):
a panel is a named device, it pairs with a six-character code instead of a
login, and it can only reach the dashboards you gave it.
## Spread it across machines
You now have one box doing everything. Two reasons to change that: something
lives on a different network, or something needs hardware the engine's host
does not have.
The unit of distribution is the **worker**. It runs the code of nodes you mark
for it, and it dials *out* to the engine — so the Pi in the shed does not need
an inbound route, and the engine does not need to reach it.
On the engine, mint a token:
```sh
curl -X POST https://api.${DOMAIN}/api/v1/workers/tokens \
-H "Authorization: Bearer $TOKEN" -d '{"name": "shed-pi"}'
```
On the other machine:
```sh
pip install fluksio-worker
fluksio-worker \
--url wss://api.${DOMAIN}/api/v1/workers/attach \
--token "$WORKER_TOKEN" \
--labels shed,gpio \
--parallel 2
```
Then mark the node that talks to the shed's GPIO with `device: shed`, and it
runs there. Everything else stays where it is. A node bound to a label no
attached worker carries simply waits rather than failing, so you can write the
flow before the hardware arrives.
!!! note "What a worker is not"
It is not a second engine. Subscriptions, schedules, webhooks and the
dashboards all stay in one process — that is what keeps a value having one
definition. A worker executes node bodies, nothing else. Scaling *the
engine* to several processes is not supported: run one.
## Make it tell you when something breaks
Under **Alerts**, add a channel and a rule. Channels are ntfy (a push
notification on your phone), SMTP, a webhook, or a message a dashboard
notification widget reads.
What you probably want on day one is everything, to ntfy:
- add an ntfy channel with your server and topic
- add a rule with no events ticked, which means *all of them*
The engine deduplicates aggressively — the same node failing every second is
one alert, not thirty-six thousand — and caps the total at ten an hour however
bad it gets. See [Secrets, modules and alerts](../interface/operations.md).
## Keep it alive
- **Back up the data volume.** That is the database, the flows, the secrets and
the artifacts. Everything else is rebuildable.
- **Watch Home.** The health summary names what is wrong — a quarantined flow, a
node that will not load, a stalled queue, a flow that cannot run because its
graph does not validate.
- **Flows are git.** `git log` inside the flow store is the history of every
change anyone made, and reverting one is a revert.
## Where to go next
- [Flows, nodes and messages](../concepts/flows.md) — the model, properly
- [Node types](../reference/node-types.md) — everything you can put on a canvas
without writing Python
- [The flow editor](../interface/flow-editor.md) — the canvas, in detail
- [Keeping state in a flow](../concepts/state.md) — running totals, debounces,
and the one rule that makes them safe
- [Accounts and the portal](../interface/portal.md) — reach the installation
from outside the house without opening a port
docs/getting-started/index.md
+72
View File
@@ -0,0 +1,72 @@
# Pick your starting point
People arrive at Fluksio from two directions, and the honest answer to "how do
I set this up?" is different for each — not just in the commands, but in how
much of an afternoon it is reasonable to spend.
Pick the one that sounds like you. Everything past this section is the same for
both.
<div class="fluksio-lanes" markdown>
<div class="fluksio-lane fluksio-lane--science" markdown>
### Data science
*"I have a training script. I want to stop losing track of what I ran."*
One `pip install`, one command, and you are writing Python again. No Docker, no
database, no ports to open. Flows are files, runs are rows, and the metrics are
just the numbers your loop already produces.
[Set up for experiments →](data-science.md)
</div>
<div class="fluksio-lane fluksio-lane--facility" markdown>
### Facility automation
*"I have a homelab and a pile of sensors. I want them to do something."*
A stack you bring up once and leave running: the engine, a broker, a
time-series database, dashboards, alerting. Most of the work happens in the
browser, and it is worth doing properly because you will live in it.
[Set up a homelab instance →](facility-automation.md)
</div>
</div>
## Not sure?
Some rough tells:
| | Data science | Facility automation |
|---|---|---|
| **The flow** | starts, finishes, has a result | never ends |
| **You mostly** | write Python | wire nodes in the browser |
| **Time to first result** | a few minutes | an afternoon |
| **Runs on** | your laptop, or a login node | a box in a cupboard |
| **Data lives in** | SQLite beside the flows | InfluxDB, usually |
| **The thing you look at** | run history and loss curves | a dashboard, maybe on a wall |
If both describe you — a lab with instruments to drive *and* models to
fit — start with the data-science path. It is the smaller installation, and it
grows into the other one without being reinstalled: the same engine, the same
flows, just more of them running all the time.
## What is the same either way
Whichever door you came in:
- **Flows are files in a git repository.** Every save is a commit. You can read
the history with ordinary git, and you can copy a flow between installations
by copying a directory.
- **Editing is separate from running.** You edit a draft; the engine keeps
running what was published until you publish.
- **Nodes are typed.** A port declares what it carries, and a mismatch is
caught at edit time rather than at three in the morning.
- **Everything the browser does is an API call.** The dashboard is a client of
the same REST API you can script against.