Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Publish the documentation site: docs.fluksio.com

Browse Source 
A zensical site under docs/, served by a new `docs` compose service behind
Traefik, built with --strict in CI. Same pattern the sibling n3xd workspace
uses.

Getting started splits the way the landing page does — one path is
`pip install fluksio` and a training script, the other is a Docker stack and
an afternoon in the browser — because the two audiences will not spend the same
amount of time. Everything after that is shared: the concepts, the web
interface (app and portal), the CLI and the API, and a reference for node types,
payload types and configuration.

The three flow guides move here from the docs submodule rather than being
copied, so there is one version of them.

Styling mirrors DESIGN-GUIDELINES.md: the app's token palette remapped onto
Material's variables in both schemes, Inter, the 16px panel radius, and the one
terracotta accent spent on the facility lane of the audience split.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M7Xv3cJEW5c8AXxn2hoojV
This commit is contained in:
stroblme
2026-08-22 05:55:34 +02:00
co-authored by Claude Opus 5
parent 8632d975e6
commit 11e032386b
34 changed files with 4167 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/concepts/flows.md
+183
View File
@@ -0,0 +1,183 @@
# Flows, nodes and messages
Three ideas hold the whole system up. They are worth twenty minutes, because
almost everything else follows from them.
## A flow is a graph you did not draw
A **flow** is a set of **nodes**. Each node declares the messages it needs
(`requires`) and the messages it produces (`provides`). The graph is whatever
those declarations imply:
```python
# node "read"
def process():
return {"temperature": read_sensor()}
# node "decide"
def process(temperature, setpoint=21.0):
return {"heat": temperature < setpoint}
```
`decide` is downstream of `read` because it needs `temperature` and `read`
produces it. Nobody drew a wire.
This is the one structural decision everything else rests on, so it is worth
being explicit about the consequences:
- **Fan-in is free.** Two nodes providing `temperature` are two producers of
one message. The consumer does not change.
- **A wire cannot be wrong.** There is no wire. There is a name that either
matches or does not, and the canvas tells you at edit time which it is.
- **Layout is not a document.** The canvas computes the arrangement, so a flow
has no stored positions to maintain, merge or fight over.
- **Flows stay small.** A graph nobody can hand-arrange is one worth keeping
small — which is the intent. Several atomic flows that name each other beat
one flow with sixty nodes in it.
### Message names are namespaced
Inside flow `house`, a message named `temperature` is really
`house.temperature`. A bare name is qualified with its own flow; a dotted name
is used as written. That is how two flows share a value:
```python
# in flow `dashboard`, reading a message that flow `house` produces
def process(house_temperature): # port bound to "house.temperature"
...
```
The canvas draws messages arriving from another flow as labelled endpoints, so
you can see where they come from without opening the other flow.
## A node is a function with declared ports
Most nodes are **Function** nodes: a Python file defining `process(...)`. Its
arguments are its input ports by name; its return value is a dict keyed by
output ports.
```python
def process(reading, unit="C"):
return {"shown": reading if unit == "C" else reading * 1.8 + 32}
```
`reading` is a port. `unit` is a **setting** — a constant of this node's code,
typed into its panel and stored with the flow. Both arrive as arguments, which
is why a setting may not share a name with a port. See
[Where a node's values come from](values.md).
The rest of the node types are the ones that would be tedious or unsafe to
write yourself: MQTT, HTTP, InfluxDB, schedules, switches, notifications. Each
one is configured by filling in a form the editor generates from its parameter
schema, so they all behave the same way. The full list is in
[Node types](../reference/node-types.md).
### Ports are typed
A port declares a `dtype`: `float`, `int`, `str`, `bool`, `json`, `record`,
`list`, `series` or `artifact`. Every value that passes through is checked
against it.
Types are not decoration. They are what lets the dashboard editor offer you
only the messages a gauge can actually draw, and what lets the canvas refuse a
binding before anything runs. See [Payload types](../reference/payload-types.md).
Everything on the wire is JSON. Bytes — a checkpoint, an image, a model —
travel as an `artifact`: the bytes go to a content-addressed store and the
message carries a small reference to them.
### Nodes are pure
A node is called with the values of the messages it declares and returns the
values of the messages it provides. There is no context object, no global
store, no handle to reach for.
That is deliberate: a node with hidden state cannot run twice in parallel,
cannot be replayed, and cannot be moved to another machine. Plenty of real
automations do need to remember something, and there is a specific way to say
so — see [Keeping state in a flow](state.md).
## Two shapes of flow
Set `mode` on the flow:
| | `live` (default) | `batch` |
|---|---|---|
| Runs | continuously | once per run, on request |
| Started by | subscriptions, schedules, webhooks | `POST /runs/flows/{name}` |
| Ends | never | when the graph drains |
| Keeps | the last value of each message | a run record: params, result, metrics, artifacts |
| Is | a thermostat, an ETL job on a cron | an experiment, a CI-style job |
A batch flow is built and validated like any other, appears on the same canvas
and is type-checked the same way. It is simply never *activated*: no
subscriptions, no schedules, no webhooks. See
[Runs: pipelines that finish](runs.md).
## Editing is separate from running
Every flow has a published version and, while you are working, a draft.
- **Saving** writes the draft. The engine keeps running the published version.
- **Publishing** promotes the draft. The engine reloads and picks it up.
- **Discarding** throws the draft away.
The store is a git repository — `flow.json` for the structure, `nodes/*.py` for
the code — and each save is a commit. So a flow's history is readable with
ordinary git tooling, and copying a flow between installations is copying a
directory.
Saving carries the version you last saw. If someone else saved in between, you
get a 409 instead of quietly overwriting their work.
## What can be wrong, and when you find out
The canvas validates continuously and names problems on the nodes they belong
to:
| Issue | What it means |
|---|---|
| `unconnected_input` | a port needs a message nothing in reach provides |
| `missing_initial_value` | the message exists but has never held a value, and nothing will give it one |
| `cycle` | A waits for B and B waits for A — nothing could ever start |
| `self_loop_needs_initial` | a node reads a message it also writes, with no starting value |
| `node_error` | the node's code did not load: a syntax error, a missing import |
| `unauthenticated_hook` | advisory — a webhook with no shared secret is open to anyone |
A flow with any of these except the advisory one does not run. The health
summary on Home counts them, so "why is nothing happening?" has an answer that
does not involve reading logs.
## What happens at runtime
- **A flow can be started and stopped.** Stopped means its subscriptions and
schedules are torn down.
- **A flow can be paused and stepped.** Paused holds messages instead of
running them; step releases exactly one. This is how you test something
before it moves a relay.
- **A failing node does not take the flow down.** It reports an error, keeps
its last error visible after it recovers, and can fire an alert.
- **A flow whose background tasks keep crashing is quarantined.** The engine
stops restarting them and says so, rather than spinning. Publishing a change
gives it another chance.
## Values that arrive from outside
Some messages are not computed by any node: a dashboard control writes them,
the API publishes them, a batch run passes them in. Declare those as the flow's
**inputs**, with the value they start from:
```json
{"inputs": [{"spec": {"name": "setpoint", "dtype": "float"}, "initial": 21.0}]}
```
Without that, the node reading `setpoint` waits for something nothing provides,
and the canvas says so. With it, the flow starts at 21.0 and whatever writes
the message afterwards takes over.
## Where to next
- [Where a node's values come from](values.md) — ports, settings, flow inputs
- [Keeping state in a flow](state.md) — the sanctioned way to remember something
- [Runs: pipelines that finish](runs.md) — the batch half of the engine
- [Writing node code](../code/nodes.md) — the practical side of `process()`
docs/concepts/runs.md
+271
View File
@@ -0,0 +1,271 @@
# Runs: pipelines that finish
A flow that automates a house never ends — a value arrives, nodes fire, and it
waits for the next one. A research pipeline is the other shape: parameters go
in, stages execute in order, and at some point it is *done* and has produced
something worth keeping. Fluksio calls the second one a **run**, and it is the
same engine either way.
This is what makes Fluksio usable where Kedro, MLflow or ClearML would be: a
run has parameters that identify it, a result, per-step metrics, artifacts and
a place in a queryable history — without a second server, and without paying a
project bootstrap on every execution.
## A batch flow
Set `mode: "batch"` on the flow and name the messages its result should hold:
```json
{
"name": "train_polymer_gnn",
"mode": "batch",
"outputs": ["final_loss", "report"],
"inputs": [
{"spec": {"name": "lr", "dtype": "float"}, "initial": 0.1},
{"spec": {"name": "steps", "dtype": "int"}, "initial": 3000}
],
"nodes": [{"id": "train", "timeout": 7200, "device": "gpu", "...": "..."}]
}
```
A batch flow is built and validated like any other — it appears on the canvas,
its ports are type-checked — but it is never *activated*: no subscriptions, no
schedules, no webhooks. It runs when a run asks it to, and not otherwise.
Its `inputs` are its parameters. A run supplies values for them; anything it
does not supply keeps the declared initial value.
One thing a batch flow may not do is rate-limit a port (`interval`). A rate
limit holds a value back for a timer to release, and a run has no timer — the
value would be dropped rather than delayed, so submitting is refused instead.
## Submitting
```bash
curl -X POST $FLUKSIO/runs/flows/train_polymer_gnn \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"params": {"lr": 0.3, "steps": 4000}, "seed": 7}'
```
The answer is immediate and the run is `queued`; a training run is measured in
hours, so nothing waits for it. Poll `GET /api/v1/runs/{id}` for its status,
result, per-node record and artifacts.
Wrong parameters are refused before anything executes — an undeclared name, or
a value of the wrong type, comes back as a 422 naming the problem.
### Sweeps
An ensemble is the same parameters at different seeds; a grid search is the
parameters spread out. Both are one call, and the caller builds the list:
```bash
curl -X POST $FLUKSIO/runs/flows/train_polymer_gnn/sweep \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{
"runs": [{"params": {"lr": 0.1}, "seed": 1}, {"params": {"lr": 0.3}, "seed": 1}]
}'
```
They share a `group_id`, so `GET /api/v1/runs?group=…` is the sweep, and they
execute in parallel. That is safe because **each run has a state backend of its
own**: message names are global keys, so two runs of one flow would otherwise
overwrite each other's values. They do not.
## Producing values before you are finished
A training loop has numbers worth keeping long before it has a result. Those
numbers are **outputs**, not logs: a node declares a port for them and produces
them over time, which in Python is a generator.
```python
def process(lr, steps):
loss = 1.0
for _ in range(steps):
loss = train_one_step(lr)
yield {"loss": loss} # published now, on the `loss` port
return {
"weights": fluksio.save_artifact(dump(model), "weights.npz"),
"final_loss": loss,
}
```
Mark the port it streams on, so the flow says what it does:
```json
{"name": "loss", "dtype": "float", "stream": true}
```
Every yield is published the instant it happens — same port, same type check,
same place on the canvas as any other value. Whatever the generator `return`s
is the node's result, and is what downstream nodes read. If you never
`return`, the last thing you yield is the result instead.
This is the whole reason the framework does not have a logging API. A metric
that escapes through `log_metric()` is undeclared: invisible to validation,
absent from the canvas, and stored somewhere the graph knows nothing about.
A metric that leaves through a port is a message — so a chart binds to it
directly, a downstream node can consume it, and the run keeps its series
without anyone asking.
**Where a yield cannot reach** — the value comes from inside somebody else's
callback, and they call you rather than the other way round — `fluksio.emit`
writes the same ports the same way:
```python
import fluksio
def process():
model.fit(callbacks=[LambdaCallback(
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
return {"weights": ...}
```
### What a run does with them
Every number a node emits is kept as the run's series, stepped by the count of
emissions on that message. Read one back with
`GET /api/v1/runs/{id}/metrics?name=<flow>.loss`, or compare runs:
```
GET /api/v1/runs/series/compare?ids=<a>,<b>,<c>&metric=<flow>.loss
```
That answers in the `series` shape a chart widget already draws, so three
training curves side by side is a widget binding. During a run the values also
arrive live on the flow socket, so a chart bound to the port fills in as the
training goes.
A streaming port may set `interval` to thin out what reaches the canvas — the
run's history still keeps every value, because the interval is asking for the
display not to be flooded, not for the curve to have holes in it.
Emitting has a second effect: **a node's timeout measures silence, not
duration**. A node that yields every few seconds can run for hours under a
`timeout` of 300; one that says nothing for longer than its timeout is killed.
Set `timeout` on a long node to how long it may plausibly go quiet.
In a *live* flow, an emission also wakes whatever is downstream of it, exactly
as a subscriber publishing does. In a run it does not: a run's graph is
scheduled once, and three thousand mid-node cascades would leave "the run has
finished" with nothing to mean.
## Artifacts
Bytes never travel as a message. `save_artifact` writes them to a
content-addressed store and returns a small reference — digest, size, media
type, name — which is what an `artifact`-typed port carries:
```python
def process(weights): # requires: weights, dtype "artifact"
path = fluksio.load_artifact(weights)
...
```
Because the address is the content's hash, a sweep whose fifty configs share
one preprocessed input stores it once, and a reference stays valid wherever
the store is reachable from. Artifacts a run produced are listed on it and
downloadable at `GET /api/v1/artifacts/{digest}`.
## Objects that cannot be serialized
A live model, a `DataLoader`, a JAX-compiled function — these do not cross a
node boundary, and no framework flag will make them. There are exactly two
patterns, and they are both deliberate:
* **Keep them in one node.** Stages that must share live memory are one node.
Building the model and training it is one stage; the fact that Kedro would
make them two nodes is Kedro's problem, not a structure worth reproducing.
* **Cross at a checkpoint.** Save what matters as an artifact and rebuild from
it on the other side. That is the boundary that also survives the next node
running on a different machine.
## Running a node somewhere else
A node that needs a GPU declares the label of the machine that has one:
```json
{"id": "train", "device": "gpu", "device_policy": "require", "timeout": 7200}
```
A worker on that machine dials **out** to the engine, because the engine
generally cannot reach it — different network, no inbound route — and because
nothing should expose Redis across hosts. Install it on the box, mint it a
token, and start it:
```bash
curl -X POST $FLUKSIO/workers/tokens -d '{"name": "gpu-dev"}' # once, as an admin
pip install fluksio-worker
fluksio-worker \
--url wss://api.example.com/api/v1/workers/attach \
--token "$FLUKSIO_WORKER_TOKEN" \
--labels gpu,cuda12 \
--python /opt/torch-venv/bin/python
```
`fluksio-worker` is its own distribution — the agent, the node runner, and
`websockets`. Nothing of the engine, so a GPU box does not install a database
driver to run a training step. Where pip is not an option, the two files still
work copied into one directory and run with `python agent.py …`; the engine
serves the runner at `GET /api/v1/workers/runtime`.
`--python` is the interpreter node code runs on, which is how the GPU box keeps
its CUDA wheels without the engine ever installing them. The node's source
travels with every call, so nothing has to be deployed there.
A few consequences worth knowing:
* `import fluksio` inside a node is the worker's own reporter — `emit`,
`save_artifact`, `load_artifact` — installed before the node's code runs, so
the installed `fluksio` package (if the box has one) never shadows it.
* A node bound to a device is **compiled on that machine**. A node importing
`torch` is correct on the GPU box and a missing module on the engine, so
checking it on the engine would fail a node that is fine.
* If nothing carrying the label is attached, the run stays `queued` and says
what it is waiting for. Submit first, switch the GPU box on later.
* Cancelling a run kills what it is executing, there or here, and leaves other
runs of the same node alone.
* If the worker disappears mid-call, the run fails in seconds with
`worker went away mid-call` rather than waiting out its timeout.
* `device_policy: "prefer"` runs locally when no such worker is attached;
`"require"` (the default) waits for one.
## Durability
Submitting journals the run to a Redis stream of its own, separate from the
one the automations use — a burst of five hundred sweep runs must not stand
between a house and its heating. An engine that is down when a run is
submitted picks it up when it starts.
From the moment a run is claimed, its database row is the record and the queue
is finished with it. Redelivering two hours of training because an
acknowledgement was late is not recovery; instead a running run refreshes a
lease, and one whose lease goes stale is marked `abandoned` — which is what a
run whose engine was killed mid-training becomes.
## What this costs, compared
The repository ships a benchmark that measures submitting a run against a
Kedro project doing the same nothing:
```
fluksio — submit accepted median 15.3 ms
submit -> result median 60.8 ms
kedro — kedro run median 1109.5 ms
```
The difference is not the orchestration; it is that Fluksio does not boot a
project per run. The engine is already up, and the workers already have the
node's code compiled. On a 510-run sweep, that gap is about nine minutes of
pure startup that never happens.
## See also
- [Getting started: data science](../getting-started/data-science.md) — the same
material as a first setup
- [Writing node code](../code/nodes.md) — generators, `fluksio.emit`, artifacts
- [Remote workers](../code/workers.md) — attaching the machine with the GPU
- [The HTTP API](../code/api.md) — every endpoint used above
docs/concepts/state.md
+75
View File
@@ -0,0 +1,75 @@
# Keeping state in a flow
Logic nodes are pure functions of their inputs. There is no `context`, no
`global`, no handle to a store — a node is called with the values of the
messages it declares and returns the values of the messages it provides.
That is deliberate: a node with hidden state cannot be run twice in parallel,
cannot be replayed, and cannot be moved to another worker. But plenty of real
automations need to remember something. This is how.
## State is a message the node both reads and writes
A running total, a debounce timer, a last-seen reading — each is a value that
survives between runs. Give it a message name, declare it as both an input and
an output, and it is state:
```python
def process(reading, total=0.0):
return {"total": total + reading}
```
with `reading` and `total` as inputs and `total` as the output. Each run reads
the total the previous run wrote.
Two things make this work rather than loop forever:
- **A node never depends on itself.** The graph is built from message names, but
a node providing a message it also requires is not placed downstream of
itself. Publishing `total` does not re-run the node that wrote it.
- **The value must start somewhere.** The first run has nothing to read. Declare
the message as a flow input with a starting value; otherwise the flow reports
`self_loop_needs_initial` at edit time and refuses to publish.
The port needs a default in the function signature (`total=0.0` above) so the
first call is well-formed even if the value is missing.
## Feeding a value back between two nodes
Sometimes the value comes from a different node — a controller reading back what
an estimator computed from its own last output. Written plainly that is a cycle,
and the validator rejects it, because a graph where A waits for B and B waits
for A can never start.
What is actually meant is that the fed-back value is *state*, not a trigger.
Say so, by marking the input non-triggering:
```json
{"name": "estimate", "port": "estimate", "dtype": "float", "trigger": false}
```
A non-triggering input:
- creates no dependency, so it cannot form a cycle;
- never makes the node wait — if the message has no value yet, the port is
simply left out of the call, and the function's default applies;
- is read fresh from state whenever the node does run, for whatever reason.
Use it for the back edge only. An input that should wake the node is an ordinary
input, and marking it non-triggering would mean the node never runs at all.
## What still holds engine-side state
Built-in nodes that are *about* time or change — rate limiting, filter-on-change,
delay, cron — keep what they need in the engine's own state backend, under keys
that never appear as messages. That is the engine's business, not a flow's: the
line is that node *code* you write never reaches for state, while node *types*
the engine ships may.
Storage and direct-I/O nodes are the other sanctioned exception, since talking
to a device or a database is stateful by nature.
## See also
- [Where a node's values come from](values.md) — ports, settings and flow inputs
- [Node types](../reference/node-types.md) — the built-ins that do keep engine-side state
docs/concepts/values.md
+67
View File
@@ -0,0 +1,67 @@
# Where a node's values come from
A node function is called with one argument per thing it needs, and every one
of them is declared somewhere you can see. There are two kinds.
## Ports: what the graph carries
A port binds to a message name. Whatever last published that message is the
value the node is called with, and the wiring follows from the name rather than
from a line somebody drew.
```python
def process(reading, setpoint):
return {"heat": reading < setpoint}
```
with `reading` and `setpoint` as inputs and `heat` as an output.
## Settings: constants of this node
A setting is a value that belongs to this node's code rather than to the graph:
how many retries, which unit, how long to wait. It is typed into the node's
**Settings** section, stored with the flow, and arrives as an argument like a
port:
```python
def process(reading, unit="C"):
return {"shown": reading if unit == "C" else reading * 1.8 + 32}
```
Because both arrive by name, a setting cannot share a name with a port — the
node reports it rather than picking one.
Settings are part of the flow document, so changing one is an edit that gets
published, not something that happens at runtime. **A value that should change
while the flow runs is a message, not a setting.**
There is no `global` and no `flow` context. A constant several nodes need is a
node that provides it: one place it is set, an ordinary message out, and every
consumer visibly downstream of it.
## Flow inputs: what arrives from outside
Some messages are not computed by any node in the flow — a dashboard control
writes them, the API publishes them, a batch run passes them in. Declare those
as the flow's **inputs**, with the value they start from:
```json
{"inputs": [{"spec": {"name": "setpoint", "dtype": "float"}, "initial": 21.0}]}
```
Without that, the node reading `setpoint` waits for something nothing provides,
and the canvas says so. With it, the flow starts from 21.0 and whatever writes
the message afterwards takes over.
The canvas draws each one as a labelled endpoint feeding the nodes that read
it, the same way it draws a dashboard tile or another flow — so a value never
appears from nowhere. They are edited in the flow's own panel.
A batch flow's inputs are also its run parameters, and its **outputs** name what
a run reports as its result; see [Runs](runs.md).
## See also
- [Flows, nodes and messages](flows.md) — how the graph is built from the names
- [Keeping state in a flow](state.md) — a value a node reads *and* writes
- [Writing node code](../code/nodes.md) — the practical side