Make the docs state things rather than argue them
Docs / docs (push) Successful in 37s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m35s
Playwright Tests / test-playwright (2, 2) (push) Failing after 17s
pre-commit / pre-commit (push) Failing after 2m8s
Test Backend / test-backend (push) Failing after 2m48s
Compose Smoke Test / test-compose (push) Failing after 13s
Playwright Tests / merge-reports (push) Failing after 2m25s

The site read as a design journal: rationale paragraphs, hedges
("deliberately", "on purpose", "genuinely"), meta-commentary about the docs
themselves, and one em-dash every ten lines carrying an aside.

Roughly twenty rationale blocks are gone or reduced to what a reader needs
in order to use the thing. Em-dashes go from 507 to 135, and what is left is
structural rather than prose: list and definition separators, table cells,
and four inside code blocks that quote what the CLI actually prints.

Also: api.example.com becomes api.fluksio.com (the emails stay, since
bootstrap.py really defaults to admin@example.com and RFC 2606 reserves it);
the mqtt table gains the two settings it had drifted behind on and inject's
wording matches the engine; llms.txt lists the two connector pages that were
in the nav but not in it; and the two device/device_policy notes now agree.

Builds clean under `zensical build --strict`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YrQnKV3bnQd4K342y8tKj
This commit is contained in:
2026-08-31 10:49:58 +02:00
co-authored by Claude Opus 5
parent 2422a9b22b
commit bdad6d7fc2
25 changed files with 450 additions and 479 deletions
+12 -14
View File
@@ -1,7 +1,6 @@
# Flows, nodes and messages
Three ideas hold the whole system up. They are worth twenty minutes, because
almost everything else follows from them.
Three ideas hold the system up. Almost everything else follows from them.
## A flow is a graph you did not draw
@@ -22,21 +21,20 @@ def process(temperature, setpoint=21.0):
`decide` is downstream of `read` because it needs `temperature` and `read`
produces it. Nobody drew a wire.
This is the one structural decision everything else rests on, so it is worth
being explicit about the consequences:
The consequences:
- **Fan-in is free.** Two nodes providing `temperature` are two producers of
one message. The consumer does not change.
- **A node runs when something it reads was published.** Not merely when
something upstream of it ran: a node that produced nothing this time held
back by a rate limit, say leaves what reads it on the value it already
something upstream of it ran: a node that produced nothing this time (held
back by a rate limit, say) leaves what reads it on the value it already
has, and so does everything behind that.
- **A wire cannot be wrong.** There is no wire. There is a name that either
matches or does not, and the canvas tells you at edit time which it is.
- **Layout is not a document.** The canvas computes the arrangement, so a flow
has no stored positions to maintain, merge or fight over.
- **Flows stay small.** A graph nobody can hand-arrange is one worth keeping
small which is the intent. Several atomic flows that name each other beat
small, which is the intent. Several atomic flows that name each other beat
one flow with sixty nodes in it.
### Message names are namespaced
@@ -65,7 +63,7 @@ def process(reading, unit="C"):
return {"shown": reading if unit == "C" else reading * 1.8 + 32}
```
`reading` is a port. `unit` is a **setting** a constant of this node's code,
`reading` is a port. `unit` is a **setting**: a constant of this node's code,
typed into its panel and stored with the flow. Both arrive as arguments, which
is why a setting may not share a name with a port. See
[Where a node's values come from](values.md).
@@ -86,7 +84,7 @@ Types are not decoration. They are what lets the dashboard editor offer you
only the messages a gauge can actually draw, and what lets the canvas refuse a
binding before anything runs. See [Payload types](../reference/payload-types.md).
Everything on the wire is JSON. Bytes a checkpoint, an image, a model
Everything on the wire is JSON. Bytes (a checkpoint, an image, a model)
travel as an `artifact`: the bytes go to a content-addressed store and the
message carries a small reference to them. The media types are that same
reference, saying what kind of bytes are behind it.
@@ -98,7 +96,7 @@ once at the end: a training loss, a progress fraction, a frame from a camera, a
second of speech. A node publishes on one by being a generator and yielding, or
by calling `fluksio.emit`.
Each value is delivered to the nodes reading it, in the order it was produced
Each value is delivered to the nodes reading it, in the order it was produced,
so a recogniser slower than the microphone in front of it still sees every
chunk rather than only the newest. What is in state remains the latest value,
which is what everything else reads, and what a run keeps is the whole series.
@@ -115,7 +113,7 @@ store, no handle to reach for.
That is deliberate: a node with hidden state cannot run twice in parallel,
cannot be replayed, and cannot be moved to another machine. Plenty of real
automations do need to remember something, and there is a specific way to say
so see [Keeping state in a flow](state.md).
so; see [Keeping state in a flow](state.md).
## Two shapes of flow
@@ -142,8 +140,8 @@ Every flow has a published version and, while you are working, a draft.
- **Publishing** promotes the draft. The engine reloads and picks it up.
- **Discarding** throws the draft away.
The store is a git repository `flow.json` for the structure, `nodes/*.py` for
the code and each save is a commit. So a flow's history is readable with
The store is a git repository (`flow.json` for the structure, `nodes/*.py` for
the code) and each save is a commit. So a flow's history is readable with
ordinary git tooling, and copying a flow between instances is copying a
directory.
@@ -166,7 +164,7 @@ to:
| `node_unhealthy` | the node loaded but is not working: a connector that cannot reach its device, or whose last publication failed |
A flow with any of these except the advisory one and `node_unhealthy` does not
run — a node reporting itself down is a live condition, not a build error, so
run. A node reporting itself down is a live condition, not a build error, so
the rest of the flow keeps going and the issue clears by itself once the node
reports well again. The health summary on Home counts them, so "why is nothing
happening?" has an answer that does not involve reading logs.
+49 -52
View File
@@ -1,6 +1,6 @@
# Runs: pipelines that finish
A flow that automates a house never ends a value arrives, nodes fire, and it
A flow that automates a house never ends: a value arrives, nodes fire, and it
waits for the next one. A research pipeline is the other shape: parameters go
in, stages execute in order, and at some point it is *done* and has produced
something worth keeping. Fluksio calls the second one a **run**, and it is the
@@ -8,7 +8,7 @@ same engine either way.
This is what makes Fluksio usable where Kedro, MLflow or ClearML would be: a
run has parameters that identify it, a result, per-step metrics, artifacts and
a place in a queryable history without a second server, and without paying a
a place in a queryable history, with no second server and no
project bootstrap on every execution.
## A batch flow
@@ -28,15 +28,15 @@ Set `mode: "batch"` on the flow and name the messages its result should hold:
}
```
A batch flow is built and validated like any other it appears on the canvas,
its ports are type-checked but it is never *activated*: no subscriptions, no
A batch flow is built and validated like any other (it appears on the canvas,
its ports are type-checked) but it is never *activated*: no subscriptions, no
schedules, no webhooks. It runs when a run asks it to, and not otherwise.
Its `inputs` are its parameters. A run supplies values for them; anything it
does not supply keeps the declared initial value.
One thing a batch flow may not do is rate-limit a port (`interval`). A rate
limit holds a value back for a timer to release, and a run has no timer the
limit holds a value back for a timer to release, and a run has no timer, so the
value would be dropped rather than delayed, so submitting is refused instead.
## Submitting
@@ -51,7 +51,7 @@ The answer is immediate and the run is `queued`; a training run is measured in
hours, so nothing waits for it. Poll `GET /api/v1/runs/{id}` for its status,
result, per-node record and artifacts.
Wrong parameters are refused before anything executes an undeclared name, or
Wrong parameters are refused before anything executes: an undeclared name, or
a value of the wrong type, comes back as a 422 naming the problem.
### Sweeps
@@ -98,20 +98,17 @@ Mark the port it streams on, so the flow says what it does:
{"name": "loss", "dtype": "float", "stream": true}
```
Every yield is published the instant it happens same port, same type check,
Every yield is published the instant it happens: same port, same type check,
same place on the canvas as any other value. Whatever the generator `return`s
is the node's result, and is what downstream nodes read. If you never
`return`, the last thing you yield is the result instead.
This is the whole reason the framework does not have a logging API. A metric
that escapes through `log_metric()` is undeclared: invisible to validation,
absent from the canvas, and stored somewhere the graph knows nothing about.
A metric that leaves through a port is a message — so a chart binds to it
directly, a downstream node can consume it, and the run keeps its series
without anyone asking.
There is no `log_metric()`. A metric leaves through a port, which makes it a
message: a chart binds to it directly, a downstream node can consume it, and the
run keeps its series.
**Where a yield cannot reach** — the value comes from inside somebody else's
callback, and they call you rather than the other way round `fluksio.emit`
callback, and they call you rather than the other way round. `fluksio.emit`
writes the same ports the same way:
```python
@@ -129,8 +126,8 @@ def process():
Every number a node emits is kept as the run's series, stepped by the count of
emissions on that message. Read one back with
`GET /api/v1/runs/{id}/metrics?name=<flow>.loss` or leave `name` off for
every series the run kept or compare runs:
`GET /api/v1/runs/{id}/metrics?name=<flow>.loss`, or leave `name` off for
every series the run kept, or compare runs:
```
GET /api/v1/runs/series/compare?ids=<a>,<b>,<c>&metric=<flow>.loss
@@ -141,15 +138,15 @@ training curves side by side is a widget binding. During a run the values also
arrive live on the flow socket, so a chart bound to the port fills in as the
training goes.
A streaming port may set `interval` to thin out what reaches the canvas — the
A streaming port may set `interval` to thin out what reaches the canvas. The
run's history still keeps every value, because the interval is asking for the
display not to be flooded, not for the curve to have holes in it.
Emitting has a second effect: **a node's timeout measures silence, not
duration**. A node that yields every few seconds can run for hours under a
`timeout` of 300; one that says nothing for longer than its timeout is killed.
There is no timeout unless one is set a training node that reports nothing is
usually working — so set `timeout` where going quiet means stuck, at how long
There is no timeout unless one is set, since a training node that reports
nothing is usually working. Set `timeout` where going quiet means stuck, at how long
the node may plausibly be.
In a *live* flow, an emission also wakes whatever is downstream of it, exactly
@@ -160,8 +157,8 @@ finished" with nothing to mean.
## Artifacts
Bytes never travel as a message. `save_artifact` writes them to a
content-addressed store and returns a small reference digest, size, media
type, name which is what an `artifact`-typed port carries:
content-addressed store and returns a small reference (digest, size, media
type, name) which is what an `artifact`-typed port carries:
```python
def process(weights): # requires: weights, dtype "artifact"
@@ -177,8 +174,8 @@ downloadable at `GET /api/v1/artifacts/{digest}`.
## Stage caching
A run mostly does not redo what an earlier one already did. Before a node
executes it is fingerprinted a sha256 over its source, its settings, the ports
it declares and the values it is about to read — and if some earlier run of that
executes it is fingerprinted with a sha256 over its source, its settings, the
ports it declares and the values it is about to read. If some earlier run of that
same fingerprint finished, what that one returned is restored into this run's
state and the node is skipped. It is recorded with the status `cached` and a
duration of zero, and its artifacts are listed on the new run as well, so they
@@ -192,14 +189,14 @@ on a run carries the `cache_key` it was looked up by.
For a [code-defined flow](../getting-started/data-science.md), "its source" is
the generated shim, which imports the real function and does not change when
that function does. So the key carries one thing more: a digest of the project
modules that node's function reaches, worked out by `fluksio sync` which is
the only side that imports your code and can see what it imports and read
modules that node's function reaches, worked out by `fluksio sync`, which is
the only side that imports your code and can see what it imports, and read
again from those files when the run starts.
Editing a helper three calls down from the node invalidates it, which is the
point: the alternative is a re-run answering with the previous code's numbers.
Editing something the node does not reach leaves the hit standing, which is
the other half a notebook two directories away is not a reason to retrain.
the other half: a notebook two directories away is not a reason to retrain.
The walk follows imports statically and stops at the standard library, at
anything installed, and at Fluksio itself; a module imported under a name the
code computes is not followed. An engine that cannot see the files keeps what
@@ -207,20 +204,20 @@ sync recorded instead of nothing, so a worker on another machine no longer
keys every run the same.
The run history *is* the cache; there is no second store. A node's returned
outputs are kept on its run record as canonical JSON, up to 256000 characters
a node returning more than that is simply not cacheable that run. An entry
outputs are kept on its run record as canonical JSON, up to 256000 characters.
A node returning more than that is not cacheable that run. An entry
whose artifact bytes have since left the store is a miss, not an error.
Only `python` nodes are cached, and by default all of them are. A built-in node
type or a connector node has side effects and no source to fingerprint, so
neither is ever a candidate. Turn it off for one node with
`@node(..., cache=False)` the flow document carries it as `cache`, so the
canvas and the API can change it too or for one run with
`@node(..., cache=False)` (the flow document carries it as `cache`, so the
canvas and the API can change it too) or for one run with
`fluksio run --no-cache`, `fluksio sweep --no-cache`, or `"no_cache": true` in
the submission body.
A cached node replays no emissions those values were the story of an
execution that is not happening this time so its series is not rewritten
A cached node replays no emissions, since those values were the story of an
execution that is not happening this time, so its series is not rewritten
either. The run it was restored from is recorded instead, and that is where the
curve is read back from: asking the reusing run for its metrics answers with
the same points, under its own flow's names. The one way to be left with a
@@ -229,7 +226,7 @@ deleting its flow does.
## Objects that cannot be serialized
A live model, a `DataLoader`, a JAX-compiled function — these do not cross a
A live model, a `DataLoader`, a JAX-compiled function: none of these cross a
node boundary, and no framework flag will make them. There are exactly two
patterns, and they are both deliberate:
@@ -249,7 +246,7 @@ A node that needs a GPU declares the label of the machine that has one:
```
A worker on that machine dials **out** to the engine, because the engine
generally cannot reach it different network, no inbound route and because
generally cannot reach it (different network, no inbound route) and because
nothing should expose Redis across hosts. Install it on the box, mint it a
token, and start it:
@@ -259,13 +256,13 @@ curl -X POST $FLUKSIO/workers/tokens -d '{"name": "gpu-dev"}' # once, as an ad
pip install fluksio-worker
fluksio-worker \
--url wss://api.example.com/api/v1/workers/attach \
--url wss://api.fluksio.com/api/v1/workers/attach \
--token "$FLUKSIO_WORKER_TOKEN" \
--labels gpu,cuda12 \
--python /opt/torch-venv/bin/python
```
`fluksio-worker` is its own distribution the agent, the node runner, and
`fluksio-worker` is its own distribution: the agent, the node runner, and
`websockets`. Nothing of the engine, so a GPU box does not install a database
driver to run a training step. Where pip is not an option, the two files still
work copied into one directory and run with `python agent.py …`; the engine
@@ -278,7 +275,7 @@ travels with every call, so nothing has to be deployed there.
A few consequences worth knowing:
* `import fluksio` inside a node is the worker's own reporter — `emit`,
`save_artifact`, `load_artifact` installed before the node's code runs, so
`save_artifact`, `load_artifact`, installed before the node's code runs, so
the installed `fluksio` package (if the box has one) never shadows it.
* A node bound to a device is **compiled on that machine**. A node importing
`torch` is correct on the GPU box and a missing module on the engine, so
@@ -295,32 +292,32 @@ A few consequences worth knowing:
## Durability
Submitting journals the run to a Redis stream of its own, separate from the
one the automations use a burst of five hundred sweep runs must not stand
one the automations use: a burst of five hundred sweep runs must not stand
between a house and its heating. An engine that is down when a run is
submitted picks it up when it starts.
From the moment a run is claimed, its database row is the record and the queue
is finished with it. Redelivering two hours of training because an
acknowledgement was late is not recovery; instead a running run refreshes a
lease, and one whose lease goes stale is marked `abandoned` which is what a
lease, and one whose lease goes stale is marked `abandoned`, which is what a
run whose engine was killed mid-training becomes.
## Looking at what ran
The **Runs** screen is the experiment log: every run newest-first, filtered by
flow, by status, or down to one sweep. A sweep is worth filtering to the
flow, by status, or down to one sweep. A sweep is worth filtering to, since the
table then draws a column per parameter that actually varied, which is what
makes fifty runs of one flow readable.
Tick two or more and their curves go side by side — shift-click to take a
Tick two or more and their curves go side by side. Shift-click to take a
range, or the header box to take everything on screen. That comparison is the
address, so a link to it is a link someone else can open.
The curves are drawn against the step by default. They can also be drawn
against elapsed seconds, which answers "which one got there sooner" and is
measured from each run's own first reading so that runs started hours apart
still lie on top of each other; or against another metric of the same runs
an epoch, or samples seen joined on the step the two share.
still lie on top of each other; or against another metric of the same runs (an
epoch, or samples seen) joined on the step the two share.
One run in full is params, the per-node record with its logs and traceback,
the artifacts it made, its metrics and its result.
@@ -328,8 +325,8 @@ the artifacts it made, its metrics and its result.
### Taking it into a dataframe
An analysis wants a table rather than a screen, and there are two it usually
wants. `fluksio export metrics` is the long one a row per run, metric and
step — and `fluksio export runs` is the wide one, a row per run with the
wants. `fluksio export metrics` is the long one, a row per run, metric and
step; `fluksio export runs` is the wide one, a row per run with the
inputs that varied as columns beside its final numbers:
```python
@@ -343,20 +340,20 @@ arms = pd.DataFrame(client.export_runs(flow="train", status="ok"))
The run id is on every row of both, so a curve joins to the arm it came from
and to the run page it was recorded on, and the wide table carries the commit
and the code digest an exported file says what produced its numbers.
and the code digest, so an exported file says what produced its numbers.
[The CLI](../code/cli.md#fluksio-export) writes the same rows as csv, jsonl or
parquet, which is where an export belongs: in the script beside the analysis.
A node's numbers usually arrive inside a record, and the wide table follows
them in: every number a result carries is a column of its own, named by its
path `metric.final_metrics.train_loss` and `metrics="final_metrics.train_loss"`
path (`metric.final_metrics.train_loss`) and `metrics="final_metrics.train_loss"`
selects one. Inputs work the same way, and are compared leaf by leaf, so two
configurations differing in one field give that field as the axis.
### A dashboard, read against runs
A run records values under the same names a dashboard binds to a run of
`study` writes `study.loss` so a dashboard is already a way of looking at
A run records values under the same names a dashboard binds to (a run of
`study` writes `study.loss`) so a dashboard is already a way of looking at
one. **Open in dashboard** from a run or a comparison opens any dashboard with
`?runs=a,b,c` on it, and the widgets resolve from those runs instead of from
the live engine: each chart draws a line per run, the tiles that show one
@@ -367,7 +364,7 @@ Nothing about the dashboard is specific to runs. The page built to watch a
training run happen is the page that shows the finished ones.
If a flow has no dashboard yet, the same menu offers to build one from what
the flow declares a chart per streaming port, a tile per output:
the flow declares: a chart per streaming port, a tile per output:
```
POST /api/v1/dashboards/from-flow/study
@@ -386,8 +383,8 @@ sweep, or specific runs. It re-reads on its own and whenever a run finishes.
### When a run draws nothing
A node restored from the [stage cache](#stage-caching) has its curve read back
from the run that recorded it. Delete that run deleting its flow does — and
the reusing run is left with a result and an empty curve, and the chart says so
from the run that recorded it. Delete that run, which deleting its flow does,
and the reusing run is left with a result and an empty curve, and the chart says so
rather than looking broken.
## What this costs, compared
+6 -6
View File
@@ -1,7 +1,7 @@
# Keeping state in a flow
Logic nodes are pure functions of their inputs. There is no `context`, no
`global`, no handle to a store — a node is called with the values of the
`global`, no handle to a store. A node is called with the values of the
messages it declares and returns the values of the messages it provides.
That is deliberate: a node with hidden state cannot be run twice in parallel,
@@ -10,7 +10,7 @@ automations need to remember something. This is how.
## State is a message the node both reads and writes
A running total, a debounce timer, a last-seen reading each is a value that
A running total, a debounce timer, a last-seen reading: each is a value that
survives between runs. Give it a message name, declare it as both an input and
an output, and it is state:
@@ -36,7 +36,7 @@ first call is well-formed even if the value is missing.
## Feeding a value back between two nodes
Sometimes the value comes from a different node a controller reading back what
Sometimes the value comes from a different node, such as a controller reading back what
an estimator computed from its own last output. Written plainly that is a cycle,
and the validator rejects it, because a graph where A waits for B and B waits
for A can never start.
@@ -51,7 +51,7 @@ Say so, by marking the input non-triggering:
A non-triggering input:
- creates no dependency, so it cannot form a cycle;
- never makes the node wait — if the message has no value yet, the port is
- never makes the node wait. If the message has no value yet, the port is
simply left out of the call, and the function's default applies;
- is read fresh from state whenever the node does run, for whatever reason.
@@ -60,8 +60,8 @@ input, and marking it non-triggering would mean the node never runs at all.
## What still holds engine-side state
Built-in nodes that are *about* time or change rate limiting, filter-on-change,
delay, cron keep what they need in the engine's own state backend, under keys
Built-in nodes that are *about* time or change (rate limiting, filter-on-change,
delay, cron) keep what they need in the engine's own state backend, under keys
that never appear as messages. That is the engine's business, not a flow's: the
line is that node *code* you write never reaches for state, while node *types*
the engine ships may.
+3 -3
View File
@@ -28,7 +28,7 @@ def process(reading, unit="C"):
return {"shown": reading if unit == "C" else reading * 1.8 + 32}
```
Because both arrive by name, a setting cannot share a name with a port — the
Because both arrive by name, a setting cannot share a name with a port. The
node reports it rather than picking one.
Settings are part of the flow document, so changing one is an edit that gets
@@ -41,7 +41,7 @@ consumer visibly downstream of it.
## Flow inputs: what arrives from outside
Some messages are not computed by any node in the flow a dashboard control
Some messages are not computed by any node in the flow, such as a dashboard control
writes them, the API publishes them, a batch run passes them in. Declare those
as the flow's **inputs**, with the value they start from:
@@ -54,7 +54,7 @@ and the canvas says so. With it, the flow starts from 21.0 and whatever writes
the message afterwards takes over.
The canvas draws each one as a labelled endpoint feeding the nodes that read
it, the same way it draws a dashboard tile or another flow so a value never
it, the same way it draws a dashboard tile or another flow, so a value never
appears from nowhere. They are edited in the flow's own panel.
A batch flow's inputs are also its run parameters, and its **outputs** name what