Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs

Browse Source
This commit is contained in:
stroblme
2026-10-13 10:02:20 +02:00
parent a93f865e91
commit 80e595c02e
24 changed files with 2210 additions and 4310 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/concepts/flows.md
+104 -157
View File
@@ -1,204 +1,151 @@
# Flows, nodes and messages
Three ideas hold the system up. Almost everything else follows from them.
A **flow** is a group of **nodes** that exchange **messages**. Nodes declare
what they consume (`requires`) and produce (`provides`); matching message names
connect them. No wires to draw.
## A flow is a graph you did not draw
## How nodes connect
A **flow** is a set of **nodes**. Each node declares the messages it needs
(`requires`) and the messages it produces (`provides`). The graph is whatever
those declarations imply:
For a Function node, declare input and output ports in its panel, then write
`process(...)`. Arguments use input port names; the returned dictionary uses
output port names:
```python
# node "read"
# Node "read": provides temperature
def process():
return {"temperature": read_sensor()}
return {"temperature": 18.5}
# node "decide"
# Node "decide": consumes temperature, provides heat
def process(temperature, setpoint=21.0):
return {"heat": temperature < setpoint}
```
`decide` is downstream of `read` because it needs `temperature` and `read`
produces it. Nobody drew a wire.
Bind both temperature ports to the same message and `decide` follows `read`.
Here `setpoint` uses a Python default; you can also add it as a node
[setting](values.md#settings-constants-of-this-node).
The consequences:
- **Fan-in is free.** Two nodes providing `temperature` are two producers of
one message. The consumer does not change.
- **A node runs when something it reads was published.** Not merely when
something upstream of it ran: a node that produced nothing this time (held
back by a rate limit, say) leaves what reads it on the value it already
has, and so does everything behind that.
- **A wire cannot be wrong.** There is no wire. There is a name that either
matches or does not, and the canvas tells you at edit time which it is.
- **Layout is not a document.** The canvas computes the arrangement, so a flow
has no stored positions to maintain, merge or fight over.
- **Flows stay small.** A graph nobody can hand-arrange is one worth keeping
small, which is the intent. Several atomic flows that name each other beat
one flow with sixty nodes in it.
- Several nodes can produce the same message. Consumers use the latest value.
- Publishing an input wakes its consumers once their required values are
available. An upstream node that publishes nothing leaves downstream nodes
waiting for a new publication.
- The editor arranges the graph automatically. Keep flows focused and share
messages between them when needed.
### Message names are namespaced
Inside flow `house`, a message named `temperature` is really
`house.temperature`. A bare name is qualified with its own flow; a dotted name
is used as written. That is how two flows share a value:
In flow `house`, `temperature` becomes `house.temperature`. A dotted name is
used as written, so another flow can consume it directly. Bind an input port
named `temperature` to `house.temperature`:
```python
# in flow `dashboard`, reading a message that flow `house` produces
def process(house_temperature): # port bound to "house.temperature"
...
def process(temperature):
return {"shown": temperature}
```
The canvas draws messages arriving from another flow as labelled endpoints, so
you can see where they come from without opening the other flow.
The canvas shows cross-flow messages as labelled endpoints.
## A node is a function with declared ports
## Node types and payloads
Most nodes are **Function** nodes: a Python file defining `process(...)`. Its
arguments are its input ports by name; its return value is a dict keyed by
output ports.
```python
def process(reading, unit="C"):
return {"shown": reading if unit == "C" else reading * 1.8 + 32}
```
`reading` is a port. `unit` is a **setting**: a constant of this node's code,
typed into its panel and stored with the flow. Both arrive as arguments, which
is why a setting may not share a name with a port. See
[Where a node's values come from](values.md).
The rest of the node types are the ones that would be tedious or unsafe to
write yourself: MQTT, HTTP, InfluxDB, schedules, switches, notifications. Each
one is configured by filling in a form the editor generates from its parameter
schema, so they all behave the same way. The full list is in
Function nodes run your Python code. Built-in nodes handle MQTT, HTTP,
InfluxDB, schedules, filtering and more through forms in their panels. See
[Node types](../reference/node-types.md).
### Ports are typed
Every port declares a payload type (`dtype`). Values are checked against it,
and dashboard widgets offer compatible messages. Scalars, structured JSON and
media references are covered in [Payload types](../reference/payload-types.md).
A port declares a `dtype`: `float`, `int`, `str`, `bool`, `json`, `record`,
`list`, `series`, `artifact`, or one of the media types `image`, `audio` and
`video`. Every value that passes through is checked against it.
Types are not decoration. They are what lets the dashboard editor offer you
only the messages a gauge can actually draw, and what lets the canvas refuse a
binding before anything runs. See [Payload types](../reference/payload-types.md).
Everything on the wire is JSON. Bytes (a checkpoint, an image, a model)
travel as an `artifact`: the bytes go to a content-addressed store and the
message carries a small reference to them. The media types are that same
reference, saying what kind of bytes are behind it.
Messages carry JSON. For binary data, store the bytes as an **artifact** and
publish its reference. `image`, `audio` and `video` are artifact references
restricted to their media family.
### Streaming ports
A port marked `stream` produces repeatedly *during* one execution rather than
once at the end: a training loss, a progress fraction, a frame from a camera, a
second of speech. A node publishes on one by being a generator and yielding, or
by calling `fluksio.emit`.
Mark an output **streaming** when it publishes during an execution: progress,
training loss or media chunks. A Function node can yield values or call
`fluksio.emit`.
Each value is delivered to the nodes reading it, in the order it was produced,
so a recogniser slower than the microphone in front of it still sees every
chunk rather than only the newest. What is in state remains the latest value,
which is what everything else reads, and what a run keeps is the whole series.
Downstream nodes receive each publication in order; message state holds the
latest value. A batch run records the full series. An output `interval` limits
live publication without thinning the run's recorded series.
An `interval` on a streaming port thins what reaches the canvas without
thinning the run's record of it.
### Keeping node code predictable
### Nodes are pure
Write Function nodes as functions of their declared inputs and settings.
Fluksio provides no hidden `context` or global flow store. To remember a value
between executions, use a message the node reads and writes; see
[Keeping state in a flow](state.md).
A node is called with the values of the messages it declares and returns the
values of the messages it provides. There is no context object, no global
store, no handle to reach for.
## Live and batch flows
That is deliberate: a node with hidden state cannot run twice in parallel,
cannot be replayed, and cannot be moved to another machine. Plenty of real
automations do need to remember something, and there is a specific way to say
so; see [Keeping state in a flow](state.md).
## Two shapes of flow
Set `mode` on the flow:
Choose **Mode** in the flow settings:
| | `live` (default) | `batch` |
|---|---|---|
| Runs | continuously | once per run, on request |
| Started by | subscriptions, schedules, webhooks | `POST /runs/flows/{name}` |
| Ends | never | when the graph drains |
| Keeps | the last value of each message | a run record: params, result, metrics, artifacts |
| Is | a thermostat, an ETL job on a cron | an experiment, a CI-style job |
| Runs | reacts continuously | once per requested run |
| Started by | subscriptions, schedules, webhooks or manual execution | Run in the editor, SDK or API |
| Finishes | stays active until stopped | when its work finishes |
| Keeps | latest message values | parameters, results, metrics and artifacts |
| Example | thermostat | experiment or build job |
A batch flow is built and validated like any other, appears on the same canvas
and is type-checked the same way. It is simply never *activated*: no
subscriptions, no schedules, no webhooks. See
[Runs: pipelines that finish](runs.md).
Batch flows use the same editor and validation, but do not activate background
subscriptions, schedules or webhooks. See [Runs](runs.md).
## Editing is separate from running
## Flow inputs
Every flow has a published version and, while you are working, a draft.
- **Saving** writes the draft. The engine keeps running the published version.
- **Publishing** promotes the draft. The engine reloads and picks it up.
- **Discarding** throws the draft away.
The store is a git repository (`flow.json` for the structure, `nodes/*.py` for
the code) and each save is a commit. So a flow's history is readable with
ordinary git tooling, and copying a flow between instances is copying a
directory.
Saving carries the version you last saw. If someone else saved in between, you
get a 409 instead of quietly overwriting their work.
## What can be wrong, and when you find out
The canvas validates continuously and names problems on the nodes they belong
to:
| Issue | What it means |
|---|---|
| `unconnected_input` | a port needs a message nothing in reach provides |
| `missing_initial_value` | the message exists but has never held a value, and nothing will give it one. Not reported on a batch flow: its inputs arrive with the run |
| `cycle` | A waits for B and B waits for A — nothing could ever start |
| `self_loop_needs_initial` | a node reads a message it also writes, with no starting value |
| `node_error` | the node's code did not load: a syntax error, a missing import |
| `unauthenticated_hook` | advisory — a webhook with no shared secret is open to anyone |
| `node_unhealthy` | the node loaded but is not working: a connector that cannot reach its device, or whose last publication failed |
A flow with any of these except the advisory one and `node_unhealthy` does not
run. A node reporting itself down is a live condition, not a build error, so
the rest of the flow keeps going and the issue clears by itself once the node
reports well again. The health summary on Home counts them, so "why is nothing
happening?" has an answer that does not involve reading logs.
## What happens at runtime
- **A flow can be started and stopped.** Stopped means its subscriptions and
schedules are torn down.
- **A flow can be paused and stepped.** Paused holds messages instead of
running them; step releases exactly one. This is how you test something
before it moves a relay.
- **A failing node does not take the flow down.** It reports an error, keeps
its last error visible after it recovers, and can fire an alert.
- **A flow whose background tasks keep crashing is quarantined.** The engine
stops restarting them and says so, rather than spinning. Publishing a change
gives it another chance.
## Values that arrive from outside
Some messages are not computed by any node: a dashboard control writes them,
the API publishes them, a batch run passes them in. Declare those as the flow's
**inputs**, with the value they start from:
Declare messages supplied by dashboard controls, the API or run parameters as
the flow's **Inputs**. Give live inputs a starting value:
```json
{"inputs": [{"spec": {"name": "setpoint", "dtype": "float"}, "initial": 21.0}]}
```
Without that, the node reading `setpoint` waits for something nothing provides,
and the canvas says so. With it, the flow starts at 21.0 and whatever writes
the message afterwards takes over.
This seeds `house.setpoint` in flow `house` if it has no value yet. Later
publications replace it. Batch inputs are run parameters, and batch outputs
select the run's result. See [Where a node's values come from](values.md).
## Drafts and publishing
- **Saving** stores a draft; the engine keeps running the published version.
- **Publishing** promotes the draft and reloads the flow.
- **Discarding** removes the draft.
Shared node code is an exception: saving it updates every flow using it
immediately. See [The flow editor](../interface/flow-editor.md#code-and-reuse).
Flows are stored in a git repository: `flow.json` describes the graph and
`nodes/*.py` holds code. Saves are versioned, and stale versions return a
conflict instead of overwriting another editor's changes.
Python installations need Git on the host to record history. Without it,
flows still save as files; Docker includes Git.
## When a flow needs attention
The canvas reports issues on the affected nodes:
| Issue | Meaning |
|---|---|
| `unconnected_input` | no node or declared flow input supplies the message |
| `missing_initial_value` | a live input has no starting value |
| `cycle` | nodes depend on each other in a loop |
| `self_loop_needs_initial` | a node reads and writes a message with no starting value |
| `missing_source` | a Function node has no saved code |
| `node_error` | code or configuration could not load |
| `unauthenticated_hook` | a webhook has no shared secret; advisory |
| `node_unhealthy` | a loaded node reports a runtime problem, such as a lost connection |
Build issues prevent the affected flow from running. The last two are runtime
or advisory issues: other work continues, and node health clears when the
connection recovers.
You can **stop** a live flow to remove its subscriptions and schedules, or
**pause** it to hold pending work while values still arrive. **Step** releases
one held item. Repeated background crashes can quarantine a flow; publishing
a change gives it another chance.
## Where to next
- [Where a node's values come from](values.md) — ports, settings, flow inputs
- [Keeping state in a flow](state.md) — the sanctioned way to remember something
- [Runs: pipelines that finish](runs.md) — the batch half of the engine
- [Writing node code](../code/nodes.md) — the practical side of `process()`
- [Where a node's values come from](values.md) — ports, settings and flow inputs
- [Keeping state in a flow](state.md) — remembering a value
- [Runs](runs.md) — batch flows
- [Writing node code](../code/nodes.md) — practical Python examples