Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs

Browse Source
This commit is contained in:
stroblme
2026-10-13 10:02:20 +02:00
parent a93f865e91
commit 80e595c02e
24 changed files with 2210 additions and 4310 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/code/nodes.md
+132 -218
View File
@@ -1,323 +1,237 @@
# Writing node code
A Function node is a Python file: no base class, no decorator, no framework
import unless you want one.
A Function node runs Python in a worker process. Write its source in the
canvas, or declare functions in your project with `@node` and
[`fluksio sync`](cli.md#fluksio-sync).
```python
def process(temperature, setpoint=21.0):
"""Ask for heat when the room is below the comfort point."""
return {"heat": temperature < setpoint}
```
Declare `temperature` as an input port, `heat` as a Boolean output port, and
`setpoint` as a setting. Now you've got a tiny thermostat.
## The rules
**One function called `process`.** If the file defines exactly one public
function under another name, that one is used instead. Two, and the node
refuses to load rather than guessing.
- **Define `process`.** Without it, the worker accepts exactly one public
function defined in the source. Multiple candidates produce an error.
- **Arguments arrive by name** from input ports and settings. A setting cannot
share a name with a port. See [Where values come from](../concepts/values.md).
- **Return a dictionary keyed by output port.** Undeclared keys and values of
the wrong type fail validation. With `@node(provides=Port("heat", "bool"))`,
the SDK also accepts a bare value and wraps it for that single port.
- **Use JSON-compatible values.** For files, use artifacts. For persistent
state, use [flow state](../concepts/state.md); worker globals are not a
reliable store.
**Arguments come from ports and settings, by name.** `temperature` above is an
input port; `setpoint` is a setting typed into the node's panel. Both arrive as
keyword arguments, which is why a setting may not share a name with a port.
See [Where a node's values come from](../concepts/values.md).
**The return value is a dict keyed by output ports.** Every value is checked
against the port's declared type before it is published. A key that is not a
declared port is an error, not a silent drop. Nothing leaves a node except
through a port it declared.
**Nothing else is importable from the engine.** Node code runs in a separate
process, on a separate interpreter, with none of Fluksio's own modules on its
path. What it can import is what the [Modules](../interface/operations.md)
screen installed, which can include [your own project](#your-own-code-as-a-package),
so a node need not be a self-contained file.
**A node is a pure function of its inputs.** No context object, no global
store, no handle to reach for. A running total or a debounce timer has a
specific shape; see [Keeping state in a flow](../concepts/state.md).
Inside a node, `import fluksio` exposes reporting and artifact helpers. It
does not expose the engine's internal modules. Other imports come from the
node's Python environment; see [Packages](#packages).
## Producing values over time
A node that produces values *during* its execution is a generator. Every
`yield` is a dict keyed by output port, published the instant it happens:
Yield dictionaries to publish outputs before the node finishes:
```python
import math
def process(lr, steps):
loss = 1.0
for _ in range(steps):
loss = train_one_step(lr)
yield {"loss": loss} # published now
for step in range(steps):
loss = math.exp(-lr * (step + 1))
yield {"loss": loss}
return {"final_loss": loss}
```
Whatever the generator `return`s at the end is the node's result, which is what
downstream nodes read. If you never `return`, the last thing you yield is the
result instead.
Mark the port so the flow says what it does:
Declare `loss` as a streaming float port:
```json
{"name": "loss", "dtype": "float", "stream": true}
```
Two consequences. In a [run](../concepts/runs.md), the whole series is kept as
that run's metrics, which is why there is no `log_metric()` anywhere in the
API. And **the node's timeout starts measuring silence rather than duration**:
each emission resets the deadline, so a node yielding every few seconds can run
for hours under a timeout of 300.
Each yield is type-checked and published immediately. The final return is
the node's result; without a return, the last yield is the result. A
[run](../concepts/runs.md) keeps numeric emissions as metric series and waits
for the result before running downstream nodes. A live flow can also trigger
downstream nodes from emissions.
### `fluksio.emit`
Where a `yield` cannot reach, because the value comes from inside somebody
else's callback and they call you rather than the other way round:
Use `emit` when a callback produces the value:
```python
import fluksio
def on_epoch_end(epoch, logs):
fluksio.emit(loss=logs["loss"])
```
Pass this callback to your training framework. It publishes through the same
declared ports as `yield`. Outside a node, `emit` does nothing, so ordinary
tests can still call your code.
## Bytes: artifacts
Save files or bytes with `save_artifact`, then return the reference through an
`artifact` port:
```python
import fluksio
def process():
model.fit(callbacks=[LambdaCallback(
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
return {"weights": ...}
```
Same ports, same type checking, same publication. Prefer `yield` where you can
reach it; `emit` where you cannot.
## Bytes: artifacts
Messages are JSON, which is what lets the same value pass through Redis, the
work queue and the worker protocol unchanged. A checkpoint is not that.
```python
import fluksio
def process(dataset):
path = fluksio.load_artifact(dataset) # → a local path to read
...
return {
"weights": fluksio.save_artifact("model.pt", media_type="application/octet-stream"),
"score": 0.94,
"report": fluksio.save_artifact(
b"Experiment complete\n", "report.txt", "text/plain"
)
}
```
`save_artifact` takes bytes or a path, stores them by their SHA-256 digest, and
returns a small reference (digest, size, media type, name) which is what an
`artifact`-typed port carries.
Another node can read it with `fluksio.load_artifact(report)`, which returns
a local file path. These two helpers require a running node; for tests,
keep the underlying calculation in ordinary Python functions.
Because the address is the content's hash, a sweep whose fifty configs share
one preprocessed input stores it once, and a reference stays valid wherever the
store is reachable from, including on another machine.
`fluksio artifacts RUN NAME` is how one comes back out at a terminal.
### `fluksio.logger`
A node's `print` is kept as that node's logs, and so is anything on
`fluksio.logger` is the same logger the SDK exports at top level, so code that
runs both inside a node and outside one says it the same way:
```python
fluksio.logger.info("resuming from epoch %d", start)
```
Artifacts are stored by their SHA-256 digest, so identical bytes share storage.
Use [`fluksio artifacts`](cli.md#fluksio-artifacts) to list and download a
run's files.
### Media
Say what the bytes are and the port can be typed for them:
`image`, `audio` and `video` ports carry artifact references with a matching
media type. For example, save JPEG bytes with `media_type="image/jpeg"` and
return the reference through an `image` port. See
[Payload types](../reference/payload-types.md#image-audio-video).
```python
def process(speech): # an `audio` port
clip = fluksio.load_artifact(speech)
words = transcribe(clip)
return {"transcript": words} # a `str` port
```
```python
def process(camera_url):
for index, jpeg in enumerate(grab(camera_url)): # a generator
frame = fluksio.save_artifact(
jpeg, f"frame-{index:05d}.jpg",
media_type="image/jpeg", volatile=True,
)
frame["meta"] = {"seq": index}
yield {"frame": frame} # an `image` stream port
```
An `image`, `audio` or `video` port is an artifact reference whose media type
has to match, so a node declaring `audio` never receives a video by accident.
See [Payload types](../reference/payload-types.md#image-audio-video) for what
each carries and what rates are realistic.
`volatile=True` is for a frame rather than a result. The bytes go to a ring in
memory instead of the data volume, and the engine pushes them down the
websocket to whichever screens are drawing that message — so a camera runs at
ten frames a second without writing anything to disk. They last as long as it
takes newer frames to need the room. Leave it off for a clip somebody asked to
keep.
!!! warning "Emitted media is not kept; returned media is"
Only what a node *returns* is recorded against its run. Frames yielded
along the way are replaced in state by the next one, and the artifact sweep
removes bytes nothing refers to any more, which is what stops a camera
filling the disk. If a particular frame matters, return it: a volatile one
is copied out of the ring when the run records it, so returning it is also
what makes it outlive the next few seconds.
For transient frames, `save_artifact(..., volatile=True)` uses an in-memory
ring and streams bytes to connected screens. New frames eventually replace
old ones. A volatile artifact returned as a run result is copied to persistent
storage; emitted frames alone are not recorded as run artifacts.
## Printing
`print` works and is captured. The first 16 KB per call is kept and shown in
the flow editor's log panel and on the run's per-node record; the rest is
dropped, so a node printing in a loop cannot fill anything up.
Use it to debug, not to record results. A number worth keeping is an output
port, not a line of text.
`print` and `fluksio.logger.info(...)` are captured in the node logs. The
worker retains the first 16,384 characters per call. Use logs to debug and
output ports for numbers you want to plot or compare.
## Errors
An exception fails that node's execution, not the flow. The message you see is
one line from the frame in *your* code, not a stack through the engine.
The node keeps its last error visible after it recovers, so a failure that
fired an alert at 03:00 still says what it was at 09:00. It can also be
acknowledged from the canvas.
Exceptions appear on the node, with a short message and a traceback. In a live
flow, one failed execution does not stop the whole flow. In a batch run, node
failure makes the run fail. The canvas keeps the last error visible after
recovery until you acknowledge it.
## Timeouts
`timeout` on a node is how many seconds its code may be *silent* before it is
stopped. A yield or an `emit` resets the clock, and the first call's imports are
not charged to it: a node importing torch is not being slow, it is loading.
`timeout` is the allowed number of seconds without an emission. A `yield` or
`emit` resets the deadline; initial compilation and imports are excluded.
There is no timeout by default. Training runs for hours and a node that reports
nothing is usually working, so the engine waits: what fails a call is the worker
dying, which arrives at once rather than after a deadline. Set a timeout on the
nodes where silence means stuck (an HTTP call that should answer in seconds, a
loop that can spin) either per node or, for all of them,
with `FLOW_NODE_TIMEOUT`. `timeout = 0` says explicitly that this node has none.
There is no timeout by default. Set one on a node where silence means stuck,
or use `FLOW_NODE_TIMEOUT` for the instance default. `timeout=0` explicitly
disables it for a node. A worker disappearing fails its active call without
waiting for that deadline.
## Running a node somewhere else
A node declares the label of the machine it needs:
Set a device label to select an attached worker:
```json
{"id": "train", "device": "gpu", "device_policy": "require", "timeout": 7200}
```
`require` (the default) waits for a worker carrying that label; `prefer` runs
locally when none is attached. A node bound to a device is compiled *on that
machine*: a node importing `torch` is correct on the GPU box and a missing
module on the engine, so checking it here would fail something that is fine.
See [Remote workers](workers.md).
`require` is the default and waits for a worker with that label. `prefer` uses
the local engine when none is attached. Code is compiled on the selected
machine, which needs the packages your node imports. See
[Remote workers](workers.md) for setup and resource requests.
## Sharing code between flows
A node's source can be promoted to the shared library from its panel, and other
flows can then use it by reference. There is one copy and one place to edit, and
every flow using it runs the edit.
Promote a node's source to the shared library from its panel. Other flows can
reference that source, and edits apply to all of them. Shared sources live in
`_lib/` in the instance's flow repository.
Shared sources live in `_lib/` in the flow repository, so they are versioned
with everything else.
For code-defined flows, reuse the same decorated function. `use(fn, ...)`
changes its wiring or settings for one flow:
```python
use(fit, wire={"dataset": "augmented"}, epochs=3)
```
## Packages
Node code runs in a virtual environment of its own, on the instance's data
volume, separate from the one Fluksio itself runs on.
The startup banner shows the interpreter used for local nodes:
Declare what you import in [Modules](../interface/operations.md), or over the
API:
| Environment | How to add packages |
|---|---|
| Fluksio installed in an existing venv | Use pip or uv in that venv, then `fluksio sync` to refresh workers. Modules is read-only. |
| Managed node venv | Edit the manifest in **Modules** and apply it. |
| Remote worker | Install packages in the interpreter selected by `--python`. |
With `NODE_VENV=auto`, Fluksio adopts its existing venv unless this instance
already has a managed one. Without an existing venv, it creates a managed
environment. `NODE_VENV=managed` always requests one; the Docker image uses
this setting.
In managed mode, the Modules manifest is installed with `uv pip sync` and
versioned with the flows. Applying it refreshes local workers. The API accepts
the same manifest:
```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
curl -X POST "$FLUKSIO/modules/apply" -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"requirements": "numpy>=2\npandas\n"}'
```
It is a pip manifest installed with `uv pip sync`, versioned alongside your
flows. An install takes effect immediately; nothing restarts.
Here `$FLUKSIO` is the API base URL ending in `/api/v1`, and `$TOKEN` is an
access token; see [The HTTP API](api.md).
### Your own code as a package
!!! tip "If Fluksio is installed in the venv you work in, skip this"
Node code then runs on that environment, so your project and everything it
imports are already importable; see
[Getting started: data science](../getting-started/data-science.md). What
follows is for a Fluksio with a venv of its own, which is what a container
always has.
A manifest line can name a directory, so the project you already have is
installable like any other dependency:
`fluksio sync` generates an import shim pointing at your local project. A
managed environment can also install your project from a Modules manifest:
```text
-e /home/you/my-research
numpy>=2
```
A node body then imports it, and the logic stays where it already lives: in
your repository, under your own version control, importing its own siblings:
Then a canvas node can import ordinary helpers from it:
```python
from myresearch.train import fit
from myresearch.training import train_model
def process(lr, epochs):
return fit(lr, epochs)
return {"score": train_model(lr, epochs)}
```
That is the whole of it. The node is three lines, `myresearch` can be as many
modules as it likes, and nothing was copied.
Editable installs expose changed files, but warm local workers still hold
imports in memory. Apply the Modules manifest or run `fluksio sync` after
edits. Attached workers start a process per call and read the files afresh.
!!! tip "You can have those three lines written for you"
Decorate `fit` with `@node(...)` where it is defined, say which nodes make
a flow with `Flow(...)`, and `fluksio sync` generates the body above,
along with the flow document, so there is nothing to PUT by hand. The
declaration lives beside the function it describes and is checked against
its signature. See
[Getting started: data science](../getting-started/data-science.md).
!!! warning "Editable, but not live"
`-e` means edits reach the venv without reinstalling, but a node's process
already holds the imported module in memory. The engine's workers are
long-lived, so a change to your code is picked up when they are retired,
which is what **Apply** does. Pressing it after an edit is the loop.
A [worker](workers.md) you attach yourself is the exception: it starts a
process per call, so it reads your code fresh every run. If you are
iterating on the code many times an hour, point one at your own
interpreter (`fluksio-worker --python "$(which python)"`) and mark the
node with its label.
!!! note "The path is a deployment detail"
It is resolved on whichever machine runs the node, and the manifest is
committed to the flow repository, so an absolute path from your laptop
means nothing inside a container or on a GPU box. Those need their own
install of the same project; a VCS requirement
(`myresearch @ git+ssh://…@a1b2c3d`) travels where a path does not.
Paths are resolved on the machine running the node. A remote worker or
container needs its own copy or installation of your project; syncing a shim
does not ship the package.
## A worked example
The repository ships a small supervised fit as a seedable demo: three nodes,
a batch flow, streaming metrics, artifacts between stages, and a GPU-labelled
node that falls back to the engine when no worker is attached. It is the
shortest complete thing to read:
[Getting started: data science](../getting-started/data-science.md) contains a
complete runnable flow. The public repository also includes
`examples/myresearch/`, with separate preparation, training and evaluation
modules. From the repository root:
```text
prepare ──dataset(artifact)──▶ train ──weights(artifact)──▶ evaluate
│
└── loss (streaming float) ──▶ chart
```sh
fluksio sync examples/myresearch
fluksio run train --lr 0.05 --wait
```
The example is built by a seed script the maintainers run against a live
stack; the flow above is what it produces.
Start an instance first, as shown in the getting started guide.
## See also
- [Where a node's values come from](../concepts/values.md)
- [Keeping state in a flow](../concepts/state.md)
- [Runs: pipelines that finish](../concepts/runs.md)
- [Node types](../reference/node-types.md) — the ones you do not have to write
- [Node types](../reference/node-types.md)