Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs

Browse Source
This commit is contained in:
stroblme
2026-10-13 10:02:20 +02:00
parent a93f865e91
commit 80e595c02e
24 changed files with 2210 additions and 4310 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/getting-started/data-science.md
+115 -719
View File
@@ -1,800 +1,196 @@
# Getting started: data science
You have a training script. It works. What you do not have is any reliable
answer to "what was the learning rate on the run that got 94%?", and the
`results_final_v3_ACTUAL.csv` in your home directory is not helping.
Turn a Python experiment into a flow with recorded parameters, training curves
and files you can find again. Goodbye, `results_final_v3.csv`.
This page adds Fluksio to what you already have. It takes about five minutes,
installs one Python package, and does not ask you to restructure anything.
## Install
This example uses a toy training curve, so you can try the whole workflow
without a dataset or GPU. You need Python 3.12 or newer on Linux or macOS.
Install Git too if you want flow changes recorded in version history.
## 1. Start your instance
Create a project directory and install Fluksio in a virtual environment:
```sh
cd my-research
mkdir research-flow
cd research-flow
python3 -m venv .venv
. .venv/bin/activate
pip install fluksio
fluksio serve
```
That is the whole instance. No Docker, no database server, no ports to
open, and no login. The first run prints something like:
Fluksio creates `.fluksio/` for this project's flows, run history and artifacts.
It prints the initial admin password once and saves a token for local CLI
commands. Keep the password somewhere safe.
```text
Created the admin account admin@example.com
password: example-password
Shown once. Change it from the dashboard.
Fluksio 0.1.0 — data in /home/you/my-research/.fluksio
API http://127.0.0.1:8000/api/v1
Nodes /home/you/my-research/.venv/bin/python
your environment, adopted. Add packages with pip.
No portal. Pair this instance with:
fluksio enroll <code>
Signed in as admin@example.com
token in /home/you/my-research/.fluksio/client.json
```
Read the last two lines: you are already signed in. Signing in to your own
machine is a formality, since the password was printed by the same process that
would have checked it, so `serve` writes the token itself and every command
below just works. `fluksio login` is for an engine somewhere *else*.
**Write that password down** anyway. It is shown once, and it is what the
dashboard asks for.
### One instance per project
`.fluksio` sits beside your code, and is found the way `.git` is: from the
directory you are standing in, or any directory above it. So two repositories
on one machine are two engines: separate flows, separate run history,
separate token, and no chance of one experiment's graph turning up in the
other's.
```text
~/research/protein-fold/
.venv/ torch, fluksio
.fluksio/ its own database, flows, artifacts, token
myresearch/
~/research/climate-sim/
.venv/ jax, fluksio
.fluksio/ its own everything
climate/
```
It holds a database and a credential, so it ignores itself from within with a
`.gitignore` of `*`, the same thing `uv` writes into `.venv`. Nothing to add
to your project's own.
Give them different ports (`--port`) if you want two running at once.
!!! tip "One engine for the machine instead"
`fluksio serve --global` uses `~/.fluksio`, shared by every directory,
which is what you want for a personal server rather than a project. When
both exist, the banner says which one you are looking at and how to reach
the other. `--data-dir` still names any directory outright, which is worth
doing on a cluster where `$HOME` is a network filesystem: SQLite's
write-ahead log does not work on NFS, and `fluksio serve` warns you when
it notices.
!!! tip "Keep it running"
The engine is meant to be resident, which is what makes submitting a run
cost about 15 ms. Leave it in a `tmux` window, or write a small
`systemd --user` unit for it.
## Talking to it over HTTP
Everything the commands do is the HTTP API, and some of this page shows it
directly. For that, take the token `serve` already wrote:
At a terminal, `serve` opens a dashboard. Leave it running, then open a second
terminal in `research-flow` and activate the same environment:
```sh
export FLUKSIO=http://127.0.0.1:8000/api/v1
export TOKEN=$(jq -r .token .fluksio/client.json)
. .venv/bin/activate
```
While you are experimenting, the interactive schema at
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
For an engine on another machine, `fluksio login --url https://…` asks for a
password and stores the token the same way: in this project's `.fluksio`, or
with `--global` in `~/.fluksio`.
## Your packages are already there
If you installed Fluksio into the environment you work in (the venv that
already has torch or numpy in it) that is the environment your nodes run on.
Nothing to declare, nothing to install twice:
For the browser dashboard, sign up at [fluksio.com](https://fluksio.com), add
an instance, and enter the code it gives you:
```sh
python -m venv .venv && . .venv/bin/activate
pip install torch numpy pandas # what you were going to install anyway
pip install fluksio # and then this
fluksio serve
fluksio enroll <code>
```
`fluksio serve` says which interpreter it settled on:
Select the instance in the portal. The CLI also works without enrolling.
```text
Nodes /home/you/research/.venv/bin/python
your environment, adopted. Add packages with pip.
```
!!! tip "Use your existing research environment"
That venv is yours. Add a package the way you added the rest (`pip install
scikit-learn`) and `fluksio sync` (or a restart) retires the workers so they
pick it up. The Modules screen lists what is installed and stays read-only,
because the alternative would be Fluksio deciding what belongs in an
environment it did not make.
You can install Fluksio in the virtual environment that already holds your
research packages. Nodes use that environment by default. The startup
banner shows the interpreter; see [Packages](../code/nodes.md#packages)
for other setups.
!!! note "Your pins and ours share a site-packages"
## 2. Write a small flow
The cost of not having two environments: a package the engine depends on
is one you can now upgrade out from under it. In practice this is what
everybody does with every other tool in the venv, and the answer when it
bites is the same: pin it back, or keep Fluksio somewhere separate with
the venv of its own below.
### A venv of Fluksio's own
Sometimes you want the isolation instead: a shared instance, a container,
or an environment too precious to let a node's dependency near. Set
`NODE_VENV=managed` and Fluksio builds and owns one under the data directory:
```text
Nodes /home/you/.fluksio/user-venv/bin/python
a venv of its own; the Modules screen installs into it.
```
Then the Modules screen is how packages get in: a pip manifest, installed
with `uv pip sync` and versioned alongside your flows, so what a run imported
is recorded with what it ran:
```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"requirements\": $(jq -Rs . < requirements.txt)}"
```
Adding a package takes effect immediately; nothing restarts. The Docker image
sets `NODE_VENV=managed` for itself, because the venv in it holds the app and
nothing of yours, so a container is always this case.
!!! tip "A GPU box works the same way"
`pip install fluksio-worker` into the environment the training code runs
in, and node code runs on it: `--python` defaults to the interpreter the
agent was started with. It reads its token from `FLUKSIO_WORKER_TOKEN`.
```sh
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach --labels gpu
```
Then give the node `device="gpu"`; see [Remote workers](../code/workers.md).
## Say which functions are nodes
A **flow** is a graph of nodes. A **batch flow** is one that runs on demand
from parameters to a result, which is what an experiment is. A node is one of
your own functions. It stays in your repository, imported by its siblings as
it always was.
Say your project looks roughly like this:
Save this as `study.py` in `research-flow`:
```python
# myresearch/train.py
def fit(dataset, lr, epochs=25):
model = build_model()
for epoch in range(epochs):
loss = step(model, lr)
print(f"epoch {epoch}: {loss}") # ← this is what you are losing
torch.save(model.state_dict(), "weights.pt")
return loss
```
import json
import math
from pathlib import Path
Two changes turn it into a node, and neither of them moves it:
```python
# myresearch/train.py
import fluksio
from fluksio import Port, node
from fluksio import Flow, Port, node
@node(
requires=["dataset", Port("lr", "float")],
requires=[Port("lr", "float"), Port("epochs", "int")],
provides=[
Port("loss", "float", stream=True),
Port("weights", "artifact"),
Port("final_loss", "float"),
],
timeout=600,
)
def fit(dataset, lr, epochs=25):
"""Fit the model. A generator, so numbers escape while it is running."""
model = build_model(fluksio.load_artifact(dataset))
def fit(lr, epochs):
loss = 1.0
for epoch in range(epochs):
loss = step(model, lr)
yield {"loss": loss} # ← published now, on the loss port
torch.save(model.state_dict(), "weights.pt")
loss = math.exp(-lr * (epoch + 1))
yield {"loss": loss}
checkpoint = json.dumps({"loss": loss}).encode()
return {
"weights": fluksio.save_artifact("weights.pt"),
"weights": fluksio.save_artifact(
checkpoint, "weights.json", "application/json"
),
"final_loss": loss,
}
```
`yield` replaces `print`. Each one publishes on the node's `loss` port the
instant it happens, and the run keeps every value as a series, which is why
Fluksio has no `log_metric()` call. A metric that leaves through a port is a
message like any other: a chart can bind to it, a downstream node can consume
it, and it shows up on the canvas. A metric that escapes through a logging
function is invisible to all three.
`fluksio.save_artifact` handles the things too big to be messages: a
checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a
small reference. Nothing changes about how you write the file.
@node(requires=["weights"], provides=Port("score", "float"))
def evaluate(weights):
checkpoint = json.loads(Path(fluksio.load_artifact(weights)).read_text())
return 1.0 - checkpoint["loss"]
The decorator returns the function untouched, so `fit(dataset, 0.05)` in a
test, a notebook or a debugger is exactly what it was before. Outside a run
`fluksio.emit` does nothing rather than failing, so a progress report never
stops your own code from running.
### What is declared, and what is not
- **Ports are declared, never inferred.** `requires` names the messages the
node reads; each arrives as the parameter of the same name. A bare string is
the shorthand: `"dataset"` takes its type from whoever provides it, so a
message's type is written down exactly once.
- **Parameters with a default that are not ports become settings.** `epochs`
above is stored with the flow and tunable on the canvas without touching
this file.
- **The declaration is checked against the function.** A port with no matching
parameter, or a parameter that is neither port nor setting, is an error when
the module is imported, not when the node is first called.
- **The decorator hands the function back untouched.** `fit(dataset, lr=0.05)`
is an ordinary call, and the module is an ordinary script, so a
`if __name__ == "__main__":` check beside the nodes runs. Syncing one
defined in a script *run as* `__main__` is what cannot work, since the
generated body would have no name to import it by.
!!! note "Where a `yield` cannot reach"
If the number comes from inside somebody else's callback (Keras, Lightning,
HuggingFace `Trainer`) `fluksio.emit(loss=...)` writes the same port the
same way:
```python
model.fit(callbacks=[LambdaCallback(
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
```
### Declaring what a node needs
A training node is not like the rest of a flow. NumPy, JAX and PyTorch each
size their thread pool to every core they can see, so a few of them at once
oversubscribe the machine badly enough to starve the engine itself: the API
stops answering and every client waiting on a run dies with it. On a GPU the
same shape deadlocks: two processes each preallocating most of the card sit at
zero utilisation with nothing failing and nothing to read.
Say what one execution takes, and the engine holds it:
```python
@node(..., resources={"cpus": 2, "env": {"XLA_FLAGS": "--xla_cpu_multi_thread_eigen=false"}})
def fit(dataset, lr, epochs=25):
...
@node(..., resources={"gpus": 1, "env": {"XLA_PYTHON_CLIENT_MEM_FRACTION": "0.9"}})
def finetune(checkpoint):
...
```
Two things follow. The node waits its turn rather than starting alongside more
of itself than the machine has room for, the same waiting it already does for
a worker. And the worker it runs in is *started* with thread limits matching
what it was given, because that is the only moment a numerical library reads
them; a GPU node is told which card is its through `CUDA_VISIBLE_DEVICES`, and
nothing else is given that card while it runs.
`env` is for the tuning the engine must not invent. `XLA_FLAGS` is one composed
string whose contents depend on the version you have installed, so writing it
for you would silently replace whatever you had put there.
The engine counts NVIDIA's device nodes to find its cards, which asks no
vendor tool. A card those do not cover has to be declared (`fluksio serve
--gpus 1`, or `FLOW_GPUS`); until it is, a node asking for one is quietly
given zero and runs alongside every other, and the log says so the first time
it happens.
Workers are kept warm on purpose, so a library that takes most of the card at
import would hold it after the run finished. The workers that ran on a card
are retired when the run ends, which gives the memory back; what it costs is
the next GPU run paying for its imports again.
**The declaration is what buys that**, not touching the card. A node that
imports jax without `resources={"gpus": 1}` runs on the shared pool, is never
given `CUDA_VISIBLE_DEVICES`, and leaves a warm worker holding whatever it
preallocated. Declaring the card is what makes it a worker the engine knows
to retire, and what stops two such nodes running at once in the first place.
Host memory is held the same way, and there the declaration is not needed. A
worker that fitted a model keeps what it allocated, and on a laptop a few warm
ones are the machine. `fluksio serve --worker-max-rss 1024` retires any worker
holding more than that when its node returns, whichever pool it is in. A
node's `ram` is counted against this machine's memory, so declaring it is what
stops three large fits from starting at once.
Declaring nothing is the default and is right for most nodes: a poll, a
threshold, a message on its way somewhere. Those share the engine's worker pool
and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops
several of them at once from each sizing to the whole box.
### Ask for a size by name
Cores and gigabytes are a property of the machines you have, and those change.
A node that names a **flavor** keeps meaning something afterwards:
```python
@node(..., resources="gpu-small")
def finetune(checkpoint):
...
@node(..., resources={"flavor": "medium", "duration_s": "2h"})
def fit(dataset, lr, epochs=25):
...
```
The flavor is read every time the node is built, so editing it changes what the
next run gets. `fluksio flavors` lists them; the Workers screen edits them.
`ram` takes `"2G"` and `duration_s` takes `"30m"`, and a flavor already says how
much, so pass one or the numbers, not both. `duration_s` is a statement about the
node for whoever is planning around it, not a limit; the limit is `timeout`.
### Where it runs
This is one decision, not two. Of every machine attached (this engine and each
worker) the node goes to one that can grant what it asked for and has it free.
So a node asking for a card finds the box that has one, without naming it:
```python
@node(..., resources={"gpus": 1}) # wherever there is a card
@node(..., device="gpu", resources="gpu-small") # that box, and this much of it
```
If nothing attached can ever grant the ask, it is cut down to what is here and
runs anyway, since a flow written on a cluster still has to work on a laptop. If
something *could* but is busy, the node waits and says so.
Ask what is free, and what is waiting for it, at
`GET /api/v1/workers/resources`, on the Workers screen, or in `fluksio status`.
!!! note "Accounted, not enforced"
Nothing stops a node that ignores its declaration; the numbers are
bookkeeping plus the environment its libraries read. Real enforcement
(cgroups, rlimits) is a next step, not this one.
## Say which nodes make a flow
Membership is a list, not a directory layout: the functions can live wherever
they already do.
```python
# myresearch/pipeline.py
from fluksio import Flow, Port, use
from myresearch.data import augment, prepare
from myresearch.evaluate import evaluate
from myresearch.train import fit
train = Flow(
"train",
nodes=[prepare, fit, evaluate],
inputs=[Port("lr", "float", initial=0.01)],
outputs=["score", "final_loss"],
nodes=[fit, evaluate],
inputs=[Port("lr", "float", initial=0.05), Port("epochs", "int", initial=10)],
outputs=["score", "final_loss", "weights"],
)
```
`inputs` are the run's parameters, with the value a run gets when it names
none; `outputs` are what a run reports as its result.
`@node` declares inputs and outputs. Matching message names connect the nodes:
`fit` produces `weights`, and `evaluate` reads it. `Flow` lists the nodes,
run parameters and final results.
Nodes are connected because one **provides** a message another **requires**,
never because one imported the other. Importing `fit` into a second flow means
"the same code", not "wired to it".
Each `yield` records a loss value. The final `return` sends the checkpoint to
the next node. Artifact ports carry a small reference; the file stays in the
artifact store.
Which is how the same function serves two flows, rewired and reconfigured for
each:
```python
finetune = Flow(
"finetune",
nodes=[
prepare,
augment,
use(fit, wire={"dataset": "augmented"}, epochs=3),
evaluate,
],
inputs=[Port("lr", "float", initial=1e-4)],
outputs=["score"],
)
```
`use(fn, ...)` is one use of a node in one flow: `wire` binds a port to a
different message, keyword arguments override settings, and `id=` tells two
uses of one function apart. `train` above is unaffected by any of it.
A test can check the declarations without an engine. Importing the module
registers its flows in `fluksio.FLOWS`, and `fluksio.build(flow)` returns the
document `sync` would upload, refusing what the engine would refuse.
## Sync it
## 3. Sync and run
```sh
fluksio sync myresearch
fluksio sync study
fluksio run train --lr 0.1 --epochs 20 --follow
```
That imports the package, checks every declaration, and uploads each flow with
a generated body per node:
`sync` uploads the flow and generates node code that imports your functions.
Keep the project files available to the engine. Later `run` commands sync
edits before submitting; `--follow` waits and prints metrics as they arrive.
This run should finish with `final_loss` around `0.1353` and `score` around
`0.8647`.
```python
# generated by fluksio sync from myresearch/train.py — edit that file instead
import sys
_REPO = '/home/you/my-research'
if _REPO not in sys.path:
sys.path.insert(0, _REPO)
from myresearch.train import fit
def process(dataset, lr, **settings):
return (yield from fit(dataset=dataset, lr=lr, **settings))
```
So the flow store still holds a complete, runnable, git-versioned definition;
the body simply happens to import rather than duplicate. Your code stays in
your repository, under your version control.
The upload is stamped with that repository's commit, and every run records it
alongside the store's own. `--dry-run` prints all of this and uploads nothing;
`--no-publish` leaves it as a draft.
!!! warning "Sync after you edit"
The engine's workers are long-lived and hold your imported modules in
memory, so an edit to `myresearch` is invisible until they are retired,
which is what every `fluksio sync` does, including one that has nothing to
upload. If you are editing many times an hour, attach your own interpreter
as a worker instead: it starts a process per call and therefore reads your
code fresh every run.
```sh
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
--labels local --python "$(which python)"
```
Then give the node `device="local"`.
!!! note "The repository path is a deployment detail"
The generated body puts your repository on `sys.path` by absolute path,
which is right for the machine you synced from and meaningless in a
container. For anything else, install the package where the node runs:
`-e /home/you/my-research` in the module manifest, or
`myresearch @ git+ssh://…@a1b2c3d`, which travels where a path does not.
### Which hardware a node runs on
`device` picks the worker, the same way an executor does elsewhere:
```python
@node(requires=["dataset"], provides=[...], device="gpu", device_policy="prefer")
def fit(dataset, lr, epochs=25):
...
```
`device_policy="require"` (the default) waits for a worker carrying that label;
`"prefer"` runs it locally when none is attached, which is what you want while
the GPU box is not switched on. See [Remote workers](../code/workers.md).
### What the canvas does with a synced flow
Its node bodies are generated, so the editor shows them read-only and says
where the real code is. Everything else behaves as usual, but ports and
settings changed there are overwritten by the next sync, which is the point of
the repository being the source of truth. A node whose code you edit on the
canvas makes the next sync stop and say so rather than discarding your edit;
`--force` overrides that.
## Run it
In the portal, open **Flows → train** to see the graph, or **Runs** to inspect
the parameters, metrics and results. From the CLI:
```sh
fluksio run train --lr 0.003 --seed 7
fluksio runs --flow train
fluksio artifacts <run-id>
fluksio artifacts <run-id> train.weights -o weights.json
```
`run` syncs first, so after an edit the command is just `fluksio run`, with no
step to forget. `--no-sync` skips it for a tight loop where nothing
changed, since syncing retires the workers and the next call pays its imports
again.
Use the ID printed by `run`. Artifact names are flow-qualified; the list
command shows the exact names to download.
`--lr` is typed by the flow's own inputs, so `0.003` arrives as a float. A
parameter you did not declare, or one of the wrong type, is refused before
anything executes. It answers immediately with a queued run, since training is
measured in hours, so nothing waits for it unless you pass `--wait`, or
`--follow`, which waits and prints the numbers as the run reports them.
## 4. Try a few learning rates
No engine has to be up for any of this: `fluksio run --local` boots the
engine inside the command instead, on the same `.fluksio`: the same database,
artifacts and history a served one would use. It costs a few seconds of
startup per invocation against the ~15 ms of submitting to an engine that is
already up, so it is for the run you want now rather than the loop you are
iterating in.
```sh
fluksio sweep train --param lr=0.01,0.05,0.1 --wait
fluksio export metrics --flow train -o curves.csv
fluksio export runs --flow train --status ok -o results.csv
```
A second run of a flow you did not change mostly does not execute. Each of
your nodes is fingerprinted by its settings, the ports it declares, the values
it reads, and a digest of your repository's Python files, so editing anything
your node calls into invalidates it, not only the decorated function itself.
One an earlier run already answered is restored from that run rather than run
again, reported as `cached`, so a flow with nothing left to do finishes as
`(3/3 cached)`. Change `--lr` and only the nodes downstream of it run. The
digest is blunt on purpose: an edit anywhere in the repository re-runs
everything. `--no-cache` turns caching off for one run, `@node(...,
cache=False)` for one node; the caveat and the details are in
[Stage caching](../concepts/runs.md#stage-caching).
The sweep creates three runs in one group. Select them on **Runs** to compare
their curves. The exports give you a row per metric point and a row per run,
ready for your analysis.
From Python, the flow you declared is also the handle to its runs:
Python can read the same tables:
```python
from myresearch.pipeline import train
from fluksio.sdk.client import Client
run = train.submit(lr=0.003, seed=7).wait()
print(run.status, run.result)
print(run.metrics("train.loss")[-1]) # the whole series is kept
open("weights.pt", "wb").write(run.download("train.weights"))
client = Client()
results = client.export_runs(flow="train", status="ok")
curves = client.export_metrics(flow="train", name="train.loss")
```
Or over HTTP, which is what both of those are:
Pass either list to `pandas.DataFrame` if you use pandas.
```sh
curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"params": {"lr": 0.003}, "seed": 7}'
## Bring your own experiment
curl -s $FLUKSIO/runs/<id> -H "Authorization: Bearer $TOKEN" | jq
curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $TOKEN" | jq
```
Replace the toy calculation in `fit` with your training code. Keep ordinary
Python helpers separate from artifact storage so you can still test them
without an engine: `save_artifact` and `load_artifact` work only inside nodes.
The public repository's `examples/myresearch/` package shows this pattern
across several files.
The run carries its parameters, a digest of them, the seed, its result, how
long each node took, what it logged, every artifact it produced, and both
commits: the flow store's and your repository's. That is the answer to "what
was the learning rate on the run that got 94%?". When a run failed,
`run.failures` is the node that did it, with its traceback and its logs.
- Defaulted function arguments that are not input ports become node settings.
Use flow inputs for values you want to vary between runs.
- Use `fluksio.emit(loss=value)` when metrics arrive through a framework
callback instead of a loop you can `yield` from.
- Runs can reuse unchanged stages. Use `--no-cache` when you want every node
to execute; see [Stage caching](../concepts/runs.md#stage-caching).
- `--seed` records a seed and fills an input named `seed` when declared. Your
code still needs to pass that value to its random number generator.
Beside your commit is `code_digest`, and `fluksio runs` prints the pair as
`a1b2c3d-dirty+9f0e1a2`. The commit alone cannot identify what ran: your node
bodies are imports, so the engine executes whatever is on disk when the worker
starts, and an uncommitted tree stamps `-dirty` for every run it ever produces.
The digest is over the modules this flow's nodes actually reach, not the
whole repository, so it moves when the code behind a number moves and stays
put when a notebook beside it changes. It is read at the moment the run
starts, so in a sweep whose runs queue for hours each one records the code
that actually executed it rather than the code that was there when you
submitted. It is what makes the column worth joining an
[exported table](../concepts/runs.md#taking-it-into-a-dataframe) on.
### Declaring what a node needs
### When the engine is busy
A driver script outlives the engine being slow, because a sweep is hours long
and a blip is seconds:
- **Reads are repeated.** Every GET, and the writes that mean the same thing
twice, are tried up to three more times on a transport error or a `502`,
`503` or `504`, waiting 1 s, 2 s, then 4 s. `Client(retries=0)` turns that
off.
- **Submitting is safe to repeat.** Each `submit()` mints a key the engine
stores with the run, so an attempt that timed out on the way back is
answered with the run it already made rather than starting a second one. A
sweep keys every entry, so retrying a half-created one creates only what is
missing.
- **Waiting tolerates a gap.** `wait()` and `--follow` survive five failed
polls in a row before giving up. An answer is not a gap: a `404` for a run
that is gone stops immediately.
- **Timeouts are split** — 5 s to connect, 120 s to read, so a wrong URL fails
at once while a busy engine is given longer than the slowest thing it does
on purpose. `Client(timeout=…)` takes a number or an `httpx.Timeout`.
A command that still cannot reach the engine says
`fluksio: engine not answering (…)` and names the run it started, which is
running on the engine regardless, and `fluksio runs` finds it again.
## Sweep it
A grid search and an ensemble are the same submission, run in parallel:
```sh
fluksio sweep train --param lr=0.001,0.003,0.01 --wait
```
Every combination of the lists you give, so a second `--param` is a grid
rather than a second sweep. Where the set you want is not a product, build the
list yourself and post it:
```sh
curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"runs": [
{"params": {"lr": 0.001}, "seed": 1},
{"params": {"lr": 0.003}, "seed": 1},
{"params": {"lr": 0.010}, "seed": 1}
]}'
```
They share a `group_id`, so the sweep is `GET /runs?group=…`. Running them
concurrently is safe because each run gets a state backend of its own: two
runs of one flow cannot overwrite each other's values.
Compare the curves in one call:
```sh
curl -s "$FLUKSIO/runs/series/compare?ids=$A,$B,$C&metric=train.loss" \
-H "Authorization: Bearer $TOKEN"
```
which answers in exactly the shape a chart widget draws.
## Take it into a dataframe
The analysis itself wants a table, and `export` writes the two an analysis
asks for: the curves long, and one row per run with the parameters that
varied beside its final numbers:
```sh
fluksio export metrics --flow train --name train.loss -o curves.csv
fluksio export runs --flow train --status ok -o arms.csv
```
Both carry the run id on every row, and `arms.csv` carries the commit and the
code digest beside it, so an exported file still says what produced its
numbers. Numbers inside a record are columns of their own
(`metric.final_metrics.train_loss`) and `--metrics` and `--params` take those
dotted paths to narrow the table. Metric names are flow-qualified, so
`--list` prints every one a selection recorded. `Client.export_metrics()` and
`Client.export_runs()` answer the same rows to a notebook, ready for
`pandas.DataFrame`.
## Looking at them in the portal
None of the above needs a browser, but there is one. **Runs** is the table of
everything that has run: filter it to a flow, to a status, or to one sweep,
where it grows a column per parameter that actually varied, and tick two runs
to put their curves side by side.
A run records its metrics under the same names a dashboard binds to, so a
dashboard is another way of reading one. **Open in dashboard** carries the
runs you picked along in the URL, and every widget then answers for those runs
instead of for the live engine. If the flow has no dashboard yet, the same
menu builds one from the ports the flow declares.
See [Runs → Looking at what ran](../concepts/runs.md#looking-at-what-ran).
## What sync does, at the API level
Nothing here is privileged: `sync` is a client, and a flow is a document you
can PUT yourself. The whole of it, for the one-node case:
For heavy computation, declare resources on the node:
```python
"""Create the `train` flow by hand. What `fluksio sync` automates."""
import httpx
API = "http://127.0.0.1:8000/api/v1"
api = httpx.Client(base_url=API, timeout=60)
token = api.post(
"/login/access-token",
data={"username": "admin@example.com", "password": "example-password"},
).json()["access_token"]
api.headers["Authorization"] = f"Bearer {token}"
api.put("/flows/train", json={
"name": "train",
"title": "Model training",
# Batch: nothing is activated, nothing fires until a run asks.
"mode": "batch",
"inputs": [
{"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01},
{"spec": {"name": "epochs", "dtype": "int"}, "initial": 50},
],
"outputs": ["final_loss", "weights"],
"nodes": [{
"id": "train",
"type": "python",
"title": "Fit the model",
# An *idle* timeout once the node streams: this is how long it may go
# quiet, not how long it may run.
"timeout": 600,
"requires": [
{"name": "lr", "dtype": "float"},
{"name": "epochs", "dtype": "int"},
],
"provides": [
# `stream` says this port publishes repeatedly during one execution.
{"name": "loss", "dtype": "float", "stream": True},
{"name": "final_loss", "dtype": "float"},
{"name": "weights", "dtype": "artifact"},
],
}],
}).raise_for_status()
api.put("/flows/train/nodes/train/source",
json={"code": open("train_node.py").read()}).raise_for_status()
version = api.get("/flows/train").json()["definition"]["version"]
api.post("/flows/train/publish", json={"version": version}).raise_for_status()
@node(requires=["dataset"], provides=Port("score", "float"),
resources={"cpus": 2, "ram": "2G"})
def score_dataset(dataset):
return {"score": calculate_score(dataset)}
```
Two things worth noticing, because the decorators only move where they are
said. Ports are declared, not inferred: `process(lr, epochs)` gets its
arguments from the ports of the same name, and the types are checked on every
value. And saving writes a *draft*; `publish` is what the engine picks up.
That separation is what lets you edit a flow that is running.
The engine accounts for these resources and sets worker thread limits. GPU
nodes can request `resources={"gpus": 1}`; check that the Workers screen
reports your GPU, or set `fluksio serve --gpus 1` when automatic detection
does not cover it. Resource accounting does not impose a hard memory limit.
A flow uploaded this way carries no `origin`, which is what tells the canvas,
and the next `fluksio sync`, that it was not generated. See
[The API](../code/api.md).
To run on another machine, attach a [remote worker](../code/workers.md) and
install your project and its dependencies there.
## Small scripts you are just playing with
## Next steps
The same machinery, minus the ceremony. If what you want is "keep a record of
every version of this thing I keep tweaking", note that **your flows are
already a git repository**:
```sh
cd .fluksio/flows
git log --oneline
```
Every save is a commit: the node source, the parameters, the graph. A run
records the commit it ran at, so `git show` on that hash is literally the code
that produced the number. You get the versioning without adopting anything.
For quick iteration, keep the flow small (one node is fine), keep the engine
running, and submit from wherever you are working:
```python
from myresearch.pipeline import train
for lr in (0.001, 0.003, 0.01):
train.submit(lr=lr)
```
A submit is around 15 ms, so calling that in a loop is a reasonable thing to do.
## Get a browser onto it
The pip install gives you the engine and the API, not a web interface, since a
machine with no inbound route cannot serve one usefully. To see the
canvas, the run history and live loss curves, pair the instance with a
portal, which serves the dashboard from its side:
```sh
fluksio enroll <claim-code> --portal https://hub.fluksio.com
```
Get the claim code from the portal under **Instances → Add instance**.
Nothing needs to be exposed: your machine dials out and holds the connection
open. See [Accounts and the portal](../interface/portal.md).
If you would rather stay entirely offline, run the dashboard SPA yourself from
the app's Docker image; see the [facility path](facility-automation.md), which
is the same stack.
## Where to go next
- [Runs: pipelines that finish](../concepts/runs.md) — the full picture:
artifacts, sweeps, durability, what happens when your engine dies mid-training
- [Writing node code](../code/nodes.md) — generators, settings, what a node may
and may not do
- [The command line](../code/cli.md) — `login`, `sync`, `run`, `runs`, `sweep`
in full
- [Remote workers](../code/workers.md) — send the training node to the GPU box
and keep the rest on your laptop
- [The flow editor](../interface/flow-editor.md) — once you have a portal, this
is where the graph gets easier to change than the script did
- [Writing node code](../code/nodes.md) — ports, streaming, artifacts and packages.
- [Runs](../concepts/runs.md) — caching, comparisons and retries.
- [CLI reference](../code/cli.md) — command options and instance settings.
docs/getting-started/facility-automation.md
+114 -275
View File
@@ -1,325 +1,164 @@
# Getting started: facility automation
You have a box in a cupboard, a handful of sensors that already publish
somewhere, and an ambition to make the house do something about them. This
page brings up a Fluksio instance you can leave running for years, then wires
the first sensor through to a dashboard.
Build a small temperature flow, then connect a real sensor when you're ready.
No hardware required for the first lap.
Budget an afternoon. Most of it is the browser, which is the point: you will be
in this interface a lot, so it is worth learning it properly.
## Start the Docker stack
## What you are standing up
```text
sensors ──MQTT──▶ ┌──────────────┐ ──▶ InfluxDB (history)
│ Fluksio │
HTTP / webhooks ──▶│ flow engine │ ──▶ dashboards (what you look at)
│ │
schedules ────────▶└──────────────┘ ──▶ MQTT / HTTP (what you control)
│
└── workers on other boxes (optional)
```
One engine holds every flow. Nothing here is a plugin you install separately:
the broker client, the time-series writer, the dashboards and the alerting are
all part of the same process, editing the same graph.
## Prerequisites
- Docker and Compose v2 on the host
- The address you will type into a browser. On a home network that is the
server's own address, `192.168.1.50`, and nothing else is needed. A
hostname is optional, and covered below
- Optionally: an MQTT broker and an InfluxDB you already run. If not, the stack
can start both for you
## Set it up
Clone the app repository and try it on the machine you are sitting at:
You need Git, Docker with Compose v2, `make`, and `openssl`. Port 80 must be
free. Clone the public app repository and run its commands from the repo root:
```sh
git clone https://git.stroblme.de/Fluksio/app.git ~/fluksio
cd ~/fluksio
make init # once: .env with generated secrets, `proxy` network, prints the admin login
make dev-local # full stack at http://app.localhost (API at http://api.localhost)
make down # stop it
git clone https://git.stroblme.de/Fluksio/app.git fluksio
cd fluksio
make init
make dev-local
```
`make dev-local` needs port 80 free and only answers on `localhost`; the
sections below make it reachable from other machines. `make init` is safe to
run again: it only fills secrets still set to `changethis`, and `.env`
otherwise stays as you left it.
`make init` creates `.env`, generates secrets, creates the `proxy` network,
and prints your admin login. It is safe to run again: existing secrets stay
as they are.
For an instance you leave running, edit `.env` first (`FIRST_SUPERUSER`,
`ENVIRONMENT=production`). It is the whole configuration. The four settings that matter on day one:
Give the containers a moment to start on the first build, then open
[http://app.localhost](http://app.localhost). Sign in with the admin login:
adding Python code requires a superuser account. The API is at
[http://api.localhost](http://api.localhost), and the docs are at
[http://docs.localhost](http://docs.localhost).
| Setting | What it does |
|---|---|
| `DOMAIN` | the hostname everything is served under, if you use one. Left alone it only names the issuer that agents authenticate against |
| `FIRST_SUPERUSER` | the account you sign in with |
| `FIRST_SUPERUSER_PASSWORD` | generated by `make init`; it prints it, and it stays in `.env` |
| `ENVIRONMENT` | `production` closes the interactive API schema; `local` leaves it open |
Everything the instance owns (the database, your flows, secrets, artifacts,
the packages your node code imports) is on one Docker volume.
Backing that volume up is backing up the instance.
??? note "Even smaller: no Docker at all"
`pip install fluksio && fluksio serve` gives you the same engine with no
containers, keeping its data in `~/.fluksio`. What it does not give you is
the web interface, which the SPA container serves, so you would drive it
from the API, or [pair it with a portal](../interface/portal.md) that
serves the dashboard for you. Good for a Raspberry Pi that only runs flows;
less good as your main instance.
## Reach it from another machine
Start it one of two ways, depending on whether you have a hostname to point at
it. Neither is more supported than the other.
### On your own network, by address
The usual case: the box is in a cupboard at `192.168.1.50` and you open it from
a laptop. Layer `docker/compose.lan.yml` on, which publishes the interface on a
port and serves the API from that same origin:
For a Docker instance reached from another machine on your network, use:
```sh
APP_PORT=8080 docker compose -p fluksio-app --env-file .env \
-f docker/compose.yml -f docker/compose.lan.yml up -d --build
make dev-lan
```
Open `http://192.168.1.50:8080`. That one address is the whole application:
the interface, the API under `/api`, and the websocket that puts live values on
the canvas.
Open `http://<host-ip>:8080`. Choose another port with
`make dev-lan APP_PORT=8081`. This is a development stack; see
[running it long term](#run-it-long-term) before using it for a deployment.
Nothing in that URL is configured anywhere. The interface is built to address
the API relative to whatever origin served the page, so the same containers
answer to the machine's address, to a hostname you add later, and through an
ssh tunnel, without being rebuilt. It is also why no CORS list needs
maintaining: the browser is talking to one origin.
!!! danger "This is your network, not the internet"
A published port is plain HTTP with no certificate. Between your own
machines that is fine, and it is not something to forward from a router. To
reach the house from outside, put a reverse proxy with TLS in front of it,
or [pair it with a portal](../interface/portal.md), which needs no inbound
route at all.
### With a hostname, behind a reverse proxy
If you have DNS, or want certificates, set `DOMAIN` and run `make up`
instead. The interface is then served on `app.${DOMAIN}` and the API on
`api.${DOMAIN}`, and nothing is published on a host port.
The stack emits Traefik labels and ships a Traefik you can bring up alongside
it (`docker/compose.traefik.yml`). With Nginx Proxy Manager or Caddy instead,
attach it to the `proxy` network and forward `app.${DOMAIN}` →
`fluksio-app:80` and `api.${DOMAIN}` → `fluksio-api:8000`.
!!! tip "Developing on the same machine"
`make dev-lan` is `make dev-local` plus the published port: the app on
`http://<host-ip>:8080`, `APP_PORT` to move it. `make dev-local` recreates
the stack without that port, so re-run `make dev-lan` after one.
## Sign in
Open whichever address you chose (`http://app.localhost` for `make dev-local`)
and sign in with the login `make init` printed. You should be looking at Home:
an empty brain graph, a health summary, and a flow list with nothing in it.
When you are done, stop the stack with `make down`. Your data volumes stay
in place for the next start.
## Your first flow
Go to **Flows → New flow** and call it `house`. You land on the canvas.
Open **Flows**, create a flow named `house`, and use **Add node** to add an
**Inject** node. In its panel:
A flow is a set of nodes that talk to each other through **named messages**.
You do not draw wires: a node says which messages it needs and which it
produces, and the canvas draws the graph that follows. Renaming is safe, fan-in
is free, and two flows share a value by naming it.
- Under **Provides**, add `living_temperature` with type `float`.
- Under **Settings**, set the `living_temperature` value to `18.0` and
**Interval** to `5` seconds.
### Read a sensor
This gives you a pretend sensor. Add a **Function** node with:
Press **Add node** (or ⌘P / Ctrl-P, which opens the command palette) and pick
**MQTT**. In its panel on the right:
- **Consumes**: `living_temperature`, type `float`.
- **Provides**: `heat_wanted`, type `bool`.
- **Broker host** — your broker's hostname, `mosquitto` if you are using the
one the stack can start
- **Topic** — map each output to a topic:
`{"living_temperature": "zigbee2mqtt/living/temperature"}`
- **Provides** — add one output port named `living_temperature`, type `float`
That is a working node. Press **Publish** (⌘S) and the engine picks it up.
The canvas now draws your node with a live value on its output as soon as the
broker sends one. Click the wire to see the last payload and its history.
### Do something with it
Add a **Function** node. This is a Python node, and the code editor opens in
its panel:
In the code editor, enter:
```python
def process(living_temperature, comfortable=21.0):
"""Ask for heat when the room is below the comfort point."""
return {"heat_wanted": living_temperature < comfortable}
def process(living_temperature):
return {"heat_wanted": living_temperature < 21.0}
```
Declare `living_temperature` as an input (type `float`) and `heat_wanted` as an
output (type `bool`). `comfortable` is not a port but a **setting**,
because it is a constant of this node rather than something the graph carries.
It shows up as a field in the node's Settings section.
Message names connect the nodes automatically; there are no wires to draw.
Save the code with ⌘S / Ctrl-S, then click **Publish** in the dock. Open
**Flow settings** from the dock's pencil button and make sure **Running** is on.
The canvas now draws MQTT → your function, because the message names line up.
Nothing else was needed.
Every five seconds, `18.0` should produce `true`. Select the Function and check
the value under its `heat_wanted` output. Change the Inject value to `22.0`,
publish again, and check that the result becomes `false`. Click the edge to
inspect the temperature readings. Your first bit of automation is alive.
### Act on it
If nothing happens, open **Issues** and **Logs** in the dock. They show missing
inputs, code errors and connection problems. See [The flow editor](../interface/flow-editor.md)
for the controls and shortcuts.
Add a second **MQTT** node, this time with `heat_wanted` as an *input*, and a
topic mapping to whatever your relay listens on. A node with inputs publishes;
a node with outputs subscribes.
## Connect a real sensor
Publish the flow. You have a thermostat.
Replace the Inject node with an **MQTT** node:
!!! warning "Test before it touches a relay"
| Field | Value |
|---|---|
| **Provides** | `living_temperature`, type `float` |
| `broker_host` | your broker's hostname or IP, reachable from the engine |
| `topic` | the sensor's temperature topic |
A flow can be **paused** (it holds messages instead of running them) and
**stepped** (release exactly one). Together with the run button, which
injects a value by hand, that is how you convince yourself the logic is
right before the contactor finds out. Both live on the dock at the bottom
of the canvas.
Publish the change. The Function now responds to sensor readings instead of
the test value. If your sensor sends a JSON object such as
`{"temperature": 18.0}`, set `json_key` to `temperature`.
## Store the history
An MQTT node with input ports publishes to its configured topic; one with
output ports subscribes. Add a publishing node only after checking the payload
format your device expects. A relay may need `ON` and `OFF` rather than a
boolean.
A live value is enough to control something and useless for answering "was last
February colder?". That is what the **InfluxDB** node is for.
!!! tip "Keep the first test on screen"
Add one, and configure it to write the message you already have:
Check both warm and cold readings before connecting an actuator. **Pause**
and **Step** let you inspect queued messages one at a time. A real heating
controller also needs appropriate limits and hysteresis.
```json
{
"url": "http://influxdb:8086",
"token": {"$secret": "influx-token"},
"org": "home",
"bucket": "sensors",
"writes": {
"living_temperature": {
"measurement": "environment",
"field": "temp_c",
"tags": {"room": "living"}
}
}
}
```
Docker development stacks include a test broker at `mosquitto:1883`; it has
no published host port, so your physical sensors cannot reach it directly.
For real sensors, use your own broker. The Docker image includes the MQTT
and InfluxDB connector dependencies.
Give it `living_temperature` as an input and every value that passes gets a
point.
## Show it on a dashboard
Note the token. Credentials never sit in a flow: `{"$secret": "influx-token"}`
is a reference into an encrypted store, and the editor renders those fields as
a secret picker. Add the actual value once under **Secrets**. Flows are a git
repository you might well push somewhere, so this keeps a password out of it.
Create a dashboard and add a gauge bound to `house.living_temperature`.
Publish the dashboard to see the current reading. Dashboard controls can also
publish messages, so choose their bindings with care.
Reading back is the same node with `queries` instead of `writes`, or, for
anything a chart asks for, a pair of small Python nodes on either side that
build a Flux query and shape its rows. That indirection is deliberate: the
database node holds the connection and nothing else, so a dashboard widget
never learns which database answered it.
See [Dashboards and panels](../interface/dashboards.md) for widgets and pairing
a wall tablet.
## Put it on a screen
For longer-term history, add an **InfluxDB** node that consumes
`living_temperature`. Configure your server URL, organization, bucket, token
and `writes` mapping. Store the token under **Secrets** and select it in the
node's credential field. [Node types](../reference/node-types.md#influxdb)
has the write and query formats.
**Dashboards → New dashboard**, then drag widgets onto the grid and bind each
one to a message. A gauge on `house.living_temperature`, a switch on
`house.heat_wanted`, a chart on the history.
The Docker development stack includes an InfluxDB test service at
`http://influxdb:8086`, with organization and bucket `fluksio` and token
`fluksio-dev-token`. It is for testing; configure a persistent InfluxDB server
for data you want to keep. The production stack includes neither InfluxDB nor
an MQTT broker.
Widgets are typed the same way ports are: a switch binds to a `bool`, a gauge
to a number, an agenda to a `list`. Bind it wrong and the editor says so rather
than drawing nothing.
## Run it long term
Controls work in the other direction: a switch on a dashboard publishes the
message it is bound to, exactly as a node would. The canvas draws it as a
labelled endpoint feeding the nodes that read it, so a value never appears from
nowhere.
For a Docker deployment, run `make init`, then edit the repository's `.env`:
For a tablet on the wall, see [Dashboards and panels](../interface/dashboards.md):
a panel is a named device, it pairs with a six-character code instead of a
login, and it can only reach the dashboards you gave it.
| Setting | Example |
|---|---|
| `DOMAIN` | `example.com` |
| `ENVIRONMENT` | `production` |
| `FRONTEND_HOST` | `https://app.example.com` |
| `VITE_API_URL` | `https://api.example.com` |
| `BACKEND_CORS_ORIGINS` | `https://app.example.com` |
| `FIRST_SUPERUSER` | your administrator email |
| `TZ` | your timezone, such as `Europe/Berlin` |
## Spread it across machines
Keep the generated secrets. Stop any development stack with `make down`.
Set up DNS and a reverse proxy with HTTPS and WebSocket support before running
`make up`. That target starts the application services; it requires a proxy
on the Docker `proxy` network. Route
`app.example.com` to `fluksio-app:80`, `api.example.com` to `fluksio-api:8000`
and, optionally, `docs.example.com` to `fluksio-docs:80`.
You now have one box doing everything. Two reasons to change that: something
lives on a different network, or something needs hardware the engine's host
does not have.
`docker/compose.traefik.yml` is an optional proxy configuration. It also needs
`EMAIL`, `USERNAME` and `HASHED_PASSWORD` for certificates and dashboard
authentication; `make init` does not set these up.
The unit of distribution is the **worker**. It runs the code of nodes you mark
for it, and it dials *out* to the engine, so the Pi in the shed does not need
an inbound route, and the engine does not need to reach it.
The published port from `make dev-lan` uses HTTP. For access outside your
network, use HTTPS or [enroll with the portal](../interface/portal.md).
On the engine, mint a token:
Back up `.env` and both persistent volumes: `app-flow-data` holds the database,
flows, secrets and artifacts; `app-redis-data` holds message values, node state
and queued work. Compose prefixes volume names with the project name
`fluksio-app`. Also back up any external MQTT or InfluxDB data you need.
```sh
curl -X POST https://api.${DOMAIN}/api/v1/workers/tokens \
-H "Authorization: Bearer $TOKEN" -d '{"name": "shed-pi", "labels": ["shed", "gpio"]}'
```
## Next steps
The labels are signed into it, so the worker can repeat them with `--labels`
but not change them.
On the other machine, with that token in `FLUKSIO_WORKER_TOKEN`:
```sh
pip install fluksio-worker
fluksio-worker \
--url wss://api.${DOMAIN}/api/v1/workers/attach \
--labels shed,gpio \
--parallel 2
```
Then mark the node that talks to the shed's GPIO with `device: shed`, and it
runs there. Everything else stays where it is. A node bound to a label no
attached worker carries simply waits rather than failing, so you can write the
flow before the hardware arrives.
!!! note "What a worker is not"
It is not a second engine. Subscriptions, schedules, webhooks and the
dashboards all stay in one process, which keeps a value having one
definition. A worker executes node bodies, nothing else. Scaling *the
engine* to several processes is not supported: run one.
## Make it tell you when something breaks
Under **Alerts**, add a channel and a rule. Channels are ntfy (a push
notification on your phone), SMTP, a webhook, or a message a dashboard
notification widget reads.
What you probably want on day one is everything, to ntfy:
- add an ntfy channel with your server and topic
- add a rule with no events ticked, which means *all of them*
The engine deduplicates aggressively, so the same node failing every second is
one alert rather than thirty-six thousand, and caps the total at ten an hour
however bad it gets. See [Secrets, modules and alerts](../interface/operations.md).
## Keep it alive
- **Back up the data volume.** That is the database, the flows, the secrets and
the artifacts. Everything else is rebuildable.
- **Watch Home.** The health summary names what is wrong — a quarantined flow, a
node that will not load, a stalled queue, a flow that cannot run because its
graph does not validate.
- **Flows are git.** `git log` inside the flow store is the history of every
change anyone made, and reverting one is a revert.
## Where to go next
- [Flows, nodes and messages](../concepts/flows.md) — the model, properly
- [Node types](../reference/node-types.md) — everything you can put on a canvas
without writing Python
- [The flow editor](../interface/flow-editor.md) — the canvas, in detail
- [Keeping state in a flow](../concepts/state.md) — running totals, debounces,
and the one rule that makes them safe
- [Accounts and the portal](../interface/portal.md) — reach the instance
from outside the house without opening a port
- [Flows, nodes and messages](../concepts/flows.md): how the graph works.
- [Secrets, modules and alerts](../interface/operations.md): credentials and notifications.
- [Remote workers](../code/workers.md): run node code on another machine.
- [Configuration](../reference/configuration.md): settings and storage paths.
docs/getting-started/index.md
+17 -44
View File
@@ -1,9 +1,8 @@
# Pick your starting point
Setup differs depending on what you are here for. Pick the path that matches.
Pick the one that sounds like you. Everything past this section is the same for
both.
Choose the setup that fits your work.
Both paths lead to the same Fluksio engine, with a first flow to try along the way.
This is really just about a decision what you're planning to do with Fluksio.
<div class="fluksio-lanes" markdown>
@@ -11,13 +10,13 @@ both.
### Data science
*"I have a training script. I want to stop losing track of what I ran."*
*"I have some Python code. I want to keep track of what I ran."*
One `pip install`, one command, and you are writing Python again. No Docker, no
database, no ports to open. Flows are files, runs are rows, and the metrics are
just the numbers your loop already produces.
Install Fluksio with pip in your research environment. Turn a function into a
node, record metrics and artifacts, and compare experiments. Connect to
[fluksio.com](https://fluksio.com) for the browser dashboard.
[Set up for experiments →](data-science.md)
[Start with Python →](data-science.md)
</div>
@@ -25,46 +24,20 @@ just the numbers your loop already produces.
### Facility automation
*"I have a homelab and a pile of sensors. I want them to do something."*
*"I have a bunch of sensors. I want them to do something."*
A stack you bring up once and leave running: the engine, a broker, a
time-series database, dashboards, alerting. Most of the work happens in the
browser, and it is worth doing properly because you will live in it.
Start the Docker stack for a local engine and browser interface. Build a live
flow, connect your devices, and put their values on a dashboard.
[Set up a homelab instance →](facility-automation.md)
[Start with Docker →](facility-automation.md)
</div>
</div>
## Not sure?
## Next steps
Some rough tells:
| | Data science | Facility automation |
|---|---|---|
| **The flow** | starts, finishes, has a result | never ends |
| **You mostly** | write Python | wire nodes in the browser |
| **Time to first result** | a few minutes | an afternoon |
| **Runs on** | your laptop, or a login node | a box in a cupboard |
| **Data lives in** | SQLite beside the flows | InfluxDB, usually |
| **The thing you look at** | run history and loss curves | a dashboard, maybe on a wall |
If both describe you (a lab with instruments to drive *and* models to fit),
start with the data-science path. It is the smaller instance, and it
grows into the other one without being reinstalled: the same engine, the same
flows, just more of them running all the time.
## What is the same either way
Whichever door you came in:
- **Flows are files in a git repository.** Every save is a commit. You can read
the history with ordinary git, and you can copy a flow between instances
by copying a directory.
- **Editing is separate from running.** You edit a draft; the engine keeps
running what was published until you publish.
- **Nodes are typed.** A port declares what it carries, and a mismatch is
caught at edit time rather than at three in the morning.
- **Everything the browser does is an API call.** The dashboard is a client of
the same REST API you can script against.
- [Flows, nodes and messages](../concepts/flows.md) explains the graph.
- [The flow editor](../interface/flow-editor.md) covers editing and publishing.
- [The command line](../code/cli.md) covers local and remote use.
- [Configuration](../reference/configuration.md) lists settings and data paths.