app/docs/getting-started/data-science.md

Getting started: data science

You have a training script. It works. What you do not have is any reliable answer to "what was the learning rate on the run that got 94%?", and the results_final_v3_ACTUAL.csv in your home directory is not helping.

This page adds Fluksio to what you already have. It takes about five minutes, installs one Python package, and does not ask you to restructure anything.

Install

cd my-research
pip install fluksio
fluksio serve

That is the whole installation. No Docker, no database server, no ports to open, and no login. The first run prints something like:

Created the admin account admin@example.com
  password: k3Qm-8vTpLdX
  Shown once. Change it from the dashboard.
Fluksio 0.1.0 — data in /home/you/my-research/.fluksio
  API      http://127.0.0.1:8000/api/v1
  Nodes    /home/you/my-research/.venv/bin/python
           your environment, adopted. Add packages with pip.
  No portal. Pair this installation with:
    fluksio enroll <code>
  Signed in as admin@example.com
           token in /home/you/my-research/.fluksio/client.json

Read the last two lines: you are already signed in. Signing in to your own machine is a formality — the password was printed by the same process that would have checked it — so serve writes the token itself and every command below just works. fluksio login is for an engine somewhere else.

Write that password down anyway. It is shown once, and it is what the dashboard asks for.

One installation per project

.fluksio sits beside your code, and is found the way .git is — from the directory you are standing in, or any directory above it. So two repositories on one machine are two engines: separate flows, separate run history, separate token, and no chance of one experiment's graph turning up in the other's.

~/research/protein-fold/
  .venv/          torch, fluksio
  .fluksio/       its own database, flows, artifacts, token
  myresearch/

~/research/climate-sim/
  .venv/          jax, fluksio
  .fluksio/       its own everything
  climate/

It holds a database and a credential, so it ignores itself from within — a .gitignore of *, the same thing uv writes into .venv. Nothing to add to your project's own.

Give them different ports (--port) if you want two running at once.

!!! tip "One engine for the machine instead"

`fluksio serve --global` uses `~/.fluksio` — shared by every directory,
which is what you want for a personal server rather than a project. When
both exist, the banner says which one you are looking at and how to reach
the other. `--data-dir` still names any directory outright, which is worth
doing on a cluster where `$HOME` is a network filesystem: SQLite's
write-ahead log does not work on NFS, and `fluksio serve` warns you when
it notices.

!!! tip "Keep it running"

The engine is meant to be resident. That is the whole reason submitting a
run costs about 15 ms instead of the second a project-bootstrapping
orchestrator spends before it does anything. Leave it in a `tmux` window,
or write a small `systemd --user` unit for it.

Talking to it over HTTP

Everything the commands do is the HTTP API, and some of this page shows it directly. For that, take the token serve already wrote:

export FLUKSIO=http://127.0.0.1:8000/api/v1
export TOKEN=$(jq -r .token .fluksio/client.json)

While you are experimenting, the interactive schema at http://127.0.0.1:8000/docs is the fastest way to see what is available.

For an engine on another machine, fluksio login --url https://… asks for a password and stores the token the same way — in this project's .fluksio, or with --global in ~/.fluksio.

Your packages are already there

If you installed Fluksio into the environment you work in — the venv that already has torch or numpy in it — that is the environment your nodes run on. Nothing to declare, nothing to install twice:

python -m venv .venv && . .venv/bin/activate
pip install torch numpy pandas      # what you were going to install anyway
pip install fluksio                 # and then this
fluksio serve

fluksio serve says which interpreter it settled on:

  Nodes    /home/you/research/.venv/bin/python
           your environment, adopted. Add packages with pip.

That venv is yours. Add a package the way you added the rest — pip install scikit-learn — and fluksio sync (or a restart) retires the workers so they pick it up. The Modules screen lists what is installed and stays read-only, because the alternative would be Fluksio deciding what belongs in an environment it did not make.

!!! note "Your pins and ours share a site-packages"

The cost of not having two environments: a package the engine depends on
is one you can now upgrade out from under it. In practice this is what
everybody does with every other tool in the venv, and the answer when it
bites is the same — pin it back, or keep Fluksio somewhere separate with
the venv of its own below.

A venv of Fluksio's own

Sometimes you want the isolation instead: a shared installation, a container, or an environment too precious to let a node's dependency near. Set NODE_VENV=managed and Fluksio builds and owns one under the data directory:

  Nodes    /home/you/.fluksio/user-venv/bin/python
           a venv of its own; the Modules screen installs into it.

Then the Modules screen is how packages get in — a pip manifest, installed with uv pip sync and versioned alongside your flows, so what a run imported is recorded with what it ran:

curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{\"requirements\": $(jq -Rs . < requirements.txt)}"

Adding a package takes effect immediately; nothing restarts. The Docker image sets NODE_VENV=managed for itself, because the venv in it holds the app and nothing of yours — so a container is always this case.

!!! tip "A GPU box works the same way"

`pip install fluksio-worker` into the environment the training code runs
in, and node code runs on it: `--python` defaults to the interpreter the
agent was started with.

```sh
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
  --token "$WORKER_TOKEN" --labels gpu
```

Then give the node `device="gpu"` — see [Remote workers](../code/workers.md).

Say which functions are nodes

A flow is a graph of nodes. A batch flow is one that runs on demand from parameters to a result, which is what an experiment is. A node is one of your own functions — it stays in your repository, imported by its siblings as it always was.

Say your project looks roughly like this:

# myresearch/train.py
def fit(dataset, lr, epochs=25):
    model = build_model()
    for epoch in range(epochs):
        loss = step(model, lr)
        print(f"epoch {epoch}: {loss}")     # ← this is what you are losing
    torch.save(model.state_dict(), "weights.pt")
    return loss

Two changes turn it into a node, and neither of them moves it:

# myresearch/train.py
import fluksio
from fluksio import Port, node


@node(
    requires=["dataset", Port("lr", "float")],
    provides=[
        Port("loss", "float", stream=True),
        Port("weights", "artifact"),
        Port("final_loss", "float"),
    ],
    timeout=600,
)
def fit(dataset, lr, epochs=25):
    """Fit the model. A generator, so numbers escape while it is running."""
    model = build_model(fluksio.load_artifact(dataset))
    for epoch in range(epochs):
        loss = step(model, lr)
        yield {"loss": loss}                # ← published now, on the loss port
    torch.save(model.state_dict(), "weights.pt")
    return {
        "weights": fluksio.save_artifact("weights.pt"),
        "final_loss": loss,
    }

yield replaces print. Each one publishes on the node's loss port the instant it happens, and the run keeps every value as a series — which is why Fluksio has no log_metric() call. A metric that leaves through a port is a message like any other: a chart can bind to it, a downstream node can consume it, and it shows up on the canvas. A metric that escapes through a logging function is invisible to all three.

fluksio.save_artifact handles the things too big to be messages — a checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a small reference. Nothing changes about how you write the file.

The decorator returns the function untouched, so fit(dataset, 0.05) in a test, a notebook or a debugger is exactly what it was before. Outside a run fluksio.emit does nothing rather than failing, so a progress report never stops your own code from running.

What is declared, and what is not

• Ports are declared, never inferred. requires names the messages the node reads; each arrives as the parameter of the same name. A bare string is the shorthand — "dataset" takes its type from whoever provides it, so a message's type is written down exactly once.
• Parameters with a default that are not ports become settings. epochs above is stored with the flow and tunable on the canvas without touching this file.
• The declaration is checked against the function. A port with no matching parameter, or a parameter that is neither port nor setting, is an error when the module is imported — not when the node is first called.

!!! note "Where a yield cannot reach"

If the number comes from inside somebody else's callback — Keras, Lightning,
HuggingFace `Trainer` — `fluksio.emit(loss=...)` writes the same port the
same way:

```python
model.fit(callbacks=[LambdaCallback(
    on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
```

Say which nodes make a flow

Membership is a list, not a directory layout: the functions can live wherever they already do.

# myresearch/pipeline.py
from fluksio import Flow, Port, use

from myresearch.data import augment, prepare
from myresearch.evaluate import evaluate
from myresearch.train import fit

train = Flow(
    "train",
    nodes=[prepare, fit, evaluate],
    inputs=[Port("lr", "float", initial=0.01)],
    outputs=["score", "final_loss"],
)

inputs are the run's parameters, with the value a run gets when it names none; outputs are what a run reports as its result.

Nodes are connected because one provides a message another requires — never because one imported the other. Importing fit into a second flow means "the same code", not "wired to it".

Which is how the same function serves two flows, rewired and reconfigured for each:

finetune = Flow(
    "finetune",
    nodes=[
        prepare,
        augment,
        use(fit, wire={"dataset": "augmented"}, epochs=3),
        evaluate,
    ],
    inputs=[Port("lr", "float", initial=1e-4)],
    outputs=["score"],
)

use(fn, ...) is one use of a node in one flow: wire binds a port to a different message, keyword arguments override settings, and id= tells two uses of one function apart. train above is unaffected by any of it.

Sync it

fluksio sync myresearch

That imports the package, checks every declaration, and uploads each flow with a generated body per node:

# generated by fluksio sync from myresearch/train.py — edit that file instead
import sys

_REPO = '/home/you/my-research'
if _REPO not in sys.path:
    sys.path.insert(0, _REPO)

from myresearch.train import fit


def process(dataset, lr, **settings):
    return (yield from fit(dataset=dataset, lr=lr, **settings))

So the flow store still holds a complete, runnable, git-versioned definition — the body simply happens to import rather than duplicate. Your code stays in your repository, under your version control.

The upload is stamped with that repository's commit, and every run records it alongside the store's own. --dry-run prints all of this and uploads nothing; --no-publish leaves it as a draft.

!!! warning "Sync after you edit"

The engine's workers are long-lived and hold your imported modules in
memory, so an edit to `myresearch` is invisible until they are retired —
which is what every `fluksio sync` does, including one that has nothing to
upload. If you are editing many times an hour, attach your own interpreter
as a worker instead: it starts a process per call and therefore reads your
code fresh every run.

```sh
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
  --token "$WORKER_TOKEN" --labels local --python "$(which python)"
```

Then give the node `device="local"`.

!!! note "The repository path is a deployment detail"

The generated body puts your repository on `sys.path` by absolute path,
which is right for the machine you synced from and meaningless in a
container. For anything else, install the package where the node runs —
`-e /home/you/my-research` in the module manifest, or
`myresearch @ git+ssh://…@a1b2c3d`, which travels where a path does not.

Which hardware a node runs on

device picks the worker, the same way an executor does elsewhere:

@node(requires=["dataset"], provides=[...], device="gpu", device_policy="prefer")
def fit(dataset, lr, epochs=25):
    ...

device_policy="require" (the default) waits for a worker carrying that label; "prefer" runs it locally when none is attached, which is what you want while the GPU box is not switched on. See Remote workers.

What the canvas does with a synced flow

Its node bodies are generated, so the editor shows them read-only and says where the real code is. Everything else behaves as usual — but ports and settings changed there are overwritten by the next sync, which is the point of the repository being the source of truth. A node whose code you edit on the canvas makes the next sync stop and say so rather than discarding your edit; --force overrides that.

Run it

fluksio run train --lr 0.003 --seed 7
fluksio runs --flow train

run syncs first, so after an edit the command is just fluksio run — there is no step to forget. --no-sync skips it for a tight loop where nothing changed, since syncing retires the workers and the next call pays its imports again.

--lr is typed by the flow's own inputs, so 0.003 arrives as a float. A parameter you did not declare, or one of the wrong type, is refused before anything executes. It answers immediately with a queued run — training is measured in hours, so nothing waits for it unless you pass --wait.

From Python, the flow you declared is also the handle to its runs:

from myresearch.pipeline import train

run = train.submit(lr=0.003, seed=7).wait()
print(run.status, run.result)
print(run.metrics("train.loss")[-1])       # the whole series is kept
open("weights.pt", "wb").write(run.download("train.weights"))

Or over HTTP, which is what both of those are:

curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"params": {"lr": 0.003}, "seed": 7}'

curl -s $FLUKSIO/runs/<id> -H "Authorization: Bearer $TOKEN" | jq
curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $TOKEN" | jq

The run carries its parameters, a digest of them, the seed, its result, how long each node took, what it logged, every artifact it produced, and both commits — the flow store's and your repository's. That is the answer to "what was the learning rate on the run that got 94%?".

Sweep it

A grid search and an ensemble are the same call — you build the list, Fluksio runs them in parallel:

curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"runs": [
    {"params": {"lr": 0.001}, "seed": 1},
    {"params": {"lr": 0.003}, "seed": 1},
    {"params": {"lr": 0.010}, "seed": 1}
  ]}'

They share a group_id, so the sweep is GET /runs?group=…. Running them concurrently is safe because each run gets a state backend of its own — two runs of one flow cannot overwrite each other's values.

Compare the curves in one call:

curl -s "$FLUKSIO/runs/series/compare?ids=$A,$B,$C&metric=train.loss" \
  -H "Authorization: Bearer $TOKEN"

which answers in exactly the shape a chart widget draws.

What sync does, at the API level

Nothing here is privileged: sync is a client, and a flow is a document you can PUT yourself. The whole of it, for the one-node case:

"""Create the `train` flow by hand. What `fluksio sync` automates."""

import httpx

API = "http://127.0.0.1:8000/api/v1"
api = httpx.Client(base_url=API, timeout=60)
token = api.post(
    "/login/access-token",
    data={"username": "admin@example.com", "password": "k3Qm-8vTpLdX"},
).json()["access_token"]
api.headers["Authorization"] = f"Bearer {token}"

api.put("/flows/train", json={
    "name": "train",
    "title": "Model training",
    # Batch: nothing is activated, nothing fires until a run asks.
    "mode": "batch",
    "inputs": [
        {"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01},
        {"spec": {"name": "epochs", "dtype": "int"}, "initial": 50},
    ],
    "outputs": ["final_loss", "weights"],
    "nodes": [{
        "id": "train",
        "type": "python",
        "title": "Fit the model",
        # An *idle* timeout once the node streams: this is how long it may go
        # quiet, not how long it may run.
        "timeout": 600,
        "requires": [
            {"name": "lr", "dtype": "float"},
            {"name": "epochs", "dtype": "int"},
        ],
        "provides": [
            # `stream` says this port publishes repeatedly during one execution.
            {"name": "loss", "dtype": "float", "stream": True},
            {"name": "final_loss", "dtype": "float"},
            {"name": "weights", "dtype": "artifact"},
        ],
    }],
}).raise_for_status()

api.put("/flows/train/nodes/train/source",
        json={"code": open("train_node.py").read()}).raise_for_status()

version = api.get("/flows/train").json()["definition"]["version"]
api.post("/flows/train/publish", json={"version": version}).raise_for_status()

Two things worth noticing, because the decorators only move where they are said. Ports are declared, not inferred — process(lr, epochs) gets its arguments from the ports of the same name, and the types are checked on every value. And saving writes a draft; publish is what the engine picks up. That separation is what lets you edit a flow that is running.

A flow uploaded this way carries no origin, which is what tells the canvas — and the next fluksio sync — that it was not generated. See The API.

Small scripts you are just playing with

The same machinery, minus the ceremony. If what you want is "keep a record of every version of this thing I keep tweaking", note that your flows are already a git repository:

cd .fluksio/flows
git log --oneline

Every save is a commit — the node source, the parameters, the graph. A run records the commit it ran at, so git show on that hash is literally the code that produced the number. You get the versioning without adopting anything.

For quick iteration, keep the flow small (one node is fine), keep the engine running, and submit from wherever you are working:

from myresearch.pipeline import train

for lr in (0.001, 0.003, 0.01):
    train.submit(lr=lr)

A submit is around 15 ms, so calling that in a loop is a reasonable thing to do.

Get a browser onto it

The pip install gives you the engine and the API, not a web interface — a machine with no inbound route cannot serve one usefully anyway. To see the canvas, the run history and live loss curves, pair the installation with a portal, which serves the dashboard from its side:

fluksio enroll <claim-code> --portal https://hub.fluksio.com

Get the claim code from the portal under Installations → Add installation. Nothing needs to be exposed: your machine dials out and holds the connection open. See Accounts and the portal.

If you would rather stay entirely offline, run the dashboard SPA yourself from the app's Docker image — see the facility path, which is the same stack.

Where to go next

• Runs: pipelines that finish — the full picture: artifacts, sweeps, durability, what happens when your engine dies mid-training
• Writing node code — generators, settings, what a node may and may not do
• The command line — login, sync, run, runs in full
• Remote workers — send the training node to the GPU box and keep the rest on your laptop
• The flow editor — once you have a portal, this is where the graph gets easier to change than the script did