app/docs/getting-started/data-science.md

Getting started: data science

You have a training script. It works. What you do not have is any reliable answer to "what was the learning rate on the run that got 94%?", and the results_final_v3_ACTUAL.csv in your home directory is not helping.

This page adds Fluksio to what you already have. It takes about five minutes, installs one Python package, and does not ask you to restructure anything.

Install

pip install fluksio
fluksio serve

That is the whole installation. No Docker, no database server, no ports to open. The first run prints something like:

Created the admin account admin@example.com
  password: k3Qm-8vTpLdX
  Shown once. Change it from the dashboard.
Fluksio 0.1.0 — data in /home/you/.fluksio
  API      http://127.0.0.1:8000/api/v1
  No portal. Pair this installation with:
    fluksio enroll <code> --portal https://hub.example.com

Write that password down. It is shown once and it is how you authenticate from here on.

Everything the installation owns lives in ~/.fluksio: a SQLite database, a git repository holding your flows, the artifact store, and a virtual environment your node code runs in. Move it with --data-dir, which is worth doing on a cluster where $HOME is a network filesystem — SQLite's write-ahead log does not work on NFS, and fluksio serve warns you when it notices.

!!! tip "Keep it running"

The engine is meant to be resident. That is the whole reason submitting a
run costs about 15 ms instead of the second a project-bootstrapping
orchestrator spends before it does anything. Leave it in a `tmux` window,
or write a small `systemd --user` unit for it.

Get a token

Everything below is the HTTP API. Grab a token once:

export FLUKSIO=http://127.0.0.1:8000/api/v1
export TOKEN=$(curl -s -X POST $FLUKSIO/login/access-token \
  -d "username=admin@example.com&password=k3Qm-8vTpLdX" | jq -r .access_token)

While you are experimenting, the interactive schema at http://127.0.0.1:8000/docs is the fastest way to see what is available.

Tell it about your packages

Node code runs in ~/.fluksio/user-venv, deliberately separate from the environment Fluksio itself is installed in — so a pin of yours can never collide with one of ours. That venv starts empty, so the first thing to do is say what your script imports:

curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{\"requirements\": $(jq -Rs . < requirements.txt)}"

It is a pip manifest, installed with uv pip sync, and it is versioned alongside your flows — so what a run imported is recorded with what it ran. Adding a package takes effect immediately; nothing restarts.

??? note "Already have a venv you would rather not duplicate?"

Attach it as a worker instead of reinstalling into it. Mint a token, then
point the agent at your existing interpreter:

```sh
curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"name": "laptop"}'

fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
  --token "$WORKER_TOKEN" --labels local --python "$(which python)"
```

Then mark the node `"device": "local"` and it runs on that interpreter. It
is the same mechanism that sends a node to a GPU box, and it is worth
knowing about early — see [Remote workers](../code/workers.md).

Wrap your training script

A flow is a graph of nodes. A batch flow is one that runs on demand from parameters to a result, which is what an experiment is. Your existing script becomes the body of a node.

Say your script looks roughly like this:

def train(lr, epochs):
    model = build_model()
    for epoch in range(epochs):
        loss = step(model, lr)
        print(f"epoch {epoch}: {loss}")     # ← this is what you are losing
    torch.save(model.state_dict(), "weights.pt")
    return loss

Two changes turn it into a node:

"""Fit the model. A generator, so numbers escape while it is still running."""

import fluksio


def process(lr, epochs):
    model = build_model()
    for epoch in range(int(epochs)):
        loss = step(model, lr)
        yield {"loss": loss}                # ← published now, on the loss port
    torch.save(model.state_dict(), "weights.pt")
    return {
        "weights": fluksio.save_artifact("weights.pt"),
        "final_loss": loss,
    }

yield replaces print. Each one publishes on the node's loss port the instant it happens, and the run keeps every value as a series — which is why Fluksio has no log_metric() call. A metric that leaves through a port is a message like any other: a chart can bind to it, a downstream node can consume it, and it shows up on the canvas. A metric that escapes through a logging function is invisible to all three.

fluksio.save_artifact handles the things too big to be messages — a checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a small reference. Nothing changes about how you write the file.

If your code does not fit in one file

The example above is a single self-contained file, which most real projects are not. If the function you want as a node imports half your repository, do not move it — install the repository into the venv the nodes run on, by adding one line to the manifest you just applied:

-e /home/you/my-research

Now the node is a wrapper over what you already have, and your code stays in your own repository, under your own version control, importing its own siblings as it always did:

"""The node. The training lives in the project, where it belongs."""

from myresearch.train import fit


def process(lr, epochs):
    return fit(lr, epochs)

A generator still works through the wrapper — yield from fit(...) — so the per-epoch metrics arrive exactly as before.

!!! warning "Apply after you edit"

The engine's workers are long-lived and hold your imported modules in
memory, so a change to `myresearch` is picked up when they are retired —
which is what **Apply** does. If you are editing many times an hour, attach
your own interpreter as a worker instead: it starts a process per call and
therefore reads your code fresh every run.

```sh
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
  --token "$WORKER_TOKEN" --labels local --python "$(which python)"
```

Then mark the node `"device": "local"`.

!!! note "Where a yield cannot reach"

If the number comes from inside somebody else's callback — Keras, Lightning,
HuggingFace `Trainer` — `fluksio.emit(loss=...)` writes the same port the
same way:

```python
model.fit(callbacks=[LambdaCallback(
    on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
)])
```

Create the flow

There is no scaffolding command yet, so a flow is created by PUTting its definition. That is a fifteen-line script you run once:

"""Create the `train` flow. Run once; edit it in the canvas afterwards."""

import httpx

API = "http://127.0.0.1:8000/api/v1"
api = httpx.Client(base_url=API, timeout=60)
token = api.post(
    "/login/access-token",
    data={"username": "admin@example.com", "password": "k3Qm-8vTpLdX"},
).json()["access_token"]
api.headers["Authorization"] = f"Bearer {token}"

api.put("/flows/train", json={
    "name": "train",
    "title": "Model training",
    # Batch: nothing is activated, nothing fires until a run asks.
    "mode": "batch",
    # Its inputs are the run's parameters, with the values a run gets when it
    # names none.
    "inputs": [
        {"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01},
        {"spec": {"name": "epochs", "dtype": "int"}, "initial": 50},
    ],
    # What a run reports as its result.
    "outputs": ["final_loss", "weights"],
    "nodes": [{
        "id": "train",
        "type": "python",
        "title": "Fit the model",
        # An *idle* timeout once the node streams: this is how long it may go
        # quiet, not how long it may run.
        "timeout": 600,
        "requires": [
            {"name": "lr", "dtype": "float"},
            {"name": "epochs", "dtype": "int"},
        ],
        "provides": [
            # `stream` says this port publishes repeatedly during one execution.
            {"name": "loss", "dtype": "float", "stream": True},
            {"name": "final_loss", "dtype": "float"},
            {"name": "weights", "dtype": "artifact"},
        ],
    }],
}).raise_for_status()

api.put("/flows/train/nodes/train/source",
        json={"code": open("train_node.py").read()}).raise_for_status()

version = api.get("/flows/train", params={"draft": True}).json()["definition"]["version"]
api.post("/flows/train/publish", json={"version": version}).raise_for_status()
print("published")

Two things worth noticing. Ports are declared, not inferred — process(lr, epochs) gets its arguments from the ports of the same name, and the types are checked on every value. And saving writes a draft; publish is what the engine picks up. That separation is what lets you edit a flow that is running.

Run it

curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"params": {"lr": 0.003, "epochs": 200}, "seed": 7}'

It answers immediately with a queued run — training is measured in hours, so nothing waits for it. A parameter you did not declare, or one of the wrong type, is refused with a 422 before anything executes.

Then, whenever you like:

curl -s $FLUKSIO/runs/<id> -H "Authorization: Bearer $TOKEN" | jq
curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $TOKEN" | jq

The run carries its parameters, a digest of them, the seed, its result, how long each node took, what it logged, and every artifact it produced. That is the answer to "what was the learning rate on the run that got 94%?".

Sweep it

A grid search and an ensemble are the same call — you build the list, Fluksio runs them in parallel:

curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"runs": [
    {"params": {"lr": 0.001}, "seed": 1},
    {"params": {"lr": 0.003}, "seed": 1},
    {"params": {"lr": 0.010}, "seed": 1}
  ]}'

They share a group_id, so the sweep is GET /runs?group=…. Running them concurrently is safe because each run gets a state backend of its own — two runs of one flow cannot overwrite each other's values.

Compare the curves in one call:

curl -s "$FLUKSIO/runs/series/compare?ids=$A,$B,$C&metric=train.loss" \
  -H "Authorization: Bearer $TOKEN"

which answers in exactly the shape a chart widget draws.

Small scripts you are just playing with

The same machinery, minus the ceremony. If what you want is "keep a record of every version of this thing I keep tweaking", note that your flows are already a git repository:

cd ~/.fluksio/flows
git log --oneline

Every save is a commit — the node source, the parameters, the graph. A run records the commit it ran at, so git show on that hash is literally the code that produced the number. You get the versioning without adopting anything.

For quick iteration, keep the flow small (one node is fine), keep the engine running, and submit from wherever you are working:

import httpx
run = httpx.post(f"{API}/runs/flows/train", json={"params": {"lr": lr}},
                 headers=auth).json()

A submit is around 15 ms, so calling that in a loop is a reasonable thing to do.

Get a browser onto it

The pip install gives you the engine and the API, not a web interface — a machine with no inbound route cannot serve one usefully anyway. To see the canvas, the run history and live loss curves, pair the installation with a portal, which serves the dashboard from its side:

fluksio enroll <claim-code> --portal https://hub.fluksio.com

Get the claim code from the portal under Installations → Add installation. Nothing needs to be exposed: your machine dials out and holds the connection open. See Accounts and the portal.

If you would rather stay entirely offline, run the dashboard SPA yourself from the app's Docker image — see the facility path, which is the same stack.

Where to go next

• Runs: pipelines that finish — the full picture: artifacts, sweeps, durability, what happens when your engine dies mid-training
• Writing node code — generators, settings, what a node may and may not do
• Remote workers — send the training node to the GPU box and keep the rest on your laptop
• The flow editor — once you have a portal, this is where the graph gets easier to change than the script did