app/docs/getting-started/data-science.md

# Getting started: data science

You have a training script. It works. What you do not have is any reliable
answer to "what was the learning rate on the run that got 94%?", and the
`results_final_v3_ACTUAL.csv` in your home directory is not helping.

This page adds Fluksio to what you already have. It takes about five minutes,
installs one Python package, and does not ask you to restructure anything.

## Install

```sh
pip install fluksio
fluksio serve
```

That is the whole installation. No Docker, no database server, no ports to
open. The first run prints something like:

```text
Created the admin account admin@example.com
  password: k3Qm-8vTpLdX
  Shown once. Change it from the dashboard.
Fluksio 0.1.0 — data in /home/you/.fluksio
  API      http://127.0.0.1:8000/api/v1
  No portal. Pair this installation with:
    fluksio enroll <code> --portal https://hub.example.com
```

**Write that password down.** It is shown once and it is how you authenticate
from here on.

Everything the installation owns lives in `~/.fluksio`: a SQLite database, a
git repository holding your flows, the artifact store, and a virtual
environment your node code runs in. Move it with `--data-dir`, which is worth
doing on a cluster where `$HOME` is a network filesystem — SQLite's
write-ahead log does not work on NFS, and `fluksio serve` warns you when it
notices.

!!! tip "Keep it running"

    The engine is meant to be resident. That is the whole reason submitting a
    run costs about 15 ms instead of the second a project-bootstrapping
    orchestrator spends before it does anything. Leave it in a `tmux` window,
    or write a small `systemd --user` unit for it.

## Get a token

Everything below is the HTTP API. Grab a token once:

```sh
export FLUKSIO=http://127.0.0.1:8000/api/v1
export TOKEN=$(curl -s -X POST $FLUKSIO/login/access-token \
  -d "username=admin@example.com&password=k3Qm-8vTpLdX" | jq -r .access_token)
```

While you are experimenting, the interactive schema at
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.

## Tell it about your packages

Node code runs in `~/.fluksio/user-venv`, deliberately separate from the
environment Fluksio itself is installed in — so a pin of yours can never
collide with one of ours. That venv starts empty, so the first thing to do is
say what your script imports:

```sh
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{\"requirements\": $(jq -Rs . < requirements.txt)}"
```

It is a pip manifest, installed with `uv pip sync`, and it is versioned
alongside your flows — so what a run imported is recorded with what it ran.
Adding a package takes effect immediately; nothing restarts.

??? note "Already have a venv you would rather not duplicate?"

    Attach it as a worker instead of reinstalling into it. Mint a token, then
    point the agent at your existing interpreter:

    ```sh
    curl -X POST $FLUKSIO/workers/tokens -H "Authorization: Bearer $TOKEN" \
      -H 'Content-Type: application/json' -d '{"name": "laptop"}'

    fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
      --token "$WORKER_TOKEN" --labels local --python "$(which python)"
    ```

    Then mark the node `"device": "local"` and it runs on that interpreter. It
    is the same mechanism that sends a node to a GPU box, and it is worth
    knowing about early — see [Remote workers](../code/workers.md).

## Wrap your training script

A **flow** is a graph of nodes. A **batch flow** is one that runs on demand
from parameters to a result, which is what an experiment is. Your existing
script becomes the body of a node.

Say your script looks roughly like this:

```python
def train(lr, epochs):
    model = build_model()
    for epoch in range(epochs):
        loss = step(model, lr)
        print(f"epoch {epoch}: {loss}")     # ← this is what you are losing
    torch.save(model.state_dict(), "weights.pt")
    return loss
```

Two changes turn it into a node:

```python
"""Fit the model. A generator, so numbers escape while it is still running."""

import fluksio


def process(lr, epochs):
    model = build_model()
    for epoch in range(int(epochs)):
        loss = step(model, lr)
        yield {"loss": loss}                # ← published now, on the loss port
    torch.save(model.state_dict(), "weights.pt")
    return {
        "weights": fluksio.save_artifact("weights.pt"),
        "final_loss": loss,
    }
```

`yield` replaces `print`. Each one publishes on the node's `loss` port the
instant it happens, and the run keeps every value as a series — which is why
Fluksio has no `log_metric()` call. A metric that leaves through a port is a
message like any other: a chart can bind to it, a downstream node can consume
it, and it shows up on the canvas. A metric that escapes through a logging
function is invisible to all three.

`fluksio.save_artifact` handles the things too big to be messages — a
checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a
small reference. Nothing changes about how you write the file.

!!! note "Where a `yield` cannot reach"

    If the number comes from inside somebody else's callback — Keras, Lightning,
    HuggingFace `Trainer` — `fluksio.emit(loss=...)` writes the same port the
    same way:

    ```python
    model.fit(callbacks=[LambdaCallback(
        on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
    )])
    ```

## Create the flow

There is no scaffolding command yet, so a flow is created by PUTting its
definition. That is a fifteen-line script you run once:

```python
"""Create the `train` flow. Run once; edit it in the canvas afterwards."""

import httpx

API = "http://127.0.0.1:8000/api/v1"
api = httpx.Client(base_url=API, timeout=60)
token = api.post(
    "/login/access-token",
    data={"username": "admin@example.com", "password": "k3Qm-8vTpLdX"},
).json()["access_token"]
api.headers["Authorization"] = f"Bearer {token}"

api.put("/flows/train", json={
    "name": "train",
    "title": "Model training",
    # Batch: nothing is activated, nothing fires until a run asks.
    "mode": "batch",
    # Its inputs are the run's parameters, with the values a run gets when it
    # names none.
    "inputs": [
        {"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01},
        {"spec": {"name": "epochs", "dtype": "int"}, "initial": 50},
    ],
    # What a run reports as its result.
    "outputs": ["final_loss", "weights"],
    "nodes": [{
        "id": "train",
        "type": "python",
        "title": "Fit the model",
        # An *idle* timeout once the node streams: this is how long it may go
        # quiet, not how long it may run.
        "timeout": 600,
        "requires": [
            {"name": "lr", "dtype": "float"},
            {"name": "epochs", "dtype": "int"},
        ],
        "provides": [
            # `stream` says this port publishes repeatedly during one execution.
            {"name": "loss", "dtype": "float", "stream": True},
            {"name": "final_loss", "dtype": "float"},
            {"name": "weights", "dtype": "artifact"},
        ],
    }],
}).raise_for_status()

api.put("/flows/train/nodes/train/source",
        json={"code": open("train_node.py").read()}).raise_for_status()

version = api.get("/flows/train", params={"draft": True}).json()["definition"]["version"]
api.post("/flows/train/publish", json={"version": version}).raise_for_status()
print("published")
```

Two things worth noticing. Ports are declared, not inferred — `process(lr,
epochs)` gets its arguments from the ports of the same name, and the types are
checked on every value. And saving writes a *draft*; `publish` is what the
engine picks up. That separation is what lets you edit a flow that is running.

## Run it

```sh
curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"params": {"lr": 0.003, "epochs": 200}, "seed": 7}'
```

It answers immediately with a queued run — training is measured in hours, so
nothing waits for it. A parameter you did not declare, or one of the wrong
type, is refused with a 422 before anything executes.

Then, whenever you like:

```sh
curl -s $FLUKSIO/runs/<id> -H "Authorization: Bearer $TOKEN" | jq
curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $TOKEN" | jq
```

The run carries its parameters, a digest of them, the seed, its result, how
long each node took, what it logged, and every artifact it produced. That is
the answer to "what was the learning rate on the run that got 94%?".

## Sweep it

A grid search and an ensemble are the same call — you build the list, Fluksio
runs them in parallel:

```sh
curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"runs": [
    {"params": {"lr": 0.001}, "seed": 1},
    {"params": {"lr": 0.003}, "seed": 1},
    {"params": {"lr": 0.010}, "seed": 1}
  ]}'
```

They share a `group_id`, so the sweep is `GET /runs?group=…`. Running them
concurrently is safe because each run gets a state backend of its own — two
runs of one flow cannot overwrite each other's values.

Compare the curves in one call:

```sh
curl -s "$FLUKSIO/runs/series/compare?ids=$A,$B,$C&metric=train.loss" \
  -H "Authorization: Bearer $TOKEN"
```

which answers in exactly the shape a chart widget draws.

## Small scripts you are just playing with

The same machinery, minus the ceremony. If what you want is "keep a record of
every version of this thing I keep tweaking", note that **your flows are
already a git repository**:

```sh
cd ~/.fluksio/flows
git log --oneline
```

Every save is a commit — the node source, the parameters, the graph. A run
records the commit it ran at, so `git show` on that hash is literally the code
that produced the number. You get the versioning without adopting anything.

For quick iteration, keep the flow small (one node is fine), keep the engine
running, and submit from wherever you are working:

```python
import httpx
run = httpx.post(f"{API}/runs/flows/train", json={"params": {"lr": lr}},
                 headers=auth).json()
```

A submit is around 15 ms, so calling that in a loop is a reasonable thing to do.

## Get a browser onto it

The pip install gives you the engine and the API, not a web interface — a
machine with no inbound route cannot serve one usefully anyway. To see the
canvas, the run history and live loss curves, pair the installation with a
portal, which serves the dashboard from its side:

```sh
fluksio enroll <claim-code> --portal https://hub.fluksio.com
```

Get the claim code from the portal under **Installations → Add installation**.
Nothing needs to be exposed: your machine dials out and holds the connection
open. See [Accounts and the portal](../interface/portal.md).

If you would rather stay entirely offline, run the dashboard SPA yourself from
the app's Docker image — see the [facility path](facility-automation.md), which
is the same stack.

## Where to go next

- [Runs: pipelines that finish](../concepts/runs.md) — the full picture:
  artifacts, sweeps, durability, what happens when your engine dies mid-training
- [Writing node code](../code/nodes.md) — generators, settings, what a node may
  and may not do
- [Remote workers](../code/workers.md) — send the training node to the GPU box
  and keep the rest on your laptop
- [The flow editor](../interface/flow-editor.md) — once you have a portal, this
  is where the graph gets easier to change than the script did