Docs / docs (push) Successful in 21s
Playwright Tests / test-playwright (1, 2) (push) Successful in 2m13s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m52s
pre-commit / pre-commit (push) Failing after 2m6s
Test Backend / test-backend (push) Successful in 2m30s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m53s
`sync` follows each node function's imports through the project's own modules — stopping at the standard library, at anything installed, and at Fluksio itself, whose checkout would otherwise be most of every digest — and records the file list with what it hashed to. The engine hashes those files again when the run is claimed, so the fingerprint is live rather than a snapshot, and falls back to what sync recorded when it cannot see them: a remote worker's runs used to share one empty digest, and therefore one key. Three things follow. Editing a helper a node calls into re-runs that node, as before. Editing something the node never reaches no longer re-runs anything — a notebook two directories away was invalidating every arm. And `Run.code_digest` is now the hash of its nodes' digests, so it is neither looser nor tighter than "the code behind these numbers", which is what makes it worth joining an exported table on. `sync` says so too: it compares the per-node digest against the stored one, so a helper edit prints `train: updated (flow, fit)` instead of `unchanged`. The digest is read when the document is built rather than when the flow is declared, so a second `sync()` in one process sees an edit between them. Also: `fluksio runs` shows only the inputs that differ from what the flow declares, fitted to the terminal, so a flow taking a few kB of json no longer wraps every line. Every existing cache entry misses once — the fingerprint changed shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9Hdrmf2cwNABCnE5x9UJa
770 lines
29 KiB
Markdown
770 lines
29 KiB
Markdown
# Getting started: data science
|
|
|
|
You have a training script. It works. What you do not have is any reliable
|
|
answer to "what was the learning rate on the run that got 94%?", and the
|
|
`results_final_v3_ACTUAL.csv` in your home directory is not helping.
|
|
|
|
This page adds Fluksio to what you already have. It takes about five minutes,
|
|
installs one Python package, and does not ask you to restructure anything.
|
|
## Install
|
|
|
|
```sh
|
|
cd my-research
|
|
pip install fluksio
|
|
fluksio serve
|
|
```
|
|
|
|
That is the whole installation. No Docker, no database server, no ports to
|
|
open, and no login. The first run prints something like:
|
|
|
|
```text
|
|
Created the admin account admin@example.com
|
|
password: k3Qm-8vTpLdX
|
|
Shown once. Change it from the dashboard.
|
|
Fluksio 0.1.0 — data in /home/you/my-research/.fluksio
|
|
API http://127.0.0.1:8000/api/v1
|
|
Nodes /home/you/my-research/.venv/bin/python
|
|
your environment, adopted. Add packages with pip.
|
|
No portal. Pair this installation with:
|
|
fluksio enroll <code>
|
|
Signed in as admin@example.com
|
|
token in /home/you/my-research/.fluksio/client.json
|
|
```
|
|
|
|
Read the last two lines: you are already signed in. Signing in to your own
|
|
machine is a formality — the password was printed by the same process that
|
|
would have checked it — so `serve` writes the token itself and every command
|
|
below just works. `fluksio login` is for an engine somewhere *else*.
|
|
|
|
**Write that password down** anyway. It is shown once, and it is what the
|
|
dashboard asks for.
|
|
|
|
### One installation per project
|
|
|
|
`.fluksio` sits beside your code, and is found the way `.git` is — from the
|
|
directory you are standing in, or any directory above it. So two repositories
|
|
on one machine are two engines: separate flows, separate run history,
|
|
separate token, and no chance of one experiment's graph turning up in the
|
|
other's.
|
|
|
|
```text
|
|
~/research/protein-fold/
|
|
.venv/ torch, fluksio
|
|
.fluksio/ its own database, flows, artifacts, token
|
|
myresearch/
|
|
|
|
~/research/climate-sim/
|
|
.venv/ jax, fluksio
|
|
.fluksio/ its own everything
|
|
climate/
|
|
```
|
|
|
|
It holds a database and a credential, so it ignores itself from within — a
|
|
`.gitignore` of `*`, the same thing `uv` writes into `.venv`. Nothing to add
|
|
to your project's own.
|
|
|
|
Give them different ports (`--port`) if you want two running at once.
|
|
|
|
!!! tip "One engine for the machine instead"
|
|
|
|
`fluksio serve --global` uses `~/.fluksio` — shared by every directory,
|
|
which is what you want for a personal server rather than a project. When
|
|
both exist, the banner says which one you are looking at and how to reach
|
|
the other. `--data-dir` still names any directory outright, which is worth
|
|
doing on a cluster where `$HOME` is a network filesystem: SQLite's
|
|
write-ahead log does not work on NFS, and `fluksio serve` warns you when
|
|
it notices.
|
|
|
|
!!! tip "Keep it running"
|
|
|
|
The engine is meant to be resident. That is the whole reason submitting a
|
|
run costs about 15 ms instead of the second a project-bootstrapping
|
|
orchestrator spends before it does anything. Leave it in a `tmux` window,
|
|
or write a small `systemd --user` unit for it.
|
|
|
|
## Talking to it over HTTP
|
|
|
|
Everything the commands do is the HTTP API, and some of this page shows it
|
|
directly. For that, take the token `serve` already wrote:
|
|
|
|
```sh
|
|
export FLUKSIO=http://127.0.0.1:8000/api/v1
|
|
export TOKEN=$(jq -r .token .fluksio/client.json)
|
|
```
|
|
|
|
While you are experimenting, the interactive schema at
|
|
<http://127.0.0.1:8000/docs> is the fastest way to see what is available.
|
|
|
|
For an engine on another machine, `fluksio login --url https://…` asks for a
|
|
password and stores the token the same way — in this project's `.fluksio`, or
|
|
with `--global` in `~/.fluksio`.
|
|
|
|
## Your packages are already there
|
|
|
|
If you installed Fluksio into the environment you work in — the venv that
|
|
already has torch or numpy in it — that is the environment your nodes run on.
|
|
Nothing to declare, nothing to install twice:
|
|
|
|
```sh
|
|
python -m venv .venv && . .venv/bin/activate
|
|
pip install torch numpy pandas # what you were going to install anyway
|
|
pip install fluksio # and then this
|
|
fluksio serve
|
|
```
|
|
|
|
`fluksio serve` says which interpreter it settled on:
|
|
|
|
```text
|
|
Nodes /home/you/research/.venv/bin/python
|
|
your environment, adopted. Add packages with pip.
|
|
```
|
|
|
|
That venv is yours. Add a package the way you added the rest — `pip install
|
|
scikit-learn` — and `fluksio sync` (or a restart) retires the workers so they
|
|
pick it up. The Modules screen lists what is installed and stays read-only,
|
|
because the alternative would be Fluksio deciding what belongs in an
|
|
environment it did not make.
|
|
|
|
!!! note "Your pins and ours share a site-packages"
|
|
|
|
The cost of not having two environments: a package the engine depends on
|
|
is one you can now upgrade out from under it. In practice this is what
|
|
everybody does with every other tool in the venv, and the answer when it
|
|
bites is the same — pin it back, or keep Fluksio somewhere separate with
|
|
the venv of its own below.
|
|
|
|
### A venv of Fluksio's own
|
|
|
|
Sometimes you want the isolation instead: a shared installation, a container,
|
|
or an environment too precious to let a node's dependency near. Set
|
|
`NODE_VENV=managed` and Fluksio builds and owns one under the data directory:
|
|
|
|
```text
|
|
Nodes /home/you/.fluksio/user-venv/bin/python
|
|
a venv of its own; the Modules screen installs into it.
|
|
```
|
|
|
|
Then the Modules screen is how packages get in — a pip manifest, installed
|
|
with `uv pip sync` and versioned alongside your flows, so what a run imported
|
|
is recorded with what it ran:
|
|
|
|
```sh
|
|
curl -X POST $FLUKSIO/modules/apply -H "Authorization: Bearer $TOKEN" \
|
|
-H 'Content-Type: application/json' \
|
|
-d "{\"requirements\": $(jq -Rs . < requirements.txt)}"
|
|
```
|
|
|
|
Adding a package takes effect immediately; nothing restarts. The Docker image
|
|
sets `NODE_VENV=managed` for itself, because the venv in it holds the app and
|
|
nothing of yours — so a container is always this case.
|
|
|
|
!!! tip "A GPU box works the same way"
|
|
|
|
`pip install fluksio-worker` into the environment the training code runs
|
|
in, and node code runs on it: `--python` defaults to the interpreter the
|
|
agent was started with.
|
|
|
|
```sh
|
|
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
|
|
--token "$WORKER_TOKEN" --labels gpu
|
|
```
|
|
|
|
Then give the node `device="gpu"` — see [Remote workers](../code/workers.md).
|
|
|
|
## Say which functions are nodes
|
|
|
|
A **flow** is a graph of nodes. A **batch flow** is one that runs on demand
|
|
from parameters to a result, which is what an experiment is. A node is one of
|
|
your own functions — it stays in your repository, imported by its siblings as
|
|
it always was.
|
|
|
|
Say your project looks roughly like this:
|
|
|
|
```python
|
|
# myresearch/train.py
|
|
def fit(dataset, lr, epochs=25):
|
|
model = build_model()
|
|
for epoch in range(epochs):
|
|
loss = step(model, lr)
|
|
print(f"epoch {epoch}: {loss}") # ← this is what you are losing
|
|
torch.save(model.state_dict(), "weights.pt")
|
|
return loss
|
|
```
|
|
|
|
Two changes turn it into a node, and neither of them moves it:
|
|
|
|
```python
|
|
# myresearch/train.py
|
|
import fluksio
|
|
from fluksio import Port, node
|
|
|
|
|
|
@node(
|
|
requires=["dataset", Port("lr", "float")],
|
|
provides=[
|
|
Port("loss", "float", stream=True),
|
|
Port("weights", "artifact"),
|
|
Port("final_loss", "float"),
|
|
],
|
|
timeout=600,
|
|
)
|
|
def fit(dataset, lr, epochs=25):
|
|
"""Fit the model. A generator, so numbers escape while it is running."""
|
|
model = build_model(fluksio.load_artifact(dataset))
|
|
for epoch in range(epochs):
|
|
loss = step(model, lr)
|
|
yield {"loss": loss} # ← published now, on the loss port
|
|
torch.save(model.state_dict(), "weights.pt")
|
|
return {
|
|
"weights": fluksio.save_artifact("weights.pt"),
|
|
"final_loss": loss,
|
|
}
|
|
```
|
|
|
|
`yield` replaces `print`. Each one publishes on the node's `loss` port the
|
|
instant it happens, and the run keeps every value as a series — which is why
|
|
Fluksio has no `log_metric()` call. A metric that leaves through a port is a
|
|
message like any other: a chart can bind to it, a downstream node can consume
|
|
it, and it shows up on the canvas. A metric that escapes through a logging
|
|
function is invisible to all three.
|
|
|
|
`fluksio.save_artifact` handles the things too big to be messages — a
|
|
checkpoint, a dataset, a plot. It stores the bytes by their hash and returns a
|
|
small reference. Nothing changes about how you write the file.
|
|
|
|
The decorator returns the function untouched, so `fit(dataset, 0.05)` in a
|
|
test, a notebook or a debugger is exactly what it was before. Outside a run
|
|
`fluksio.emit` does nothing rather than failing, so a progress report never
|
|
stops your own code from running.
|
|
|
|
### What is declared, and what is not
|
|
|
|
- **Ports are declared, never inferred.** `requires` names the messages the
|
|
node reads; each arrives as the parameter of the same name. A bare string is
|
|
the shorthand — `"dataset"` takes its type from whoever provides it, so a
|
|
message's type is written down exactly once.
|
|
- **Parameters with a default that are not ports become settings.** `epochs`
|
|
above is stored with the flow and tunable on the canvas without touching
|
|
this file.
|
|
- **The declaration is checked against the function.** A port with no matching
|
|
parameter, or a parameter that is neither port nor setting, is an error when
|
|
the module is imported — not when the node is first called.
|
|
|
|
!!! note "Where a `yield` cannot reach"
|
|
|
|
If the number comes from inside somebody else's callback — Keras, Lightning,
|
|
HuggingFace `Trainer` — `fluksio.emit(loss=...)` writes the same port the
|
|
same way:
|
|
|
|
```python
|
|
model.fit(callbacks=[LambdaCallback(
|
|
on_epoch_end=lambda epoch, logs: fluksio.emit(loss=logs["loss"])
|
|
)])
|
|
```
|
|
|
|
### Declaring what a node needs
|
|
|
|
A training node is not like the rest of a flow. NumPy, JAX and PyTorch each
|
|
size their thread pool to every core they can see, so a few of them at once
|
|
oversubscribe the machine badly enough to starve the engine itself — the API
|
|
stops answering and every client waiting on a run dies with it. On a GPU the
|
|
same shape deadlocks: two processes each preallocating most of the card sit at
|
|
zero utilisation with nothing failing and nothing to read.
|
|
|
|
Say what one execution takes, and the engine holds it:
|
|
|
|
```python
|
|
@node(..., resources={"cpus": 2, "env": {"XLA_FLAGS": "--xla_cpu_multi_thread_eigen=false"}})
|
|
def fit(dataset, lr, epochs=25):
|
|
...
|
|
|
|
@node(..., resources={"gpus": 1, "env": {"XLA_PYTHON_CLIENT_MEM_FRACTION": "0.9"}})
|
|
def finetune(checkpoint):
|
|
...
|
|
```
|
|
|
|
Two things follow. The node waits its turn rather than starting alongside more
|
|
of itself than the machine has room for — the same waiting it already does for
|
|
a worker. And the worker it runs in is *started* with thread limits matching
|
|
what it was given, because that is the only moment a numerical library reads
|
|
them; a GPU node is told which card is its through `CUDA_VISIBLE_DEVICES`, and
|
|
nothing else is given that card while it runs.
|
|
|
|
`env` is for the tuning the engine must not invent. `XLA_FLAGS` is one composed
|
|
string whose contents depend on the version you have installed, so writing it
|
|
for you would silently replace whatever you had put there.
|
|
|
|
Declaring nothing is the default and is right for most nodes — a poll, a
|
|
threshold, a message on its way somewhere. Those share the engine's worker pool
|
|
and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops
|
|
several of them at once from each sizing to the whole box.
|
|
|
|
### Ask for a size by name
|
|
|
|
Cores and gigabytes are a property of the machines you have, and those change.
|
|
A node that names a **flavor** keeps meaning something afterwards:
|
|
|
|
```python
|
|
@node(..., resources="gpu-small")
|
|
def finetune(checkpoint):
|
|
...
|
|
|
|
@node(..., resources={"flavor": "medium", "duration_s": "2h"})
|
|
def fit(dataset, lr, epochs=25):
|
|
...
|
|
```
|
|
|
|
The flavor is read every time the node is built, so editing it changes what the
|
|
next run gets. `fluksio flavors` lists them; the Workers screen edits them.
|
|
|
|
`ram` takes `"2G"` and `duration_s` takes `"30m"`, and a flavor already says how
|
|
much — pass one or the numbers, not both. `duration_s` is a statement about the
|
|
node for whoever is planning around it, not a limit; the limit is `timeout`.
|
|
|
|
### Where it runs
|
|
|
|
This is one decision, not two. Of every machine attached — this engine and each
|
|
worker — the node goes to one that can grant what it asked for and has it free.
|
|
So a node asking for a card finds the box that has one, without naming it:
|
|
|
|
```python
|
|
@node(..., resources={"gpus": 1}) # wherever there is a card
|
|
@node(..., device="gpu", resources="gpu-small") # that box, and this much of it
|
|
```
|
|
|
|
If nothing attached can ever grant the ask, it is cut down to what is here and
|
|
runs anyway — a flow written on a cluster still has to work on a laptop. If
|
|
something *could* but is busy, the node waits and says so.
|
|
|
|
Ask what is free, and what is waiting for it, at
|
|
`GET /api/v1/workers/resources`, on the Workers screen, or in `fluksio status`.
|
|
|
|
!!! note "Accounted, not enforced"
|
|
|
|
Nothing stops a node that ignores its declaration; the numbers are
|
|
bookkeeping plus the environment its libraries read. Real enforcement —
|
|
cgroups, rlimits — is a next step, not this one.
|
|
|
|
## Say which nodes make a flow
|
|
|
|
Membership is a list, not a directory layout: the functions can live wherever
|
|
they already do.
|
|
|
|
```python
|
|
# myresearch/pipeline.py
|
|
from fluksio import Flow, Port, use
|
|
|
|
from myresearch.data import augment, prepare
|
|
from myresearch.evaluate import evaluate
|
|
from myresearch.train import fit
|
|
|
|
train = Flow(
|
|
"train",
|
|
nodes=[prepare, fit, evaluate],
|
|
inputs=[Port("lr", "float", initial=0.01)],
|
|
outputs=["score", "final_loss"],
|
|
)
|
|
```
|
|
|
|
`inputs` are the run's parameters, with the value a run gets when it names
|
|
none; `outputs` are what a run reports as its result.
|
|
|
|
Nodes are connected because one **provides** a message another **requires** —
|
|
never because one imported the other. Importing `fit` into a second flow means
|
|
"the same code", not "wired to it".
|
|
|
|
Which is how the same function serves two flows, rewired and reconfigured for
|
|
each:
|
|
|
|
```python
|
|
finetune = Flow(
|
|
"finetune",
|
|
nodes=[
|
|
prepare,
|
|
augment,
|
|
use(fit, wire={"dataset": "augmented"}, epochs=3),
|
|
evaluate,
|
|
],
|
|
inputs=[Port("lr", "float", initial=1e-4)],
|
|
outputs=["score"],
|
|
)
|
|
```
|
|
|
|
`use(fn, ...)` is one use of a node in one flow: `wire` binds a port to a
|
|
different message, keyword arguments override settings, and `id=` tells two
|
|
uses of one function apart. `train` above is unaffected by any of it.
|
|
|
|
## Sync it
|
|
|
|
```sh
|
|
fluksio sync myresearch
|
|
```
|
|
|
|
That imports the package, checks every declaration, and uploads each flow with
|
|
a generated body per node:
|
|
|
|
```python
|
|
# generated by fluksio sync from myresearch/train.py — edit that file instead
|
|
import sys
|
|
|
|
_REPO = '/home/you/my-research'
|
|
if _REPO not in sys.path:
|
|
sys.path.insert(0, _REPO)
|
|
|
|
from myresearch.train import fit
|
|
|
|
|
|
def process(dataset, lr, **settings):
|
|
return (yield from fit(dataset=dataset, lr=lr, **settings))
|
|
```
|
|
|
|
So the flow store still holds a complete, runnable, git-versioned definition —
|
|
the body simply happens to import rather than duplicate. Your code stays in
|
|
your repository, under your version control.
|
|
|
|
The upload is stamped with that repository's commit, and every run records it
|
|
alongside the store's own. `--dry-run` prints all of this and uploads nothing;
|
|
`--no-publish` leaves it as a draft.
|
|
|
|
!!! warning "Sync after you edit"
|
|
|
|
The engine's workers are long-lived and hold your imported modules in
|
|
memory, so an edit to `myresearch` is invisible until they are retired —
|
|
which is what every `fluksio sync` does, including one that has nothing to
|
|
upload. If you are editing many times an hour, attach your own interpreter
|
|
as a worker instead: it starts a process per call and therefore reads your
|
|
code fresh every run.
|
|
|
|
```sh
|
|
fluksio worker --url ws://127.0.0.1:8000/api/v1/workers/attach \
|
|
--token "$WORKER_TOKEN" --labels local --python "$(which python)"
|
|
```
|
|
|
|
Then give the node `device="local"`.
|
|
|
|
!!! note "The repository path is a deployment detail"
|
|
|
|
The generated body puts your repository on `sys.path` by absolute path,
|
|
which is right for the machine you synced from and meaningless in a
|
|
container. For anything else, install the package where the node runs —
|
|
`-e /home/you/my-research` in the module manifest, or
|
|
`myresearch @ git+ssh://…@a1b2c3d`, which travels where a path does not.
|
|
|
|
### Which hardware a node runs on
|
|
|
|
`device` picks the worker, the same way an executor does elsewhere:
|
|
|
|
```python
|
|
@node(requires=["dataset"], provides=[...], device="gpu", device_policy="prefer")
|
|
def fit(dataset, lr, epochs=25):
|
|
...
|
|
```
|
|
|
|
`device_policy="require"` (the default) waits for a worker carrying that label;
|
|
`"prefer"` runs it locally when none is attached, which is what you want while
|
|
the GPU box is not switched on. See [Remote workers](../code/workers.md).
|
|
|
|
### What the canvas does with a synced flow
|
|
|
|
Its node bodies are generated, so the editor shows them read-only and says
|
|
where the real code is. Everything else behaves as usual — but ports and
|
|
settings changed there are overwritten by the next sync, which is the point of
|
|
the repository being the source of truth. A node whose code you edit on the
|
|
canvas makes the next sync stop and say so rather than discarding your edit;
|
|
`--force` overrides that.
|
|
|
|
## Run it
|
|
|
|
```sh
|
|
fluksio run train --lr 0.003 --seed 7
|
|
fluksio runs --flow train
|
|
```
|
|
|
|
`run` syncs first, so after an edit the command is just `fluksio run` — there
|
|
is no step to forget. `--no-sync` skips it for a tight loop where nothing
|
|
changed, since syncing retires the workers and the next call pays its imports
|
|
again.
|
|
|
|
`--lr` is typed by the flow's own inputs, so `0.003` arrives as a float. A
|
|
parameter you did not declare, or one of the wrong type, is refused before
|
|
anything executes. It answers immediately with a queued run — training is
|
|
measured in hours, so nothing waits for it unless you pass `--wait`, or
|
|
`--follow`, which waits and prints the numbers as the run reports them.
|
|
|
|
No engine has to be up for any of this: `fluksio run --local` boots the
|
|
engine inside the command instead, on the same `.fluksio` — the same database,
|
|
artifacts and history a served one would use. It costs a few seconds of
|
|
startup per invocation against the ~15 ms of submitting to an engine that is
|
|
already up, so it is for the run you want now rather than the loop you are
|
|
iterating in.
|
|
|
|
A second run of a flow you did not change mostly does not execute. Each of
|
|
your nodes is fingerprinted by its settings, the ports it declares, the values
|
|
it reads, and a digest of your repository's Python files — so editing anything
|
|
your node calls into invalidates it, not only the decorated function itself.
|
|
One an earlier run already answered is restored from that run rather than run
|
|
again — reported as `cached`, so a flow with nothing left to do finishes as
|
|
`(3/3 cached)`. Change `--lr` and only the nodes downstream of it run. The
|
|
digest is blunt on purpose: an edit anywhere in the repository re-runs
|
|
everything. `--no-cache` turns caching off for one run, `@node(...,
|
|
cache=False)` for one node; the caveat and the details are in
|
|
[Stage caching](../concepts/runs.md#stage-caching).
|
|
|
|
From Python, the flow you declared is also the handle to its runs:
|
|
|
|
```python
|
|
from myresearch.pipeline import train
|
|
|
|
run = train.submit(lr=0.003, seed=7).wait()
|
|
print(run.status, run.result)
|
|
print(run.metrics("train.loss")[-1]) # the whole series is kept
|
|
open("weights.pt", "wb").write(run.download("train.weights"))
|
|
```
|
|
|
|
Or over HTTP, which is what both of those are:
|
|
|
|
```sh
|
|
curl -X POST $FLUKSIO/runs/flows/train -H "Authorization: Bearer $TOKEN" \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"params": {"lr": 0.003}, "seed": 7}'
|
|
|
|
curl -s $FLUKSIO/runs/<id> -H "Authorization: Bearer $TOKEN" | jq
|
|
curl -s "$FLUKSIO/runs/<id>/metrics?name=train.loss" -H "Authorization: Bearer $TOKEN" | jq
|
|
```
|
|
|
|
The run carries its parameters, a digest of them, the seed, its result, how
|
|
long each node took, what it logged, every artifact it produced, and both
|
|
commits — the flow store's and your repository's. That is the answer to "what
|
|
was the learning rate on the run that got 94%?". When a run failed,
|
|
`run.failures` is the node that did it, with its traceback and its logs.
|
|
|
|
Beside your commit is `code_digest`, and `fluksio runs` prints the pair as
|
|
`a1b2c3d-dirty+9f0e1a2`. The commit alone cannot identify what ran: your node
|
|
bodies are imports, so the engine executes whatever is on disk when the worker
|
|
starts, and an uncommitted tree stamps `-dirty` for every run it ever produces.
|
|
The digest is over the modules this flow's nodes actually reach — not the
|
|
whole repository, so it moves when the code behind a number moves and stays
|
|
put when a notebook beside it changes — and it is read at the moment the run
|
|
starts, so in a sweep whose runs queue for hours each one records the code
|
|
that actually executed it rather than the code that was there when you
|
|
submitted. It is what makes the column worth joining an
|
|
[exported table](../concepts/runs.md#taking-it-into-a-dataframe) on.
|
|
|
|
### When the engine is busy
|
|
|
|
A driver script outlives the engine being slow, because a sweep is hours long
|
|
and a blip is seconds:
|
|
|
|
- **Reads are repeated.** Every GET, and the writes that mean the same thing
|
|
twice, are tried up to three more times on a transport error or a `502`,
|
|
`503` or `504`, waiting 1 s, 2 s, then 4 s. `Client(retries=0)` turns that
|
|
off.
|
|
- **Submitting is safe to repeat.** Each `submit()` mints a key the engine
|
|
stores with the run, so an attempt that timed out on the way back is
|
|
answered with the run it already made rather than starting a second one. A
|
|
sweep keys every entry, so retrying a half-created one creates only what is
|
|
missing.
|
|
- **Waiting tolerates a gap.** `wait()` and `--follow` survive five failed
|
|
polls in a row before giving up. An answer is not a gap: a `404` for a run
|
|
that is gone stops immediately.
|
|
- **Timeouts are split** — 5 s to connect, 120 s to read, so a wrong URL fails
|
|
at once while a busy engine is given longer than the slowest thing it does
|
|
on purpose. `Client(timeout=…)` takes a number or an `httpx.Timeout`.
|
|
|
|
A command that still cannot reach the engine says
|
|
`fluksio: engine not answering (…)` and names the run it started, which is
|
|
running on the engine regardless — `fluksio runs` finds it again.
|
|
|
|
## Sweep it
|
|
|
|
A grid search and an ensemble are the same submission, run in parallel:
|
|
|
|
```sh
|
|
fluksio sweep train --param lr=0.001,0.003,0.01 --wait
|
|
```
|
|
|
|
Every combination of the lists you give, so a second `--param` is a grid
|
|
rather than a second sweep. Where the set you want is not a product, build the
|
|
list yourself and post it:
|
|
|
|
```sh
|
|
curl -X POST $FLUKSIO/runs/flows/train/sweep -H "Authorization: Bearer $TOKEN" \
|
|
-H 'Content-Type: application/json' -d '{"runs": [
|
|
{"params": {"lr": 0.001}, "seed": 1},
|
|
{"params": {"lr": 0.003}, "seed": 1},
|
|
{"params": {"lr": 0.010}, "seed": 1}
|
|
]}'
|
|
```
|
|
|
|
They share a `group_id`, so the sweep is `GET /runs?group=…`. Running them
|
|
concurrently is safe because each run gets a state backend of its own — two
|
|
runs of one flow cannot overwrite each other's values.
|
|
|
|
Compare the curves in one call:
|
|
|
|
```sh
|
|
curl -s "$FLUKSIO/runs/series/compare?ids=$A,$B,$C&metric=train.loss" \
|
|
-H "Authorization: Bearer $TOKEN"
|
|
```
|
|
|
|
which answers in exactly the shape a chart widget draws.
|
|
|
|
## Take it into a dataframe
|
|
|
|
The analysis itself wants a table, and `export` writes the two an analysis
|
|
asks for — the curves long, and one row per run with the parameters that
|
|
varied beside its final numbers:
|
|
|
|
```sh
|
|
fluksio export metrics --flow train --name train.loss -o curves.csv
|
|
fluksio export runs --flow train --status ok -o arms.csv
|
|
```
|
|
|
|
Both carry the run id on every row, and `arms.csv` carries the commit and the
|
|
code digest beside it, so an exported file still says what produced its
|
|
numbers. Numbers inside a record are columns of their own —
|
|
`metric.final_metrics.train_loss` — and `--metrics` and `--params` take those
|
|
dotted paths to narrow the table. Metric names are flow-qualified, so
|
|
`--list` prints the ones a selection carries. `Client.export_metrics()` and
|
|
`Client.export_runs()` answer the same rows to a notebook, ready for
|
|
`pandas.DataFrame`.
|
|
|
|
## Looking at them in the portal
|
|
|
|
None of the above needs a browser, but there is one. **Runs** is the table of
|
|
everything that has run: filter it to a flow, to a status, or to one sweep —
|
|
where it grows a column per parameter that actually varied — and tick two runs
|
|
to put their curves side by side.
|
|
|
|
A run records its metrics under the same names a dashboard binds to, so a
|
|
dashboard is another way of reading one. **Open in dashboard** carries the
|
|
runs you picked along in the URL, and every widget then answers for those runs
|
|
instead of for the live engine. If the flow has no dashboard yet, the same
|
|
menu builds one from the ports the flow declares.
|
|
|
|
See [Runs → Looking at what ran](../concepts/runs.md#looking-at-what-ran).
|
|
|
|
## What sync does, at the API level
|
|
|
|
Nothing here is privileged: `sync` is a client, and a flow is a document you
|
|
can PUT yourself. The whole of it, for the one-node case:
|
|
|
|
```python
|
|
"""Create the `train` flow by hand. What `fluksio sync` automates."""
|
|
|
|
import httpx
|
|
|
|
API = "http://127.0.0.1:8000/api/v1"
|
|
api = httpx.Client(base_url=API, timeout=60)
|
|
token = api.post(
|
|
"/login/access-token",
|
|
data={"username": "admin@example.com", "password": "k3Qm-8vTpLdX"},
|
|
).json()["access_token"]
|
|
api.headers["Authorization"] = f"Bearer {token}"
|
|
|
|
api.put("/flows/train", json={
|
|
"name": "train",
|
|
"title": "Model training",
|
|
# Batch: nothing is activated, nothing fires until a run asks.
|
|
"mode": "batch",
|
|
"inputs": [
|
|
{"spec": {"name": "lr", "dtype": "float"}, "initial": 0.01},
|
|
{"spec": {"name": "epochs", "dtype": "int"}, "initial": 50},
|
|
],
|
|
"outputs": ["final_loss", "weights"],
|
|
"nodes": [{
|
|
"id": "train",
|
|
"type": "python",
|
|
"title": "Fit the model",
|
|
# An *idle* timeout once the node streams: this is how long it may go
|
|
# quiet, not how long it may run.
|
|
"timeout": 600,
|
|
"requires": [
|
|
{"name": "lr", "dtype": "float"},
|
|
{"name": "epochs", "dtype": "int"},
|
|
],
|
|
"provides": [
|
|
# `stream` says this port publishes repeatedly during one execution.
|
|
{"name": "loss", "dtype": "float", "stream": True},
|
|
{"name": "final_loss", "dtype": "float"},
|
|
{"name": "weights", "dtype": "artifact"},
|
|
],
|
|
}],
|
|
}).raise_for_status()
|
|
|
|
api.put("/flows/train/nodes/train/source",
|
|
json={"code": open("train_node.py").read()}).raise_for_status()
|
|
|
|
version = api.get("/flows/train").json()["definition"]["version"]
|
|
api.post("/flows/train/publish", json={"version": version}).raise_for_status()
|
|
```
|
|
|
|
Two things worth noticing, because the decorators only move where they are
|
|
said. Ports are declared, not inferred — `process(lr, epochs)` gets its
|
|
arguments from the ports of the same name, and the types are checked on every
|
|
value. And saving writes a *draft*; `publish` is what the engine picks up.
|
|
That separation is what lets you edit a flow that is running.
|
|
|
|
A flow uploaded this way carries no `origin`, which is what tells the canvas —
|
|
and the next `fluksio sync` — that it was not generated. See
|
|
[The API](../code/api.md).
|
|
|
|
## Small scripts you are just playing with
|
|
|
|
The same machinery, minus the ceremony. If what you want is "keep a record of
|
|
every version of this thing I keep tweaking", note that **your flows are
|
|
already a git repository**:
|
|
|
|
```sh
|
|
cd .fluksio/flows
|
|
git log --oneline
|
|
```
|
|
|
|
Every save is a commit — the node source, the parameters, the graph. A run
|
|
records the commit it ran at, so `git show` on that hash is literally the code
|
|
that produced the number. You get the versioning without adopting anything.
|
|
|
|
For quick iteration, keep the flow small (one node is fine), keep the engine
|
|
running, and submit from wherever you are working:
|
|
|
|
```python
|
|
from myresearch.pipeline import train
|
|
|
|
for lr in (0.001, 0.003, 0.01):
|
|
train.submit(lr=lr)
|
|
```
|
|
|
|
A submit is around 15 ms, so calling that in a loop is a reasonable thing to do.
|
|
|
|
## Get a browser onto it
|
|
|
|
The pip install gives you the engine and the API, not a web interface — a
|
|
machine with no inbound route cannot serve one usefully anyway. To see the
|
|
canvas, the run history and live loss curves, pair the installation with a
|
|
portal, which serves the dashboard from its side:
|
|
|
|
```sh
|
|
fluksio enroll <claim-code> --portal https://hub.fluksio.com
|
|
```
|
|
|
|
Get the claim code from the portal under **Installations → Add installation**.
|
|
Nothing needs to be exposed: your machine dials out and holds the connection
|
|
open. See [Accounts and the portal](../interface/portal.md).
|
|
|
|
If you would rather stay entirely offline, run the dashboard SPA yourself from
|
|
the app's Docker image — see the [facility path](facility-automation.md), which
|
|
is the same stack.
|
|
|
|
## Where to go next
|
|
|
|
- [Runs: pipelines that finish](../concepts/runs.md) — the full picture:
|
|
artifacts, sweeps, durability, what happens when your engine dies mid-training
|
|
- [Writing node code](../code/nodes.md) — generators, settings, what a node may
|
|
and may not do
|
|
- [The command line](../code/cli.md) — `login`, `sync`, `run`, `runs`, `sweep`
|
|
in full
|
|
- [Remote workers](../code/workers.md) — send the training node to the GPU box
|
|
and keep the rest on your laptop
|
|
- [The flow editor](../interface/flow-editor.md) — once you have a portal, this
|
|
is where the graph gets easier to change than the script did
|