Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs

Browse Source
This commit is contained in:
stroblme
2026-10-13 10:02:20 +02:00
parent a93f865e91
commit 80e595c02e
24 changed files with 2210 additions and 4310 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/code/cli.md
+221 -468
View File
@@ -1,467 +1,261 @@
# The `fluksio` command
Install on Linux or macOS with Python 3.12 or newer:
```sh
pip install fluksio
```
Installs the engine and the `fluksio` command. Python 3.12 or newer, Linux or
macOS.
The MQTT and InfluxDB connectors, outbound mail and error reporting are
`pip install 'fluksio[server]'`: a deployment talking to devices wants them,
and a laptop waiting on them to install does not. Everything else, the engine
and every python node included, is in the plain install; a node type whose
library is missing says which extra to add when one is actually built.
There is a second, smaller distribution, `fluksio-worker`, for a machine that
should only *run nodes* for an engine elsewhere. It has none of the engine in
it. See [Remote workers](workers.md).
The command is two things at once: `serve`, `enroll` and `worker` *are* an
instance, while `login`, `sync`, `run`, `runs`, `artifacts`, `sweep` and
`status` talk to one that may be anywhere.
## Where an instance lives
`.fluksio` beside your code, found the way `.git` is: from the working
directory, or any directory above it. Two repositories on one machine are
therefore two engines, with their own flows, runs and token. `fluksio serve`
makes one where there is none, and it ignores itself from within with a
`.gitignore` of `*`, so a database and a credential cannot be committed by
accident.
`--global` uses `~/.fluksio` instead, shared by every directory. `--data-dir`
(or `FLUKSIO_HOME`) names any directory outright and wins over both.
## `fluksio serve`
Runs the engine.
```sh
fluksio serve
```
On the first start it creates an admin account and prints its password **once**.
Nothing else has to be running: no database server, no message broker, no
Docker.
For the browser dashboard, sign up at [fluksio.com](https://fluksio.com), add
an instance, then run `fluksio enroll <code>` in the same directory.
At a terminal this opens a dashboard with the engine running under it; see
[below](#the-dashboard). `--plain` prints the log stream instead, which is
also what happens with no terminal: in a container, under systemd, or in CI.
Use `fluksio --help` or `fluksio COMMAND --help` for command options.
`pip install 'fluksio[server]'` adds MQTT, InfluxDB, outbound mail and error
reporting. A machine that only executes remote nodes can install
[`fluksio-worker`](workers.md).
The default port moves out of the way when something already has it (8001,
8002, and so on) and says which one it took; the URL written to
`client.json` is the one it is actually on. A port you *asked* for is never
moved off: `--port 9000` on a taken 9000 fails, because something else is
there and you named it. It fails before `client.json` is written, so every
client keeps talking to the engine that is actually up.
## Where an instance lives
What it will *not* do is start a second engine for the same instance. Before
anything else, `serve` looks for an engine already serving this directory —
the pidfile beside the data says where, and a token this directory's key
signed says whether what answers there is ours — and stops if it finds one,
whatever port the second was asked for. One SQLite database wants one engine,
and the second would repoint `client.json` at a port that dies with it.
Another instance's Fluksio on the port is named, and the move happens as usual.
By default, Fluksio finds the nearest `.fluksio/` directory by walking up from
your working directory. `serve` creates one in the current directory if needed.
Separate project directories can therefore hold separate instances.
| Option | Default | What it does |
|---|---|---|
| `--data-dir PATH` | `./.fluksio` (or `$FLUKSIO_HOME`) | where this instance keeps everything |
| `--host HOST` | `127.0.0.1` | what to bind |
| `--port PORT` | `8000`, or the next free one | what to listen on |
| `--plain` | off at a terminal | the log stream rather than the dashboard |
| `--log-level LEVEL` | `info` | uvicorn's log level |
| `--admin-email ADDR` | `admin@example.com` | the account created on first run |
| `--admin-password PW` | generated | set it instead of having one generated |
| `--enroll CODE` | — | pair with a portal as part of coming up |
| `--portal URL` | — | the portal `--enroll` redeems at |
| `--max-runs N` | 4 | batch runs driven at once (`FLOW_MAX_RUNS`) |
| `--max-cascades N` | 4 | cascades in flight at once (`FLOW_MAX_CASCADES`) |
| `--max-workers N` | 4 | python worker processes (`FLOW_MAX_WORKERS`) |
| `--gpus N` | counted | GPUs on this machine a node may be given (`FLOW_GPUS`) |
| `--ram MB` | worked out | memory on this machine a node may be given (`FLOW_RAM`) |
| `--worker-max-rss MB` | 0 (never) | retire a warm worker holding more when its node returns (`FLOW_WORKER_MAX_RSS_MB`) |
`--global` selects `~/.fluksio`. `--data-dir PATH` or `FLUKSIO_HOME`
selects a directory explicitly. The data directory contains a `.gitignore`
that excludes its database, credentials and other contents from your project.
Cards are counted from NVIDIA's device nodes (`/dev/nvidia0`, `/dev/nvidia1`, …),
which asks no vendor tool and so keeps the one dependency from becoming two.
Anything they do not cover — another vendor, or a card this process cannot see
— reports none until `--gpus` says otherwise, and a node asking for one is then
clamped to zero and runs alongside every other. `--gpus 1` is what serialises
them. Passing a number always wins over the count.
## `fluksio serve`
`--enroll` with `--portal` is the one-command setup: it pairs before the engine
starts, so the connection is dialled as part of coming up rather than needing a
restart. It is skipped if the instance is already enrolled.
Starts the API and flow engine. No external database, broker or Docker is
needed for the default setup.
!!! warning "One process"
`fluksio serve` holds the flow engine. A second one is a *second engine*:
duplicated subscriptions, duplicated cron ticks, two webhooks answering the
same path. Run one, and distribute work with
[workers](workers.md) instead.
!!! note "`$HOME` on a cluster"
A login node's home directory is often NFS, where SQLite's write-ahead log
does not work, since the database would be locked or corrupt. `fluksio serve`
warns when it notices; point `--data-dir` at local disk.
### What it prints
```text
Created the admin account admin@example.com
password: example-password
Shown once. Change it from the dashboard.
Fluksio 0.1.0 — data in /home/you/.fluksio
API http://127.0.0.1:8000/api/v1
No portal. Pair this instance with:
fluksio enroll <code>
```sh
fluksio serve
fluksio serve --plain --port 9000
```
An enrolled instance says which portal it is on instead, and notes that the
dashboard is served from there rather than here.
The first start prints an admin password once. Local CLI commands receive a
saved token automatically; `login` is for connecting to another engine.
The default port is 8000, advancing to the next free port if necessary.
An explicit `--port` fails when occupied. The startup banner and saved client
configuration show the selected URL.
Run one engine per instance. At a terminal, a second `serve` reattaches to
that instance's running engine; plain mode refuses to start a duplicate.
| Option | Default | Purpose |
|---|---|---|
| `--data-dir PATH` | Nearest `.fluksio`, or a new one here | Instance storage |
| `--global` | Off | Use `~/.fluksio` |
| `--host HOST` | `127.0.0.1` | Bind address |
| `--port PORT` | `8000`, or next free | Listening port |
| `--plain` | Off at a terminal | Print logs instead of opening the terminal dashboard |
| `--log-level LEVEL` | `info` | Uvicorn log level |
| `--admin-email ADDR` | `admin@example.com` | First admin account |
| `--admin-password PW` | Generated | First admin password |
| `--enroll CODE` | Unset | Enroll before starting, if not already paired |
| `--portal URL` | `https://hub.fluksio.com` | Portal used by `--enroll` |
| `--max-runs N` | `4` | Concurrent batch runs |
| `--max-cascades N` | `4` | Concurrent live cascades |
| `--max-workers N` | `4` | Local Python worker processes |
| `--gpus N` | Auto-detected | Available local GPUs |
| `--ram MB` | Auto-detected | Available local memory |
| `--worker-max-rss MB` | `0` (disabled) | Retire a worker above this resident memory after its call |
Resource defaults can also come from the corresponding `FLOW_*` settings;
see [Configuration](../reference/configuration.md). GPU detection uses
NVIDIA device nodes. Set `--gpus` when your hardware is not detected.
Use local disk for the data directory on a cluster: SQLite's write-ahead log
does not support NFS.
### The dashboard
At a terminal, `serve` opens three tabs: **Overview** is the health block and
the flows, **Runs** is the history, and **Logs** is the engine's own output.
At a terminal, `serve` opens **Overview**, **Runs** and **Logs**:
| Key | What it does |
| Key | Action |
|---|---|
| `1` `2` `3` | the Overview, the Runs and the Logs |
| `q` | close the dashboard. **The engine keeps running**, and the pid is printed |
| `s` | stop the engine, or start it again |
| `r` | restart it |
| `e` | pair with a portal, without leaving the screen |
| `1`, `2`, `3` | Switch tabs |
| `q` | Close the dashboard; the engine keeps running |
| `s` | Stop or start the engine |
| `r` | Restart the engine |
| `e` | Enroll with a portal |
On the Runs tab the toolbar carries what a run is for:
On **Runs**, `space` selects runs, `enter` opens a chart or comparison,
`c` cancels a run, and `a` lists or downloads artifacts. Compare curves by
step, elapsed time or another metric.
| Key | What it does |
|---|---|
| `space` | tick the run under the cursor into a comparison |
| `enter` | compare the ticked runs, or draw the one under the cursor |
| `c` | cancel the run the cursor is on |
| `a` | list what the run left behind, and fetch it |
The engine is a child process rather than a thread, which is what makes those
possible, and what makes `q` a way out of the screen rather than a way to
stop the engine. Running `fluksio serve` again reattaches to it.
Its output goes to `serve.log` in the data directory rather than down a pipe,
which is what lets the screen be closed while the engine keeps running — a
pipe with nobody reading it breaks the next line the engine writes, and a
node's `print` is one of those. It also means the Logs tab has the output of
an engine this screen only adopted, and the scrollback of the one before it.
The engine cuts the file back to nothing once it passes 5 MB, so the bound
holds whether or not a screen is open.
The screen takes its colours from the terminal rather than painting its own,
so it sits inside a light profile as readily as a dark one.
An engine started elsewhere is adopted rather than duplicated, and can be
stopped from here only when it is this instance's own: both the pidfile
beside the data and a token this directory's key signed have to agree. Another
instance's engine is named and left alone.
The screen subscribes to the engine's event bus over the same websocket a
browser uses, so a run appears the moment it starts. It reconnects on its own,
and an engine stopped from this screen is a normal state rather than an error.
#### Comparing runs
`enter` opens a comparison of the ticked runs: one metric's curve for each,
over a table of what differs between them.
The curves are drawn in braille, five to a chart, in five distinct colours: a
departure from the browser's chart palette, where the five series
step one hue by lightness. Hue is what survives a terminal. Every curve is
named in the legend either way.
Two pickers sit above the chart. The first is the metric, named as the run
records it (`train.loss` rather than `loss`, since a name is qualified by the
node that published it). The second is what it is plotted against: the step,
`time (s)` measured from each run's own first reading so runs started hours
apart lie on top of each other, or another metric of the same runs.
The table below keeps only what actually differs (the inputs whose values are
not the same in every run, and the seed when it varies) beside each run's
status, duration and final numbers. A parameter every run shared is not a
column worth scanning.
A run that is still going has its curve re-read once a second, and the whole
comparison refreshes when the engine says a run finished.
Run `fluksio serve` again to reopen the dashboard. Its engine writes to
`serve.log` in the data directory; the log is truncated after 5 MB.
Without a terminal, or with `--plain`, the command prints logs directly.
## `fluksio enroll`
Pairs an existing instance with a portal.
Connects this instance to the portal:
```sh
fluksio enroll ABCD-1234
```
| Option | What it does |
Get a code from **Instances → Add instance** in the portal. Run the command
from the instance's project directory, or select it explicitly.
| Option | Purpose |
|---|---|
| `--portal URL` | a portal of your own, instead of `https://hub.fluksio.com` |
| `--as EMAIL` | the local account a portal session arrives as |
| `--data-dir PATH` | which instance, if not the one this directory is in |
| `--portal URL` | Use another portal instead of `https://hub.fluksio.com` |
| `--as EMAIL` | Choose the local superuser represented by portal sessions |
| `--data-dir PATH` | Select instance storage |
| `--global` | Select `~/.fluksio` |
Get the code from the portal under **Instances → Add instance**. It is
single-use and expires in fifteen minutes. `--as` matters when the instance
has several superusers. Without it, enrolment refuses rather than guessing.
With multiple local superusers, `--as` is required. A new code replaces an
existing enrollment only after the portal accepts it. A running engine picks
up the change and reconnects automatically.
An instance that is already paired is re-paired: a new claim code replaces the
connection, at the same portal or another one, and a running engine drops the
old tunnel within a few seconds. The code is redeemed before anything is
written, so one the portal rejects leaves the connection as it was. Moving to
a different portal clears the local accounts' mappings to the old one, since
nothing in a mapping says which portal issued it — the people who had access
are admitted again from the new portal.
Afterwards, `fluksio serve` dials the portal as it comes up, and keeps dialling:
a portal that restarts, a wifi that changes, a laptop that suspends and wakes
somewhere else all end the same connection, and the link is put back up without
anybody noticing. A connection that stood up and then dropped is retried at
once; one that never stood up waits a little longer each time, up to half a
minute. See [Accounts and the portal](../interface/portal.md).
The instance keeps an outbound connection to the portal and reconnects after
network interruptions. See [Accounts and the portal](../interface/portal.md).
## `fluksio worker`
Runs nodes for an engine elsewhere. Everything after `worker` belongs to the
agent's own parser: it is the same program `fluksio-worker` installs, so the
two are interchangeable:
Starts the same agent as the standalone `fluksio-worker` command:
```sh
fluksio worker --url wss://api.${DOMAIN}/api/v1/workers/attach --labels gpu
FLUKSIO_WORKER_TOKEN=your-token fluksio worker \
--url wss://api.example.com/api/v1/workers/attach --labels gpu
```
See [Remote workers](workers.md).
Create the worker token on the engine first. See [Remote workers](workers.md)
for token creation, interpreter selection and labels.
## Talking to an engine
The commands below are the client half: they run wherever you work, and
address an engine over its API rather than being one, except under `--local`,
which boots one inside the command instead.
Client commands use the nearest `.fluksio/client.json`, or the global
configuration when there is no project instance. Legacy client configuration
is a fallback.
`FLUKSIO_URL` and `FLUKSIO_TOKEN` override saved values; command options
`--url` and `--token` override both.
### `fluksio login`
```sh
fluksio login --url https://api.${DOMAIN}
fluksio login --url https://api.example.com
```
For an engine somewhere *else*. One you started yourself needs no login:
`fluksio serve` writes the token as it comes up and says where it put it.
The token goes in this project's `.fluksio/client.json`, or with `--global` in
`~/.fluksio/client.json`. Every command below reads it from there, nearest
first walking up from the working directory, or from `FLUKSIO_URL` and
`FLUKSIO_TOKEN`, or from its own `--url` and `--token`. A token an older
version wrote to `~/.config/fluksio/client.json` is still read.
Prompts for credentials and saves an API token for a remote engine.
`--global` saves it in `~/.fluksio` instead of this project. Your locally
started engine already saves a token.
### `fluksio sync`
```sh
fluksio sync [PATH_OR_MODULE ...] # default: the current directory
fluksio sync study
fluksio sync examples/myresearch
fluksio sync --dry-run study
```
Imports what you name, collects the flows the decorators declared, and uploads
each one with a generated import shim per node. A directory that is a package
is walked; a dotted name is imported as it stands; nothing is loaded from a
file path, because the shim has to import the same way.
Imports modules, packages or directories and uploads the declared flows.
With no target, it searches the current directory. Directory searches skip
hidden folders, virtual environments, `__pycache__` and `node_modules`.
File paths are not targets: use `study`, not `study.py`.
A plain directory is walked all the way down, so one folder per study
(`fluksio sync dev` over `dev/s1_baseline/study.py`) needs no naming. Hidden
directories, `__pycache__`, `node_modules` and virtualenvs are left alone.
The generated node bodies import your functions. Keep the project available
to each machine running them; sync does not copy your package to a remote
worker. Uploading source requires a superuser token.
Each file is imported under the name its path spells beneath the directory
being synced, so `dev/s1/study.py` and `dev/s2/study.py` are `s1.study` and
`s2.study` and a `study.py` per study collides with nothing. No `__init__.py`
is needed, since the directories in between are namespace packages, which leaves a
bare `from study import ...` in a test beside it working. A file at the top of
what is synced keeps its plain name.
| Flag | What it does |
| Flag | Purpose |
|---|---|
| `--dry-run` | print the flow documents and shims, upload nothing |
| `--no-publish` | leave the upload as a draft |
| `--force` | overwrite a flow, or a node body, that was edited on the canvas |
| `--dry-run` | Print flow documents and node bodies without uploading |
| `--no-publish` | Save changes as drafts |
| `--force` | Overwrite canvas edits that conflict with your local definitions |
Every sync retires the engine's workers, including one that had nothing to
upload: a worker holds your package in memory, so an edit to it is invisible
until the process goes.
It also records, per node, which of your modules that node's function imports
its way to, and what they hash to. That is what the
[stage cache](../concepts/runs.md#stage-caching) keys on, so editing a
helper a node calls into is reported as that node changing (`train: updated
(flow, fit)`) and re-runs it, while editing something the node never reaches
is left alone.
Uploading node code takes a superuser's token; anyone else's sync is refused.
A sync that changed nothing says `unchanged`, which is the answer worth
having. `published` and `draft` are said only when there was something to
publish or a draft was genuinely left behind. An engine too old to store what
a node's code reaches says so in a line naming both versions; until it is
upgraded its cache is keyed on the whole repository, as it was before. See
[Getting started: data science](../getting-started/data-science.md).
Sync records source digests for [stage caching](../concepts/runs.md#stage-caching)
and refreshes local workers, so edited imports are picked up. An unchanged
flow reports `unchanged`. See the
[data science guide](../getting-started/data-science.md) for a complete example.
### `fluksio run`
```sh
fluksio run train --lr 0.05 --seed 7 [--wait]
fluksio run train --lr 0.05 --epochs 20 --follow
```
Syncs the flow, then submits a run, so the command after an edit is this one
and nothing else, from the repository root as readily as from the study's own
folder. What it imports is the module that declared the flow, as the last
`sync` recorded it, under the same module name. Importing a module runs it, so
the other scripts in the repository are left alone.
Syncs local code, then submits a run. Flags such as `--lr` name the flow's
declared inputs and are converted to their declared types. Unknown inputs are
rejected. Use `--param name=a,b` with `sweep` for multiple values.
A flow the engine has no such record for (never synced, drawn on the canvas,
or on an engine older than the record) syncs the working directory *and
everything under it* instead, skipping whatever the repository's `.gitignore`
covers. A study there that will not import is a warning rather than a stopped
run. `--sync dev/s1_baseline` (repeatable) names what to sync outright, and
`--no-sync` skips it entirely.
After the first sync, `run` imports the module that declared the flow. If no
declaration is recorded, it searches the working directory and skips paths
covered by `.gitignore`. Import errors in that fallback search produce
warnings. Use `--sync PATH` to choose a target explicitly.
Flags that are not its own are the flow's inputs, typed by what the flow
declares them as, so a name the flow has not got is refused by name, and
`--param lr=0.002` is told that one value is `--lr 0.002` and several is a
[sweep](#fluksio-sweep). `--wait` blocks until the run
finishes and exits non-zero if it failed. `--follow` waits as well, and prints
the numbers the run reports as they arrive:
| Flag | Purpose |
|---|---|
| `--wait` | Wait for completion; exit non-zero if the run fails |
| `--follow` | Wait and print emitted metrics |
| `--timeout SECONDS` | Stop waiting after this long; leave the run active |
| `--seed N` | Record the seed and fill a declared input named `seed` |
| `--defaults` | Use declared input defaults without prompting |
| `--sync PATH` | Choose what to sync; repeat for multiple targets |
| `--no-sync` | Run the version already on the engine |
| `--no-cache` | Execute every node |
| `--local` | Start an engine in this command and wait for the result |
```text
train.loss[14] = 3.40295e-06
```
With no parameters at a terminal, the command prompts for each input.
Press Enter to keep its default. Scripted commands do not prompt.
Ctrl-C while waiting cancels the run and exits with status 130.
Ctrl-C while either is waiting cancels the run on the engine rather than only
stopping the watching, and exits 130. `--timeout SECONDS` gives up *waiting*
after that long and leaves the run going; it is nothing to do with a node's own
timeout.
A seed does not initialize your libraries' random number generators. Declare
a `seed` input and pass it to your code if it controls randomness.
`--seed N` is the experiment's seed, and it does three things. It is recorded
on the run, so what a result came from is answerable later. It goes into the
digest that identifies a run's inputs, so two runs of one configuration with
different seeds are different runs rather than a cache hit. And if the flow
declares an input named `seed`, that is what fills it, so the number the run
is labelled with is the number your code actually drew from, instead of merely
looking like it. A flow that declares no such input still records it, and
nothing reads it. Sweep over seeds with `--param seed=1,2,3`.
Any input takes what a previous run produced, named rather than typed out: a
checkpoint, but equally a config object nobody wants to paste into a shell:
Inputs can reference earlier results:
```sh
fluksio run evaluate --dataset @run:1758042000123-9f2ab41c.dataset
fluksio run evaluate --dataset sha256:6dd1f0…
fluksio run train --meta @run:1758042000123-9f2ab41c.dataset_meta
fluksio run evaluate --dataset @run:<run-id>.dataset
fluksio run evaluate --dataset sha256:<digest>
```
`@run:<id>.<output>` is whatever that run's output was, whole and with its own
type; a bare digest is the content itself, resolved into a reference. Both
spellings are reserved on every input, `str` included, so an input that has to
carry one of them literally cannot. Passing the value as JSON still works and
is what a script that already holds one does, the same thing
`flow.submit(dataset=run.result["dataset"])` does from Python.
`@run:<id>.<output>` resolves a run output; `sha256:<digest>` resolves
stored bytes into an artifact reference.
Run a flow with no parameters at a terminal and it asks for them, one line per
declared input, with the declared value in brackets:
```text
lr (float) [0.05]: 0.01
epochs (int) [10]:
dataset (artifact): @run:1758042000123-9f2ab41c.dataset
```
Enter keeps what is in brackets, so pressing it through the lot runs the
defaults. Nothing changes for a scripted run: passing any parameter, or piping
the command, skips the questions, and `--defaults` skips them explicitly.
`--no-sync` runs what is already on the engine. Worth it in a tight loop where
you know nothing changed, since syncing retires the workers and the next call
pays its imports again. A directory that declares no flows syncs nothing and
says nothing: a flow drawn on the canvas is run the same way.
`--no-cache` executes every node, including one an earlier run already
answered. See [Stage caching](../concepts/runs.md#stage-caching).
`--local` boots the engine inside this process instead of talking to a served
one, so there is no `fluksio serve` terminal to keep open. It is the same
instance either way (the same `.fluksio`, the same database, artifacts and run
history) so a run made this way and a run made through a served
engine cache against each other. It always waits, because the engine it starts
lives exactly as long as the command. Starting one costs a few seconds of
worker pool and module reconcile, against the ~15 ms of submitting to an
engine that is already up: `--local` is for "I just want to run it", not for a
loop you are iterating in.
`--local` uses the same data directory, history and artifacts as `serve`.
It starts and stops the engine for each command, so keep `serve` running
when iterating frequently.
### `fluksio status`
```sh
fluksio status [--watch]
fluksio status
fluksio status --watch
```
The home screen's top half in a terminal: whether the engine is healthy and
what is wrong if not, whether it is paired with a portal and reaching it, then
every flow with its state, its node count and whether it has unpublished
changes, and the last few runs and failures under them.
A `resources` line names each machine the engine can run a node on and how much
of it is in use, plus how many nodes are queued for one. It is absent on an
engine that accounts for nothing.
The portal reads one of three ways. `no portal` means this instance was
never enrolled. `portal hub.fluksio.com` means the link is up. `portal
unreachable` names the error, and is the one worth acting on: the dashboard is
served from the other end, so nobody can reach it while that is showing.
`--watch` keeps it on screen and refreshes every five seconds until Ctrl-C, the
cadence the dashboard polls at, since nothing here moves faster. It needs a
terminal; without one, run it without `--watch` and the output pipes cleanly.
`--local` reads the flows and history out of this directory with no engine
served, the same way `runs --local` does. It cannot be watched: an in-process
engine is the command itself, so nothing changes under it.
Shows engine health, portal connectivity, flows, recent runs and available
resources. `--watch` refreshes every five seconds. `--local` reads the local
instance without a served engine.
### `fluksio runs`
```sh
fluksio runs [--flow train] [--limit 20]
fluksio runs --flow train --limit 10
```
The runs an engine has recorded, newest first: id, status, flow, duration, how
long ago it was submitted, the commit of the repository it came from, and the
inputs it was given. Statuses are coloured when a terminal is reading the
output: `ok` green, `error` red, `cached` cyan.
Only the inputs that *differ from what the flow declares* are shown, and they
are clamped to what is left of the terminal's width. A run that took the
defaults lists none at all, and a flow taking a few kB of JSON does not push
everything else off the line. `Client.runs()` and
[`fluksio export runs`](#fluksio-export) are where the whole value is read.
`--local` reads the same history from an in-process engine, without one having
to be served.
Lists recent runs, newest first. Omit `--flow` for all flows.
`--local` reads the local instance.
### `fluksio retry`
```sh
fluksio retry <run-id> [<run-id> ...]
fluksio retry <run-id>
fluksio retry --group <sweep-id>
```
Runs the same thing again as a run of its own, carrying the flow, the inputs,
the seed and the group of the one it repeats, and recording it as its parent.
What it is for is a run an engine restart interrupted: those are marked
`abandoned` once their lease goes stale, and `--group` retries every run of a
sweep that did not end `ok`, so the missing combinations are filled in rather
than the whole grid being submitted again. The stage cache is what makes it
cheap — the nodes that finished are restored rather than recomputed.
Submits a new run with the previous inputs, seed and group, linked to its
parent. `--group` retries finished runs in that group that did not end
`ok`. Active runs must be cancelled before retrying.
A run that has not finished is refused; cancel it first.
Runs that were still *queued* when an engine stopped need none of this: the
next engine reads them back out of its own history and picks them up.
Queued runs survive engine restarts. Interrupted running runs become
`abandoned` after their lease expires; retry them to resume with any
available cached stages.
### `fluksio flavors`
@@ -469,137 +263,96 @@ next engine reads them back out of its own history and picks them up.
fluksio flavors
```
The named sizes a node can ask for, as in `@node(resources="gpu-small")`, with the
cores, memory and cards each stands for. Editing them is the Workers screen or
`POST /api/v1/flavors`; this is the read.
Lists named resource sizes, usable as `@node(resources="gpu-small")`.
Edit them in **Workers** or through the flavors API.
### `fluksio sweep`
```sh
fluksio sweep train --param lr=0.1,0.01 --param epochs=10,50 --wait
fluksio sweep train --param lr=0.01,0.1 --param epochs=10,50 --wait
```
Every combination of the parameter lists, submitted as one group: four runs
above, sharing a `group_id` and executing in parallel. Values are typed by the
flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`,
`--no-cache` and `--local` mean what they do there. `--wait` blocks until all
of them are finished and exits non-zero if any failed.
Submits every combination of the declared input values: four runs here,
sharing a `group_id`. Runs execute concurrently within the engine's limits.
To vary seeds, the flow must declare a `seed` input; use
`--param seed=1,2,3`.
`--seed`, `--sync`, `--no-sync`, `--no-cache`, `--timeout` and
`--local` work as for `run`. `--wait` exits non-zero if any run fails.
### `fluksio export`
```sh
fluksio export metrics --flow train --list
fluksio export metrics --flow train --name train.train_loss --stride 10 -o curves.csv
fluksio export runs --flow train --status ok > arms.csv
fluksio export metrics --flow train --name train.loss -o curves.csv
fluksio export runs --flow train --status ok -o results.csv
```
The two tables an analysis reads. `export metrics` is the long one, a row per
run, metric and step, which is what a plotting library takes without
reshaping; `--name` keeps the metrics it lists and `--stride` keeps every Nth
point of *each* curve. `export runs` is the wide one: a row per run with its
inputs as columns, its final numbers, its status, its duration and the commit
and digest of the code it ran.
`metrics` exports a row per run, metric and step. `--list` lists metric
names; `--name A,B` selects them; `--stride N` keeps every Nth point.
Every input the selected runs recorded becomes a column, so the schema does
not move with the selection and a filter written against one export keeps
working on the next; `--params lr,seed` narrows it to the axis a comparison is
read along. `--metrics` narrows the final numbers the same way.
`runs` exports a row per run with inputs, final numeric results, status,
duration and code identifiers. Use `--params A,B` and `--metrics A,B` to
select columns, including dotted paths such as `final_metrics.train_loss`.
An input left out of a submit is recorded at the value the flow declares for
it, so every row says what it was actually run with rather than leaving the
cell blank.
Both support `--flow`, repeated `--run ID`, `--group`, `--status`,
`--since`, `--until` and `--local`. The run ID joins results to metrics.
A node usually returns a record rather than a scalar, so both sides take
dotted paths into one: `--metrics final_metrics.train_loss,test_metrics.known.perfect`
selects three fields rather than two blobs, and `--params model.ansatz` does
the same for an input. The defaults reach the same depth: every number a
result carries becomes a column wherever it sits, and a record's inputs are
taken leaf by leaf rather than as one blob.
Output defaults to CSV on stdout. `-o FILE` writes a file; `--format jsonl`
writes JSON Lines. Parquet requires `pip install 'fluksio[parquet]'` and
`--format parquet -o FILE`.
Metric names are flow-qualified (a node of `train` writing `train_loss` records
`train.train_loss`) so `--list` prints the names the selected runs
carry when the spelling is not obvious. The engine answers it over the whole
selection, so a name only an older run ever recorded is listed too.
Both take `--flow`, `--run ID` (repeat it), `--group`, `--status`, `--since`,
`--until` and `--local`, and both put the run id on every row: it is the join
back to the run page and to what the run made.
`--format` is `csv` (the default), `jsonl` or `parquet`; output goes to stdout
unless `-o FILE` names somewhere. Parquet keeps the types, needs pyarrow
(`pip install 'fluksio[parquet]'`) and needs a file to write, since it is not a
stream. In a notebook, `Client.export_metrics()` and `Client.export_runs()`
answer the same rows as a list of dicts, which `pandas.DataFrame` takes
directly.
In Python, `Client.export_metrics()` and `Client.export_runs()` return the
same rows as lists of dictionaries.
### `fluksio artifacts`
```sh
fluksio artifacts 1758042000123-9f2ab41c
fluksio artifacts 1758042000123-9f2ab41c weights -o model.npz
fluksio artifacts <run-id>
fluksio artifacts <run-id> train.weights -o weights.json
```
The files a run produced: what `fluksio.save_artifact(...)` wrote, and any
artifact a node returned. Named alone it lists them: the message each left on,
its size, its media type and the filename the node gave it. Name one and it is
written here, under that filename unless `-o` says otherwise.
The message name is the one to pass, since it is what addresses the bytes;
`--local` reads them from this directory without an engine served.
Lists a run's returned artifacts, including their message names, sizes and
filenames. Pass a name from that list to download it. Output uses the stored
filename unless `-o` selects another path. `--local` reads the local instance.
## What lives in the data directory
```text
.fluksio/ (or ~/.fluksio, with `--global`)
├── client.json the token `serve` wrote, mode 600
├── serve.pid the engine serving this directory, while one is
├── serve.log what the engine under the dashboard printed
├── .gitignore `*`, so a database and a credential are ignored from within
├── fluksio.db SQLite: users, runs, metrics, observability, agents
├── flows/ a git repository, one directory per flow
│ ├── house/
│ │ ├── flow.json the published structure
│ │ ├── nodes/*.py the published node code
│ │ ├── flow.draft.json unpublished edits, if any
│ │ └── nodes.draft/*.py
│ ├── _lib/ shared node sources
│ ├── _dashboards/ dashboards, drafts and all
│ └── requirements.txt what the Modules screen installs
├── artifacts/ content-addressed bytes, two levels deep
├── user-venv/ the interpreter your node code runs on
├── secrets.enc encrypted credentials, deliberately outside flows/
├── alerts.json alert channels and rules
├── panels.json wall-panel pairings
├── oauth-key.pem signs agent tokens
├── cloud.json the portal enrolment, if there is one
├── secret_key signs sessions and derives the secrets key
└── env optional settings file
```
The main files and directories are:
Two things follow from this layout and are worth internalising:
| Path | Contents |
|---|---|
| `client.json` | Local API URL and token |
| `fluksio.db` | SQLite database: accounts, runs, metrics and operations history |
| `flows/` | Git repository of flows, node sources, dashboards and module requirements |
| `artifacts/` | Files stored by content digest |
| `user-venv/` | Managed node environment, when used |
| `secrets.enc` | Encrypted credentials, outside the flow repository |
| `cloud.json` | Portal enrollment, when paired |
| `secret_key` | Session signing and encryption key |
| `serve.pid`, `serve.log` | Running engine information and terminal dashboard logs |
| `env` | Optional instance settings |
**`flows/` is a real git repository.** `git log` is the history of every change
anyone made to any flow. A run records the commit it ran at, so `git show` on
that hash is literally the code that produced the number.
The flow repository records edits in Git. Runs record its commit and code
digests; code-defined nodes also depend on your project files, so keep that
project's history too.
**Backing up the data directory backs up the instance.** Everything else is
rebuildable. Copy it while the engine is stopped, or use SQLite's online backup
for the database if it is not.
Back up the data directory while the engine is stopped. An adopted venv and
your project files live outside it and need their own backup.
## Settings
Settings come from the environment, or from an `env` file in the data
directory. The ones you are most likely to touch:
Settings come from environment variables or an `env` file in the data
directory. Common choices:
| Variable | Default | What it does |
|---|---|---|
| `DATA_DIR` | `./.fluksio` via the CLI; `~/.fluksio` with `--global` | everything below it derives from this |
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links |
| `ENVIRONMENT` | `local` | `production` closes the interactive API schema |
| `MCP_ENABLED` | `false` | opens the [agent endpoint](agents.md) |
| `SECRET_KEY` | generated once | signs sessions, derives the secrets key |
| Variable | Purpose |
|---|---|
| `FLUKSIO_HOME` | Select the CLI's data directory |
| `NODE_VENV` | `auto` adopts an existing venv; `managed` requests a separate one; a path selects an interpreter |
| `REDIS_HOST` | Use Redis for flow state and queues |
| `FRONTEND_HOST` | Browser address used in mail, OAuth metadata and panel links |
| `ENVIRONMENT` | `production` disables interactive API documentation |
| `MCP_ENABLED` | Enable the [agent endpoint](agents.md) |
The full list is in [Configuration](../reference/configuration.md).
See [Configuration](../reference/configuration.md) for defaults and the full
list.