Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m5s
Playwright Tests / test-playwright (2, 2) (push) Failing after 20s
pre-commit / pre-commit (push) Failing after 2m33s
Test Backend / test-backend (push) Successful in 2m7s
Compose Smoke Test / test-compose (push) Failing after 20s
Playwright Tests / merge-reports (push) Failing after 1m3s
Publish / publish (push) Failing after 12s
A code node in a batch run is now fingerprinted by its source, its raw settings and the values it reads — an artifact input counting as its digest, which is what the content addressing was always for. A run that finds the key restores what the earlier one returned and skips the node, recorded as `cached`. The run history is the cache: `run_node.outputs` beside the `cache_key` the schema already had, no second store. On for code nodes, never for the built-in and connector types that have side effects; off per node with `@node(cache=False)` and per run with `--no-cache`. Emissions are not replayed on a hit, so a cached training node returns its result without redrawing its curve. Recorded in NOTEPAD.md with the two other deliberate limits. `fluksio run --local` boots the real app in the command's own process and drives it through its ASGI interface behind the ordinary client, so a run no longer needs a `serve` terminal beside it — same data directory, same history, and the cache carries between the two. It always waits, because the engine it starts lives exactly as long as the command. Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists, `run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited run rather than abandoning it, coloured statuses on a terminal, and `name` made optional on the metrics endpoint so a follower can ask for every series. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
281 lines
11 KiB
Markdown
281 lines
11 KiB
Markdown
# The `fluksio` command
|
|
|
|
```sh
|
|
pip install fluksio
|
|
```
|
|
|
|
Installs the engine and the `fluksio` command. Python 3.12 or newer, Linux or
|
|
macOS.
|
|
|
|
There is a second, smaller distribution — `fluksio-worker` — for a machine that
|
|
should only *run nodes* for an engine elsewhere. It has none of the engine in
|
|
it. See [Remote workers](workers.md).
|
|
|
|
The command is two things at once: `serve`, `enroll` and `worker` *are* an
|
|
installation, while `login`, `sync`, `run`, `runs` and `sweep` talk to one that
|
|
may be anywhere.
|
|
|
|
## Where an installation lives
|
|
|
|
`.fluksio` beside your code, found the way `.git` is: from the working
|
|
directory, or any directory above it. Two repositories on one machine are
|
|
therefore two engines, with their own flows, runs and token. `fluksio serve`
|
|
makes one where there is none, and it ignores itself from within — a
|
|
`.gitignore` of `*`, so a database and a credential cannot be committed by
|
|
accident.
|
|
|
|
`--global` uses `~/.fluksio` instead, shared by every directory. `--data-dir`
|
|
(or `FLUKSIO_HOME`) names any directory outright and wins over both.
|
|
|
|
## `fluksio serve`
|
|
|
|
Runs the engine.
|
|
|
|
```sh
|
|
fluksio serve
|
|
```
|
|
|
|
On the first start it creates an admin account and prints its password **once**.
|
|
Nothing else has to be running: no database server, no message broker, no
|
|
Docker.
|
|
|
|
| Option | Default | What it does |
|
|
|---|---|---|
|
|
| `--data-dir PATH` | `./.fluksio` (or `$FLUKSIO_HOME`) | where this installation keeps everything |
|
|
| `--host HOST` | `127.0.0.1` | what to bind |
|
|
| `--port PORT` | `8000` | what to listen on |
|
|
| `--log-level LEVEL` | `info` | uvicorn's log level |
|
|
| `--admin-email ADDR` | `admin@example.com` | the account created on first run |
|
|
| `--admin-password PW` | generated | set it instead of having one generated |
|
|
| `--enroll CODE` | — | pair with a portal as part of coming up |
|
|
| `--portal URL` | — | the portal `--enroll` redeems at |
|
|
|
|
`--enroll` with `--portal` is the one-command setup: it pairs before the engine
|
|
starts, so the connection is dialled as part of coming up rather than needing a
|
|
restart. It is skipped if the installation is already enrolled.
|
|
|
|
!!! warning "One process"
|
|
|
|
`fluksio serve` holds the flow engine. A second one is a *second engine* —
|
|
duplicated subscriptions, duplicated cron ticks, two webhooks answering the
|
|
same path. Run one, and distribute work with
|
|
[workers](workers.md) instead.
|
|
|
|
!!! note "`$HOME` on a cluster"
|
|
|
|
A login node's home directory is often NFS, where SQLite's write-ahead log
|
|
does not work — the database would be locked or corrupt. `fluksio serve`
|
|
warns when it notices; point `--data-dir` at local disk.
|
|
|
|
### What it prints
|
|
|
|
```text
|
|
Created the admin account admin@example.com
|
|
password: k3Qm-8vTpLdX
|
|
Shown once. Change it from the dashboard.
|
|
Fluksio 0.1.0 — data in /home/you/.fluksio
|
|
API http://127.0.0.1:8000/api/v1
|
|
No portal. Pair this installation with:
|
|
fluksio enroll <code>
|
|
```
|
|
|
|
An enrolled installation says which portal it is on instead, and notes that the
|
|
dashboard is served from there rather than here.
|
|
|
|
## `fluksio enroll`
|
|
|
|
Pairs an existing installation with a portal.
|
|
|
|
```sh
|
|
fluksio enroll ABCD-1234
|
|
```
|
|
|
|
| Option | What it does |
|
|
|---|---|
|
|
| `--portal URL` | a portal of your own, instead of `https://hub.fluksio.com` |
|
|
| `--as EMAIL` | the local account a portal session arrives as |
|
|
| `--data-dir PATH` | which installation, if not the one this directory is in |
|
|
|
|
Get the code from the portal under **Installations → Add installation**. It is
|
|
single-use and expires in fifteen minutes. `--as` matters when the installation
|
|
has several superusers — without it, enrolment refuses rather than guessing.
|
|
|
|
Afterwards, `fluksio serve` dials the portal as it comes up, and keeps dialling:
|
|
a portal that restarts, a wifi that changes, a laptop that suspends and wakes
|
|
somewhere else all end the same connection, and the link is put back up without
|
|
anybody noticing. A connection that stood up and then dropped is retried at
|
|
once; one that never stood up waits a little longer each time, up to half a
|
|
minute. See [Accounts and the portal](../interface/portal.md).
|
|
|
|
## `fluksio worker`
|
|
|
|
Runs nodes for an engine elsewhere. Everything after `worker` belongs to the
|
|
agent's own parser — it is the same program `fluksio-worker` installs, so the
|
|
two are interchangeable:
|
|
|
|
```sh
|
|
fluksio worker --url wss://api.example.com/api/v1/workers/attach \
|
|
--token "$FLUKSIO_WORKER_TOKEN" --labels gpu
|
|
```
|
|
|
|
See [Remote workers](workers.md).
|
|
|
|
## Talking to an engine
|
|
|
|
The commands below are the client half: they run wherever you work, and
|
|
address an engine over its API rather than being one — except under `--local`,
|
|
which boots one inside the command instead.
|
|
|
|
### `fluksio login`
|
|
|
|
```sh
|
|
fluksio login --url https://api.example.com
|
|
```
|
|
|
|
For an engine somewhere *else*. One you started yourself needs no login:
|
|
`fluksio serve` writes the token as it comes up and says where it put it.
|
|
|
|
The token goes in this project's `.fluksio/client.json`, or with `--global` in
|
|
`~/.fluksio/client.json`. Every command below reads it from there — nearest
|
|
first, walking up from the working directory — or from `FLUKSIO_URL` and
|
|
`FLUKSIO_TOKEN`, or from its own `--url` and `--token`. A token an older
|
|
version wrote to `~/.config/fluksio/client.json` is still read.
|
|
|
|
### `fluksio sync`
|
|
|
|
```sh
|
|
fluksio sync [PATH_OR_MODULE ...] # default: the current directory
|
|
```
|
|
|
|
Imports what you name, collects the flows the decorators declared, and uploads
|
|
each one with a generated import shim per node. A directory that is a package
|
|
is walked; a dotted name is imported as it stands; nothing is loaded from a
|
|
file path, because the shim has to import the same way.
|
|
|
|
| Flag | What it does |
|
|
|---|---|
|
|
| `--dry-run` | print the flow documents and shims, upload nothing |
|
|
| `--no-publish` | leave the upload as a draft |
|
|
| `--force` | overwrite a flow, or a node body, that was edited on the canvas |
|
|
|
|
Every sync retires the engine's workers, including one that had nothing to
|
|
upload — a worker holds your package in memory, so an edit to it is invisible
|
|
until the process goes. See
|
|
[Getting started: data science](../getting-started/data-science.md).
|
|
|
|
### `fluksio run`
|
|
|
|
```sh
|
|
fluksio run train --lr 0.05 --seed 7 [--wait]
|
|
```
|
|
|
|
Syncs the working directory, then submits a run — so the command after an edit
|
|
is this one and nothing else. Flags that are not its own are the flow's
|
|
inputs, typed by what the flow declares them as. `--wait` blocks until the run
|
|
finishes and exits non-zero if it failed. `--follow` waits as well, and prints
|
|
the numbers the run reports as they arrive:
|
|
|
|
```text
|
|
train.loss[14] = 3.40295e-06
|
|
```
|
|
|
|
Ctrl-C while either is waiting cancels the run on the engine rather than only
|
|
stopping the watching, and exits 130.
|
|
|
|
`--no-sync` runs what is already on the engine. Worth it in a tight loop where
|
|
you know nothing changed, since syncing retires the workers and the next call
|
|
pays its imports again. A directory that declares no flows syncs nothing and
|
|
says nothing — a flow drawn on the canvas is run the same way.
|
|
|
|
`--no-cache` executes every node, including one an earlier run already
|
|
answered. See [Stage caching](../concepts/runs.md#stage-caching).
|
|
|
|
`--local` boots the engine inside this process instead of talking to a served
|
|
one, so there is no `fluksio serve` terminal to keep open. It is the same
|
|
installation either way — the same `.fluksio`, the same database, artifacts
|
|
and run history — so a run made this way and a run made through a served
|
|
engine cache against each other. It always waits, because the engine it starts
|
|
lives exactly as long as the command. Starting one costs a few seconds of
|
|
worker pool and module reconcile, against the ~15 ms of submitting to an
|
|
engine that is already up: `--local` is for "I just want to run it", not for a
|
|
loop you are iterating in.
|
|
|
|
### `fluksio runs`
|
|
|
|
```sh
|
|
fluksio runs [--flow train] [--limit 20]
|
|
```
|
|
|
|
The runs an engine has recorded, newest first: id, status, flow, duration, the
|
|
commit of the repository it came from, and its parameters. Statuses are
|
|
coloured when a terminal is reading the output — `ok` green, `error` red,
|
|
`cached` cyan. `--local` reads the same history from an in-process engine,
|
|
without one having to be served.
|
|
|
|
### `fluksio sweep`
|
|
|
|
```sh
|
|
fluksio sweep train --param lr=0.1,0.01 --param epochs=10,50 --wait
|
|
```
|
|
|
|
Every combination of the parameter lists, submitted as one group — four runs
|
|
above, sharing a `group_id` and executing in parallel. Values are typed by the
|
|
flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`,
|
|
`--no-cache` and `--local` mean what they do there. `--wait` blocks until all
|
|
of them are finished and exits non-zero if any failed.
|
|
|
|
## What lives in the data directory
|
|
|
|
```text
|
|
.fluksio/ (or ~/.fluksio, with `--global`)
|
|
├── client.json the token `serve` wrote, mode 600
|
|
├── .gitignore `*` — a database and a credential, ignored from within
|
|
├── fluksio.db SQLite: users, runs, metrics, observability, agents
|
|
├── flows/ a git repository — one directory per flow
|
|
│ ├── house/
|
|
│ │ ├── flow.json the published structure
|
|
│ │ ├── nodes/*.py the published node code
|
|
│ │ ├── flow.draft.json unpublished edits, if any
|
|
│ │ └── nodes.draft/*.py
|
|
│ ├── _lib/ shared node sources
|
|
│ ├── _dashboards/ dashboards, drafts and all
|
|
│ └── requirements.txt what the Modules screen installs
|
|
├── artifacts/ content-addressed bytes, two levels deep
|
|
├── user-venv/ the interpreter your node code runs on
|
|
├── secrets.enc encrypted credentials, deliberately outside flows/
|
|
├── alerts.json alert channels and rules
|
|
├── panels.json wall-panel pairings
|
|
├── oauth-key.pem signs agent tokens
|
|
├── cloud.json the portal enrolment, if there is one
|
|
├── secret_key signs sessions and derives the secrets key
|
|
└── env optional settings file
|
|
```
|
|
|
|
Two things follow from this layout and are worth internalising:
|
|
|
|
**`flows/` is a real git repository.** `git log` is the history of every change
|
|
anyone made to any flow. A run records the commit it ran at, so `git show` on
|
|
that hash is literally the code that produced the number.
|
|
|
|
**Backing up the data directory backs up the installation.** Everything else is
|
|
rebuildable. Copy it while the engine is stopped, or use SQLite's online backup
|
|
for the database if it is not.
|
|
|
|
## Settings
|
|
|
|
Settings come from the environment, or from an `env` file in the data
|
|
directory. The ones you are most likely to touch:
|
|
|
|
| Variable | Default | What it does |
|
|
|---|---|---|
|
|
| `DATA_DIR` | `./.fluksio` via the CLI; `~/.fluksio` with `--global` | everything below it derives from this |
|
|
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
|
|
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
|
|
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
|
|
| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links |
|
|
| `ENVIRONMENT` | `local` | `production` closes the interactive API schema |
|
|
| `MCP_ENABLED` | `false` | opens the [agent endpoint](agents.md) |
|
|
| `SECRET_KEY` | generated once | signs sessions, derives the secrets key |
|
|
|
|
The full list is in [Configuration](../reference/configuration.md).
|