Four things the python SDK turned up, each fixed where every client sees it. A key no port declares is now an error rather than a silent drop, on the return, the yield and the emit alike — the contract the docs already stated. The SDK reads literal yields at sync time, so a typo fails before anything runs, and an emission of one fails the call rather than being logged where nobody looks. NaN and infinity are refused at the port. JSON cannot spell either, so one that travelled came back as a 500, a socket frame that stopped the canvas, or a metric batch the database dropped whole. An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the engine — so the CLI, the run dialog and a python caller mean the same thing, and a sweep can pass one at all. Node timeouts are off by default. The clock measured silence, which a training node is full of, and remote workers had already stopped enforcing it — their heartbeat reset it. Now a heartbeat proves the agent rather than the node, ninety seconds of nothing fails the call either way, and the engine touches work it is still running so a long node is not redelivered at sixty seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
295 lines
12 KiB
Markdown
295 lines
12 KiB
Markdown
# The `fluksio` command
|
|
|
|
```sh
|
|
pip install fluksio
|
|
```
|
|
|
|
Installs the engine and the `fluksio` command. Python 3.12 or newer, Linux or
|
|
macOS.
|
|
|
|
There is a second, smaller distribution — `fluksio-worker` — for a machine that
|
|
should only *run nodes* for an engine elsewhere. It has none of the engine in
|
|
it. See [Remote workers](workers.md).
|
|
|
|
The command is two things at once: `serve`, `enroll` and `worker` *are* an
|
|
installation, while `login`, `sync`, `run`, `runs` and `sweep` talk to one that
|
|
may be anywhere.
|
|
|
|
## Where an installation lives
|
|
|
|
`.fluksio` beside your code, found the way `.git` is: from the working
|
|
directory, or any directory above it. Two repositories on one machine are
|
|
therefore two engines, with their own flows, runs and token. `fluksio serve`
|
|
makes one where there is none, and it ignores itself from within — a
|
|
`.gitignore` of `*`, so a database and a credential cannot be committed by
|
|
accident.
|
|
|
|
`--global` uses `~/.fluksio` instead, shared by every directory. `--data-dir`
|
|
(or `FLUKSIO_HOME`) names any directory outright and wins over both.
|
|
|
|
## `fluksio serve`
|
|
|
|
Runs the engine.
|
|
|
|
```sh
|
|
fluksio serve
|
|
```
|
|
|
|
On the first start it creates an admin account and prints its password **once**.
|
|
Nothing else has to be running: no database server, no message broker, no
|
|
Docker.
|
|
|
|
| Option | Default | What it does |
|
|
|---|---|---|
|
|
| `--data-dir PATH` | `./.fluksio` (or `$FLUKSIO_HOME`) | where this installation keeps everything |
|
|
| `--host HOST` | `127.0.0.1` | what to bind |
|
|
| `--port PORT` | `8000` | what to listen on |
|
|
| `--log-level LEVEL` | `info` | uvicorn's log level |
|
|
| `--admin-email ADDR` | `admin@example.com` | the account created on first run |
|
|
| `--admin-password PW` | generated | set it instead of having one generated |
|
|
| `--enroll CODE` | — | pair with a portal as part of coming up |
|
|
| `--portal URL` | — | the portal `--enroll` redeems at |
|
|
|
|
`--enroll` with `--portal` is the one-command setup: it pairs before the engine
|
|
starts, so the connection is dialled as part of coming up rather than needing a
|
|
restart. It is skipped if the installation is already enrolled.
|
|
|
|
!!! warning "One process"
|
|
|
|
`fluksio serve` holds the flow engine. A second one is a *second engine* —
|
|
duplicated subscriptions, duplicated cron ticks, two webhooks answering the
|
|
same path. Run one, and distribute work with
|
|
[workers](workers.md) instead.
|
|
|
|
!!! note "`$HOME` on a cluster"
|
|
|
|
A login node's home directory is often NFS, where SQLite's write-ahead log
|
|
does not work — the database would be locked or corrupt. `fluksio serve`
|
|
warns when it notices; point `--data-dir` at local disk.
|
|
|
|
### What it prints
|
|
|
|
```text
|
|
Created the admin account admin@example.com
|
|
password: k3Qm-8vTpLdX
|
|
Shown once. Change it from the dashboard.
|
|
Fluksio 0.1.0 — data in /home/you/.fluksio
|
|
API http://127.0.0.1:8000/api/v1
|
|
No portal. Pair this installation with:
|
|
fluksio enroll <code>
|
|
```
|
|
|
|
An enrolled installation says which portal it is on instead, and notes that the
|
|
dashboard is served from there rather than here.
|
|
|
|
## `fluksio enroll`
|
|
|
|
Pairs an existing installation with a portal.
|
|
|
|
```sh
|
|
fluksio enroll ABCD-1234
|
|
```
|
|
|
|
| Option | What it does |
|
|
|---|---|
|
|
| `--portal URL` | a portal of your own, instead of `https://hub.fluksio.com` |
|
|
| `--as EMAIL` | the local account a portal session arrives as |
|
|
| `--data-dir PATH` | which installation, if not the one this directory is in |
|
|
|
|
Get the code from the portal under **Installations → Add installation**. It is
|
|
single-use and expires in fifteen minutes. `--as` matters when the installation
|
|
has several superusers — without it, enrolment refuses rather than guessing.
|
|
|
|
Afterwards, `fluksio serve` dials the portal as it comes up, and keeps dialling:
|
|
a portal that restarts, a wifi that changes, a laptop that suspends and wakes
|
|
somewhere else all end the same connection, and the link is put back up without
|
|
anybody noticing. A connection that stood up and then dropped is retried at
|
|
once; one that never stood up waits a little longer each time, up to half a
|
|
minute. See [Accounts and the portal](../interface/portal.md).
|
|
|
|
## `fluksio worker`
|
|
|
|
Runs nodes for an engine elsewhere. Everything after `worker` belongs to the
|
|
agent's own parser — it is the same program `fluksio-worker` installs, so the
|
|
two are interchangeable:
|
|
|
|
```sh
|
|
fluksio worker --url wss://api.example.com/api/v1/workers/attach \
|
|
--token "$FLUKSIO_WORKER_TOKEN" --labels gpu
|
|
```
|
|
|
|
See [Remote workers](workers.md).
|
|
|
|
## Talking to an engine
|
|
|
|
The commands below are the client half: they run wherever you work, and
|
|
address an engine over its API rather than being one — except under `--local`,
|
|
which boots one inside the command instead.
|
|
|
|
### `fluksio login`
|
|
|
|
```sh
|
|
fluksio login --url https://api.example.com
|
|
```
|
|
|
|
For an engine somewhere *else*. One you started yourself needs no login:
|
|
`fluksio serve` writes the token as it comes up and says where it put it.
|
|
|
|
The token goes in this project's `.fluksio/client.json`, or with `--global` in
|
|
`~/.fluksio/client.json`. Every command below reads it from there — nearest
|
|
first, walking up from the working directory — or from `FLUKSIO_URL` and
|
|
`FLUKSIO_TOKEN`, or from its own `--url` and `--token`. A token an older
|
|
version wrote to `~/.config/fluksio/client.json` is still read.
|
|
|
|
### `fluksio sync`
|
|
|
|
```sh
|
|
fluksio sync [PATH_OR_MODULE ...] # default: the current directory
|
|
```
|
|
|
|
Imports what you name, collects the flows the decorators declared, and uploads
|
|
each one with a generated import shim per node. A directory that is a package
|
|
is walked; a dotted name is imported as it stands; nothing is loaded from a
|
|
file path, because the shim has to import the same way.
|
|
|
|
| Flag | What it does |
|
|
|---|---|
|
|
| `--dry-run` | print the flow documents and shims, upload nothing |
|
|
| `--no-publish` | leave the upload as a draft |
|
|
| `--force` | overwrite a flow, or a node body, that was edited on the canvas |
|
|
|
|
Every sync retires the engine's workers, including one that had nothing to
|
|
upload — a worker holds your package in memory, so an edit to it is invisible
|
|
until the process goes. See
|
|
[Getting started: data science](../getting-started/data-science.md).
|
|
|
|
### `fluksio run`
|
|
|
|
```sh
|
|
fluksio run train --lr 0.05 --seed 7 [--wait]
|
|
```
|
|
|
|
Syncs the working directory, then submits a run — so the command after an edit
|
|
is this one and nothing else. Flags that are not its own are the flow's
|
|
inputs, typed by what the flow declares them as. `--wait` blocks until the run
|
|
finishes and exits non-zero if it failed. `--follow` waits as well, and prints
|
|
the numbers the run reports as they arrive:
|
|
|
|
```text
|
|
train.loss[14] = 3.40295e-06
|
|
```
|
|
|
|
Ctrl-C while either is waiting cancels the run on the engine rather than only
|
|
stopping the watching, and exits 130.
|
|
|
|
An input declared as an `artifact` takes the file a previous run produced,
|
|
named rather than typed out:
|
|
|
|
```sh
|
|
fluksio run evaluate --dataset @run:1758042000123-9f2ab41c.dataset
|
|
fluksio run evaluate --dataset sha256:6dd1f0…
|
|
```
|
|
|
|
`@run:<id>.<output>` is what that run's output was, and a bare digest is the
|
|
content itself; the engine resolves either into the reference. Passing the
|
|
whole reference as JSON still works and is what a script that already holds one
|
|
does — which is the same thing `flow.submit(dataset=run.result["dataset"])`
|
|
does from Python.
|
|
|
|
`--no-sync` runs what is already on the engine. Worth it in a tight loop where
|
|
you know nothing changed, since syncing retires the workers and the next call
|
|
pays its imports again. A directory that declares no flows syncs nothing and
|
|
says nothing — a flow drawn on the canvas is run the same way.
|
|
|
|
`--no-cache` executes every node, including one an earlier run already
|
|
answered. See [Stage caching](../concepts/runs.md#stage-caching).
|
|
|
|
`--local` boots the engine inside this process instead of talking to a served
|
|
one, so there is no `fluksio serve` terminal to keep open. It is the same
|
|
installation either way — the same `.fluksio`, the same database, artifacts
|
|
and run history — so a run made this way and a run made through a served
|
|
engine cache against each other. It always waits, because the engine it starts
|
|
lives exactly as long as the command. Starting one costs a few seconds of
|
|
worker pool and module reconcile, against the ~15 ms of submitting to an
|
|
engine that is already up: `--local` is for "I just want to run it", not for a
|
|
loop you are iterating in.
|
|
|
|
### `fluksio runs`
|
|
|
|
```sh
|
|
fluksio runs [--flow train] [--limit 20]
|
|
```
|
|
|
|
The runs an engine has recorded, newest first: id, status, flow, duration, the
|
|
commit of the repository it came from, and its parameters. Statuses are
|
|
coloured when a terminal is reading the output — `ok` green, `error` red,
|
|
`cached` cyan. `--local` reads the same history from an in-process engine,
|
|
without one having to be served.
|
|
|
|
### `fluksio sweep`
|
|
|
|
```sh
|
|
fluksio sweep train --param lr=0.1,0.01 --param epochs=10,50 --wait
|
|
```
|
|
|
|
Every combination of the parameter lists, submitted as one group — four runs
|
|
above, sharing a `group_id` and executing in parallel. Values are typed by the
|
|
flow's inputs, the same as `run`'s are, and `--seed`, `--no-sync`,
|
|
`--no-cache` and `--local` mean what they do there. `--wait` blocks until all
|
|
of them are finished and exits non-zero if any failed.
|
|
|
|
## What lives in the data directory
|
|
|
|
```text
|
|
.fluksio/ (or ~/.fluksio, with `--global`)
|
|
├── client.json the token `serve` wrote, mode 600
|
|
├── .gitignore `*` — a database and a credential, ignored from within
|
|
├── fluksio.db SQLite: users, runs, metrics, observability, agents
|
|
├── flows/ a git repository — one directory per flow
|
|
│ ├── house/
|
|
│ │ ├── flow.json the published structure
|
|
│ │ ├── nodes/*.py the published node code
|
|
│ │ ├── flow.draft.json unpublished edits, if any
|
|
│ │ └── nodes.draft/*.py
|
|
│ ├── _lib/ shared node sources
|
|
│ ├── _dashboards/ dashboards, drafts and all
|
|
│ └── requirements.txt what the Modules screen installs
|
|
├── artifacts/ content-addressed bytes, two levels deep
|
|
├── user-venv/ the interpreter your node code runs on
|
|
├── secrets.enc encrypted credentials, deliberately outside flows/
|
|
├── alerts.json alert channels and rules
|
|
├── panels.json wall-panel pairings
|
|
├── oauth-key.pem signs agent tokens
|
|
├── cloud.json the portal enrolment, if there is one
|
|
├── secret_key signs sessions and derives the secrets key
|
|
└── env optional settings file
|
|
```
|
|
|
|
Two things follow from this layout and are worth internalising:
|
|
|
|
**`flows/` is a real git repository.** `git log` is the history of every change
|
|
anyone made to any flow. A run records the commit it ran at, so `git show` on
|
|
that hash is literally the code that produced the number.
|
|
|
|
**Backing up the data directory backs up the installation.** Everything else is
|
|
rebuildable. Copy it while the engine is stopped, or use SQLite's online backup
|
|
for the database if it is not.
|
|
|
|
## Settings
|
|
|
|
Settings come from the environment, or from an `env` file in the data
|
|
directory. The ones you are most likely to touch:
|
|
|
|
| Variable | Default | What it does |
|
|
|---|---|---|
|
|
| `DATA_DIR` | `./.fluksio` via the CLI; `~/.fluksio` with `--global` | everything below it derives from this |
|
|
| `DATABASE_URL` | SQLite in the data dir | any SQLAlchemy URL |
|
|
| `NODE_VENV` | `auto` | which interpreter node code runs on: `auto` adopts the venv Fluksio was installed into, `managed` builds one of its own, or name an interpreter |
|
|
| `REDIS_HOST` | unset | flow state in Redis instead of memory; survives a restart |
|
|
| `FRONTEND_HOST` | — | the address used in mails, OAuth metadata and panel links |
|
|
| `ENVIRONMENT` | `local` | `production` closes the interactive API schema |
|
|
| `MCP_ENABLED` | `false` | opens the [agent endpoint](agents.md) |
|
|
| `SECRET_KEY` | generated once | signs sessions, derives the secrets key |
|
|
|
|
The full list is in [Configuration](../reference/configuration.md).
|