Document the scheduler
Docs / docs (push) Successful in 32s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m56s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m0s
pre-commit / pre-commit (push) Failing after 3m8s
Test Backend / test-backend (push) Successful in 2m45s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m15s

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
This commit is contained in:
2026-08-27 09:36:39 +02:00
co-authored by Claude Opus 5
parent 7d87dfffdd
commit e2f97d36e4
3 changed files with 63 additions and 7 deletions
+14
View File
@@ -251,6 +251,10 @@ what is wrong if not, whether it is paired with a portal and reaching it, then
every flow with its state, its node count and whether it has unpublished
changes, and the last few runs and failures under them.
A `resources` line names each machine the engine can run a node on and how much
of it is in use, plus how many nodes are queued for one. It is absent on an
engine that accounts for nothing.
The portal reads one of three ways. `no portal` means this installation was
never enrolled. `portal hub.fluksio.com` means the link is up. `portal
unreachable` names the error, and is the one worth acting on — the dashboard is
@@ -278,6 +282,16 @@ kB of JSON still lists as a table; `Client.runs()` is where the whole value is
read. `--local` reads the same history from an in-process engine, without one
having to be served.
### `fluksio flavors`
```sh
fluksio flavors
```
The named sizes a node can ask for — `@node(resources="gpu-small")` — with the
cores, memory and cards each stands for. Editing them is the Workers screen or
`POST /api/v1/flavors`; this is the read.
### `fluksio sweep`
```sh
+40 -5
View File
@@ -299,16 +299,51 @@ threshold, a message on its way somewhere. Those share the engine's worker pool
and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops
several of them at once from each sizing to the whole box.
### Ask for a size by name
Cores and gigabytes are a property of the machines you have, and those change.
A node that names a **flavor** keeps meaning something afterwards:
```python
@node(..., resources="gpu-small")
def finetune(checkpoint):
...
@node(..., resources={"flavor": "medium", "duration_s": "2h"})
def fit(dataset, lr, epochs=25):
...
```
The flavor is read every time the node is built, so editing it changes what the
next run gets. `fluksio flavors` lists them; the Workers screen edits them.
`ram` takes `"2G"` and `duration_s` takes `"30m"`, and a flavor already says how
much — pass one or the numbers, not both. `duration_s` is a statement about the
node for whoever is planning around it, not a limit; the limit is `timeout`.
### Where it runs
This is one decision, not two. Of every machine attached — this engine and each
worker — the node goes to one that can grant what it asked for and has it free.
So a node asking for a card finds the box that has one, without naming it:
```python
@node(..., resources={"gpus": 1}) # wherever there is a card
@node(..., device="gpu", resources="gpu-small") # that box, and this much of it
```
If nothing attached can ever grant the ask, it is cut down to what is here and
runs anyway — a flow written on a cluster still has to work on a laptop. If
something *could* but is busy, the node waits and says so.
Ask what is free, and what is waiting for it, at
`GET /api/v1/workers/resources` — a node queued for cores looks exactly like a
node that has hung unless you can see the queue.
`GET /api/v1/workers/resources`, on the Workers screen, or in `fluksio status`.
!!! note "Accounted, not enforced"
Nothing stops a node that ignores its declaration; the numbers are
bookkeeping plus the environment its libraries read. Scheduling across
several machines, named hardware flavours and real enforcement are the
next steps, not this one.
bookkeeping plus the environment its libraries read. Real enforcement —
cgroups, rlimits — is a next step, not this one.
## Say which nodes make a flow
+9 -2
View File
@@ -19,6 +19,7 @@ Anything already exported wins over the file.
| `FLOWS_DIR` | `$DATA_DIR/flows` | the git repository holding flows |
| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials, deliberately outside the repo |
| `ALERTS_FILE` | `$DATA_DIR/alerts.json` | channels and rules |
| `PROVISIONERS_FILE` | `$DATA_DIR/provisioners.json` | clusters a machine can be started from; absent means none |
| `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings |
| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens |
| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any |
@@ -141,8 +142,14 @@ A node that declares nothing is not accounted against `FLOW_CPUS`; it runs on
the shared pool and is given `FLOW_CPUS / FLOW_MAX_WORKERS` as a thread cap, so
several at once cannot each size themselves to the whole machine. Setting
`OMP_NUM_THREADS` (or any of its siblings) on the engine yourself overrides
that default. What is free, and which nodes are queued for it, is
`GET /api/v1/workers/resources`. See
that default.
These two are this machine's figures. An attached worker reports its own when
it dials in, and a node goes to whichever machine can grant what it asked for —
so a GPU on a worker needs no `FLOW_GPUS` here. What every machine has free, and
which nodes are queued, is `GET /api/v1/workers/resources` and the Workers
screen. Named sizes live in the database and are `GET /api/v1/flavors`;
`PROVISIONERS_FILE` is where machines can be started from. See
[declaring resources](../getting-started/data-science.md#declaring-what-a-node-needs).
## Agents