Document the scheduler
Docs / docs (push) Successful in 32s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m56s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m0s
pre-commit / pre-commit (push) Failing after 3m8s
Test Backend / test-backend (push) Successful in 2m45s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m15s
Docs / docs (push) Successful in 32s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m56s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m0s
pre-commit / pre-commit (push) Failing after 3m8s
Test Backend / test-backend (push) Successful in 2m45s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m15s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
This commit is contained in:
@@ -251,6 +251,10 @@ what is wrong if not, whether it is paired with a portal and reaching it, then
|
||||
every flow with its state, its node count and whether it has unpublished
|
||||
changes, and the last few runs and failures under them.
|
||||
|
||||
A `resources` line names each machine the engine can run a node on and how much
|
||||
of it is in use, plus how many nodes are queued for one. It is absent on an
|
||||
engine that accounts for nothing.
|
||||
|
||||
The portal reads one of three ways. `no portal` means this installation was
|
||||
never enrolled. `portal hub.fluksio.com` means the link is up. `portal
|
||||
unreachable` names the error, and is the one worth acting on — the dashboard is
|
||||
@@ -278,6 +282,16 @@ kB of JSON still lists as a table; `Client.runs()` is where the whole value is
|
||||
read. `--local` reads the same history from an in-process engine, without one
|
||||
having to be served.
|
||||
|
||||
### `fluksio flavors`
|
||||
|
||||
```sh
|
||||
fluksio flavors
|
||||
```
|
||||
|
||||
The named sizes a node can ask for — `@node(resources="gpu-small")` — with the
|
||||
cores, memory and cards each stands for. Editing them is the Workers screen or
|
||||
`POST /api/v1/flavors`; this is the read.
|
||||
|
||||
### `fluksio sweep`
|
||||
|
||||
```sh
|
||||
|
||||
@@ -299,16 +299,51 @@ threshold, a message on its way somewhere. Those share the engine's worker pool
|
||||
and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops
|
||||
several of them at once from each sizing to the whole box.
|
||||
|
||||
### Ask for a size by name
|
||||
|
||||
Cores and gigabytes are a property of the machines you have, and those change.
|
||||
A node that names a **flavor** keeps meaning something afterwards:
|
||||
|
||||
```python
|
||||
@node(..., resources="gpu-small")
|
||||
def finetune(checkpoint):
|
||||
...
|
||||
|
||||
@node(..., resources={"flavor": "medium", "duration_s": "2h"})
|
||||
def fit(dataset, lr, epochs=25):
|
||||
...
|
||||
```
|
||||
|
||||
The flavor is read every time the node is built, so editing it changes what the
|
||||
next run gets. `fluksio flavors` lists them; the Workers screen edits them.
|
||||
|
||||
`ram` takes `"2G"` and `duration_s` takes `"30m"`, and a flavor already says how
|
||||
much — pass one or the numbers, not both. `duration_s` is a statement about the
|
||||
node for whoever is planning around it, not a limit; the limit is `timeout`.
|
||||
|
||||
### Where it runs
|
||||
|
||||
This is one decision, not two. Of every machine attached — this engine and each
|
||||
worker — the node goes to one that can grant what it asked for and has it free.
|
||||
So a node asking for a card finds the box that has one, without naming it:
|
||||
|
||||
```python
|
||||
@node(..., resources={"gpus": 1}) # wherever there is a card
|
||||
@node(..., device="gpu", resources="gpu-small") # that box, and this much of it
|
||||
```
|
||||
|
||||
If nothing attached can ever grant the ask, it is cut down to what is here and
|
||||
runs anyway — a flow written on a cluster still has to work on a laptop. If
|
||||
something *could* but is busy, the node waits and says so.
|
||||
|
||||
Ask what is free, and what is waiting for it, at
|
||||
`GET /api/v1/workers/resources` — a node queued for cores looks exactly like a
|
||||
node that has hung unless you can see the queue.
|
||||
`GET /api/v1/workers/resources`, on the Workers screen, or in `fluksio status`.
|
||||
|
||||
!!! note "Accounted, not enforced"
|
||||
|
||||
Nothing stops a node that ignores its declaration; the numbers are
|
||||
bookkeeping plus the environment its libraries read. Scheduling across
|
||||
several machines, named hardware flavours and real enforcement are the
|
||||
next steps, not this one.
|
||||
bookkeeping plus the environment its libraries read. Real enforcement —
|
||||
cgroups, rlimits — is a next step, not this one.
|
||||
|
||||
## Say which nodes make a flow
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ Anything already exported wins over the file.
|
||||
| `FLOWS_DIR` | `$DATA_DIR/flows` | the git repository holding flows |
|
||||
| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials, deliberately outside the repo |
|
||||
| `ALERTS_FILE` | `$DATA_DIR/alerts.json` | channels and rules |
|
||||
| `PROVISIONERS_FILE` | `$DATA_DIR/provisioners.json` | clusters a machine can be started from; absent means none |
|
||||
| `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings |
|
||||
| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens |
|
||||
| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any |
|
||||
@@ -141,8 +142,14 @@ A node that declares nothing is not accounted against `FLOW_CPUS`; it runs on
|
||||
the shared pool and is given `FLOW_CPUS / FLOW_MAX_WORKERS` as a thread cap, so
|
||||
several at once cannot each size themselves to the whole machine. Setting
|
||||
`OMP_NUM_THREADS` (or any of its siblings) on the engine yourself overrides
|
||||
that default. What is free, and which nodes are queued for it, is
|
||||
`GET /api/v1/workers/resources`. See
|
||||
that default.
|
||||
|
||||
These two are this machine's figures. An attached worker reports its own when
|
||||
it dials in, and a node goes to whichever machine can grant what it asked for —
|
||||
so a GPU on a worker needs no `FLOW_GPUS` here. What every machine has free, and
|
||||
which nodes are queued, is `GET /api/v1/workers/resources` and the Workers
|
||||
screen. Named sizes live in the database and are `GET /api/v1/flavors`;
|
||||
`PROVISIONERS_FILE` is where machines can be started from. See
|
||||
[declaring resources](../getting-started/data-science.md#declaring-what-a-node-needs).
|
||||
|
||||
## Agents
|
||||
|
||||
Reference in New Issue
Block a user