Document the scheduler
Docs / docs (push) Successful in 32s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m56s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m0s
pre-commit / pre-commit (push) Failing after 3m8s
Test Backend / test-backend (push) Successful in 2m45s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m15s
Docs / docs (push) Successful in 32s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m56s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m0s
pre-commit / pre-commit (push) Failing after 3m8s
Test Backend / test-backend (push) Successful in 2m45s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m15s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6HeySA27EkGANZN95QySW
This commit is contained in:
@@ -251,6 +251,10 @@ what is wrong if not, whether it is paired with a portal and reaching it, then
|
|||||||
every flow with its state, its node count and whether it has unpublished
|
every flow with its state, its node count and whether it has unpublished
|
||||||
changes, and the last few runs and failures under them.
|
changes, and the last few runs and failures under them.
|
||||||
|
|
||||||
|
A `resources` line names each machine the engine can run a node on and how much
|
||||||
|
of it is in use, plus how many nodes are queued for one. It is absent on an
|
||||||
|
engine that accounts for nothing.
|
||||||
|
|
||||||
The portal reads one of three ways. `no portal` means this installation was
|
The portal reads one of three ways. `no portal` means this installation was
|
||||||
never enrolled. `portal hub.fluksio.com` means the link is up. `portal
|
never enrolled. `portal hub.fluksio.com` means the link is up. `portal
|
||||||
unreachable` names the error, and is the one worth acting on — the dashboard is
|
unreachable` names the error, and is the one worth acting on — the dashboard is
|
||||||
@@ -278,6 +282,16 @@ kB of JSON still lists as a table; `Client.runs()` is where the whole value is
|
|||||||
read. `--local` reads the same history from an in-process engine, without one
|
read. `--local` reads the same history from an in-process engine, without one
|
||||||
having to be served.
|
having to be served.
|
||||||
|
|
||||||
|
### `fluksio flavors`
|
||||||
|
|
||||||
|
```sh
|
||||||
|
fluksio flavors
|
||||||
|
```
|
||||||
|
|
||||||
|
The named sizes a node can ask for — `@node(resources="gpu-small")` — with the
|
||||||
|
cores, memory and cards each stands for. Editing them is the Workers screen or
|
||||||
|
`POST /api/v1/flavors`; this is the read.
|
||||||
|
|
||||||
### `fluksio sweep`
|
### `fluksio sweep`
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
|||||||
@@ -299,16 +299,51 @@ threshold, a message on its way somewhere. Those share the engine's worker pool
|
|||||||
and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops
|
and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops
|
||||||
several of them at once from each sizing to the whole box.
|
several of them at once from each sizing to the whole box.
|
||||||
|
|
||||||
|
### Ask for a size by name
|
||||||
|
|
||||||
|
Cores and gigabytes are a property of the machines you have, and those change.
|
||||||
|
A node that names a **flavor** keeps meaning something afterwards:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@node(..., resources="gpu-small")
|
||||||
|
def finetune(checkpoint):
|
||||||
|
...
|
||||||
|
|
||||||
|
@node(..., resources={"flavor": "medium", "duration_s": "2h"})
|
||||||
|
def fit(dataset, lr, epochs=25):
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
The flavor is read every time the node is built, so editing it changes what the
|
||||||
|
next run gets. `fluksio flavors` lists them; the Workers screen edits them.
|
||||||
|
|
||||||
|
`ram` takes `"2G"` and `duration_s` takes `"30m"`, and a flavor already says how
|
||||||
|
much — pass one or the numbers, not both. `duration_s` is a statement about the
|
||||||
|
node for whoever is planning around it, not a limit; the limit is `timeout`.
|
||||||
|
|
||||||
|
### Where it runs
|
||||||
|
|
||||||
|
This is one decision, not two. Of every machine attached — this engine and each
|
||||||
|
worker — the node goes to one that can grant what it asked for and has it free.
|
||||||
|
So a node asking for a card finds the box that has one, without naming it:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@node(..., resources={"gpus": 1}) # wherever there is a card
|
||||||
|
@node(..., device="gpu", resources="gpu-small") # that box, and this much of it
|
||||||
|
```
|
||||||
|
|
||||||
|
If nothing attached can ever grant the ask, it is cut down to what is here and
|
||||||
|
runs anyway — a flow written on a cluster still has to work on a laptop. If
|
||||||
|
something *could* but is busy, the node waits and says so.
|
||||||
|
|
||||||
Ask what is free, and what is waiting for it, at
|
Ask what is free, and what is waiting for it, at
|
||||||
`GET /api/v1/workers/resources` — a node queued for cores looks exactly like a
|
`GET /api/v1/workers/resources`, on the Workers screen, or in `fluksio status`.
|
||||||
node that has hung unless you can see the queue.
|
|
||||||
|
|
||||||
!!! note "Accounted, not enforced"
|
!!! note "Accounted, not enforced"
|
||||||
|
|
||||||
Nothing stops a node that ignores its declaration; the numbers are
|
Nothing stops a node that ignores its declaration; the numbers are
|
||||||
bookkeeping plus the environment its libraries read. Scheduling across
|
bookkeeping plus the environment its libraries read. Real enforcement —
|
||||||
several machines, named hardware flavours and real enforcement are the
|
cgroups, rlimits — is a next step, not this one.
|
||||||
next steps, not this one.
|
|
||||||
|
|
||||||
## Say which nodes make a flow
|
## Say which nodes make a flow
|
||||||
|
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ Anything already exported wins over the file.
|
|||||||
| `FLOWS_DIR` | `$DATA_DIR/flows` | the git repository holding flows |
|
| `FLOWS_DIR` | `$DATA_DIR/flows` | the git repository holding flows |
|
||||||
| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials, deliberately outside the repo |
|
| `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials, deliberately outside the repo |
|
||||||
| `ALERTS_FILE` | `$DATA_DIR/alerts.json` | channels and rules |
|
| `ALERTS_FILE` | `$DATA_DIR/alerts.json` | channels and rules |
|
||||||
|
| `PROVISIONERS_FILE` | `$DATA_DIR/provisioners.json` | clusters a machine can be started from; absent means none |
|
||||||
| `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings |
|
| `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings |
|
||||||
| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens |
|
| `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens |
|
||||||
| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any |
|
| `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any |
|
||||||
@@ -141,8 +142,14 @@ A node that declares nothing is not accounted against `FLOW_CPUS`; it runs on
|
|||||||
the shared pool and is given `FLOW_CPUS / FLOW_MAX_WORKERS` as a thread cap, so
|
the shared pool and is given `FLOW_CPUS / FLOW_MAX_WORKERS` as a thread cap, so
|
||||||
several at once cannot each size themselves to the whole machine. Setting
|
several at once cannot each size themselves to the whole machine. Setting
|
||||||
`OMP_NUM_THREADS` (or any of its siblings) on the engine yourself overrides
|
`OMP_NUM_THREADS` (or any of its siblings) on the engine yourself overrides
|
||||||
that default. What is free, and which nodes are queued for it, is
|
that default.
|
||||||
`GET /api/v1/workers/resources`. See
|
|
||||||
|
These two are this machine's figures. An attached worker reports its own when
|
||||||
|
it dials in, and a node goes to whichever machine can grant what it asked for —
|
||||||
|
so a GPU on a worker needs no `FLOW_GPUS` here. What every machine has free, and
|
||||||
|
which nodes are queued, is `GET /api/v1/workers/resources` and the Workers
|
||||||
|
screen. Named sizes live in the database and are `GET /api/v1/flavors`;
|
||||||
|
`PROVISIONERS_FILE` is where machines can be started from. See
|
||||||
[declaring resources](../getting-started/data-science.md#declaring-what-a-node-needs).
|
[declaring resources](../getting-started/data-science.md#declaring-what-a-node-needs).
|
||||||
|
|
||||||
## Agents
|
## Agents
|
||||||
|
|||||||
Reference in New Issue
Block a user