diff --git a/docs/code/cli.md b/docs/code/cli.md index 17f838f..01b7f9f 100644 --- a/docs/code/cli.md +++ b/docs/code/cli.md @@ -251,6 +251,10 @@ what is wrong if not, whether it is paired with a portal and reaching it, then every flow with its state, its node count and whether it has unpublished changes, and the last few runs and failures under them. +A `resources` line names each machine the engine can run a node on and how much +of it is in use, plus how many nodes are queued for one. It is absent on an +engine that accounts for nothing. + The portal reads one of three ways. `no portal` means this installation was never enrolled. `portal hub.fluksio.com` means the link is up. `portal unreachable` names the error, and is the one worth acting on — the dashboard is @@ -278,6 +282,16 @@ kB of JSON still lists as a table; `Client.runs()` is where the whole value is read. `--local` reads the same history from an in-process engine, without one having to be served. +### `fluksio flavors` + +```sh +fluksio flavors +``` + +The named sizes a node can ask for — `@node(resources="gpu-small")` — with the +cores, memory and cards each stands for. Editing them is the Workers screen or +`POST /api/v1/flavors`; this is the read. + ### `fluksio sweep` ```sh diff --git a/docs/getting-started/data-science.md b/docs/getting-started/data-science.md index 6684ec2..3bfb8f3 100644 --- a/docs/getting-started/data-science.md +++ b/docs/getting-started/data-science.md @@ -299,16 +299,51 @@ threshold, a message on its way somewhere. Those share the engine's worker pool and are given a fair share of `FLOW_CPUS` as a thread cap, which is what stops several of them at once from each sizing to the whole box. +### Ask for a size by name + +Cores and gigabytes are a property of the machines you have, and those change. +A node that names a **flavor** keeps meaning something afterwards: + +```python +@node(..., resources="gpu-small") +def finetune(checkpoint): + ... + +@node(..., resources={"flavor": "medium", "duration_s": "2h"}) +def fit(dataset, lr, epochs=25): + ... +``` + +The flavor is read every time the node is built, so editing it changes what the +next run gets. `fluksio flavors` lists them; the Workers screen edits them. + +`ram` takes `"2G"` and `duration_s` takes `"30m"`, and a flavor already says how +much — pass one or the numbers, not both. `duration_s` is a statement about the +node for whoever is planning around it, not a limit; the limit is `timeout`. + +### Where it runs + +This is one decision, not two. Of every machine attached — this engine and each +worker — the node goes to one that can grant what it asked for and has it free. +So a node asking for a card finds the box that has one, without naming it: + +```python +@node(..., resources={"gpus": 1}) # wherever there is a card +@node(..., device="gpu", resources="gpu-small") # that box, and this much of it +``` + +If nothing attached can ever grant the ask, it is cut down to what is here and +runs anyway — a flow written on a cluster still has to work on a laptop. If +something *could* but is busy, the node waits and says so. + Ask what is free, and what is waiting for it, at -`GET /api/v1/workers/resources` — a node queued for cores looks exactly like a -node that has hung unless you can see the queue. +`GET /api/v1/workers/resources`, on the Workers screen, or in `fluksio status`. !!! note "Accounted, not enforced" Nothing stops a node that ignores its declaration; the numbers are - bookkeeping plus the environment its libraries read. Scheduling across - several machines, named hardware flavours and real enforcement are the - next steps, not this one. + bookkeeping plus the environment its libraries read. Real enforcement — + cgroups, rlimits — is a next step, not this one. ## Say which nodes make a flow diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 1c61895..c367e79 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -19,6 +19,7 @@ Anything already exported wins over the file. | `FLOWS_DIR` | `$DATA_DIR/flows` | the git repository holding flows | | `SECRETS_FILE` | `$DATA_DIR/secrets.enc` | encrypted credentials, deliberately outside the repo | | `ALERTS_FILE` | `$DATA_DIR/alerts.json` | channels and rules | +| `PROVISIONERS_FILE` | `$DATA_DIR/provisioners.json` | clusters a machine can be started from; absent means none | | `PANELS_FILE` | `$DATA_DIR/panels.json` | wall-panel pairings | | `OAUTH_PRIVATE_KEY_FILE` | `$DATA_DIR/oauth-key.pem` | signs agent and worker tokens | | `CLOUD_CONFIG_FILE` | `$DATA_DIR/cloud.json` | the portal enrolment, if any | @@ -141,8 +142,14 @@ A node that declares nothing is not accounted against `FLOW_CPUS`; it runs on the shared pool and is given `FLOW_CPUS / FLOW_MAX_WORKERS` as a thread cap, so several at once cannot each size themselves to the whole machine. Setting `OMP_NUM_THREADS` (or any of its siblings) on the engine yourself overrides -that default. What is free, and which nodes are queued for it, is -`GET /api/v1/workers/resources`. See +that default. + +These two are this machine's figures. An attached worker reports its own when +it dials in, and a node goes to whichever machine can grant what it asked for — +so a GPU on a worker needs no `FLOW_GPUS` here. What every machine has free, and +which nodes are queued, is `GET /api/v1/workers/resources` and the Workers +screen. Named sizes live in the database and are `GET /api/v1/flavors`; +`PROVISIONERS_FILE` is where machines can be started from. See [declaring resources](../getting-started/data-science.md#declaring-what-a-node-needs). ## Agents