Close eight open SDK tasks: the pidfile, the log, cards, names and a live curve
Each was a loose end recorded under `### SDK` in the notepad. `serve` takes its own pidfile down on SIGTERM. uvicorn restores the handler it found and re-raises the signal it stopped on, so the default handler ended the process without unwinding and the `finally` never ran — which is what a stop sends, and what left `serve.pid` behind. `serve.log` is cut back past 5 MB by the engine rather than by the screen that started it, so an adopted engine is bounded too. Gated on its own stdout being an appended regular file, which is what makes the cut safe: the kernel then puts the next write at the new end. Cards are counted from `/dev/nvidia[0-9]*`, so `FLOW_GPUS`/`--gpus` of 0 means "work it out" the way `FLOW_CPUS` always has. The engine counts, not the accountant — a remote worker builds one of those from its own inventory, and detecting there would hand it the engine host's cards. The worker counts last: what a batch job says it was granted still wins. `GET /runs/metrics/names` is the distinct over a selection that `--list` and the terminal's metric picker were approximating by reading the newest run that had measured anything, which missed a name only an older run ever wrote. `MetricSink` announces each batch it has written (`run_metric`, carrying the names). Not a per-point event: one covers up to 500 points or two seconds of them, and the rows stay the record. The terminal comparison fills in as the first readings land instead of staying blank until reopened, and the browser refetches the run and any comparison rather than the list behind them. `retry --group` pages the list route by `before` instead of stopping at 500. The terminal dashboard takes the terminal's colours (`ansi-dark`), and the web UI can re-pair from Settings without disconnecting first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRQ9bmTvCbqCwXo9mxZzzV
This commit is contained in:
+10
-7
@@ -36,7 +36,7 @@ driver in order to run a training step. An engine host already has it, and
|
||||
| `--parallel` | `1` | how many node calls it will take at once |
|
||||
| `--artifact-url` | derived from `--url` | where the artifact store is, if not beside the socket |
|
||||
| `--cpus` | what the job or the machine has | cores to advertise |
|
||||
| `--gpus` | what the job says, else none | GPUs to advertise; never probed |
|
||||
| `--gpus` | what the job says, else counted | GPUs to advertise |
|
||||
| `--ram-mb` | what the job or the machine has | memory to advertise, in MB |
|
||||
| `--max-idle` | never | stop after this many seconds with nothing running |
|
||||
|
||||
@@ -51,12 +51,15 @@ the engine schedules against it: a node asking for two cores and a GPU goes to
|
||||
a machine that has them free, not merely to one carrying the right label.
|
||||
|
||||
Cores and memory are read off the machine, or off the batch job that started
|
||||
this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **GPUs are never
|
||||
probed.** Asking a vendor tool would make the one dependency two, so a GPU is
|
||||
something the job says it was given (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`,
|
||||
or `FLUKSIO_WORKER_GPUS`) or something you say with `--gpus`. A worker that
|
||||
reports nothing still attaches and is scheduled by its label alone, as every
|
||||
worker was before any of them reported anything.
|
||||
this worker (`SLURM_CPUS_ON_NODE`, `SLURM_MEM_PER_NODE`). **What the job says
|
||||
it was given always wins for GPUs** (`SLURM_GPUS_ON_NODE`, `SLURM_JOB_GPUS`,
|
||||
or `FLUKSIO_WORKER_GPUS`): a node with eight cards may have granted this job
|
||||
one, and advertising eight would be a lie the scheduler acts on. With nothing
|
||||
said, NVIDIA's device nodes are counted, the same as the engine does for its
|
||||
own machine, and `--gpus` overrides either. No vendor tool is asked, which is
|
||||
what keeps the one dependency from becoming two. A worker that reports nothing
|
||||
still attaches and is scheduled by its label alone, as every worker was before
|
||||
any of them reported anything.
|
||||
|
||||
The engine tells each call what it may use: thread caps, and the devices it
|
||||
may see. The worker starts a process per call, so it applies them at the only
|
||||
|
||||
Reference in New Issue
Block a user