An example to evaluate: a training run, its dashboard, and two bugs it found
make seed-demo builds demo_training — prepare on the engine, a GPU-bound train, evaluate back here — and a panel that draws the loss curve while the training is still going. It is the session's whole argument in one flow: batch runs with parameters and a result, a generator yielding on a declared port rather than logging, fluksio.emit from inside a callback, artifacts carrying the dataset and the weights between machines, and a sweep whose configs are isolated from each other. The train node prefers its label rather than requiring it, so it runs before a GPU box exists and says which machine and which numeric backend it actually used. Building it turned up two real bugs. A run waited for a worker its flow only *preferred*, because required_labels ignored device_policy — so the example hung on a label it did not need. And a run's seed never reached the flow, so sweeping over seeds ran the same experiment N times; it now fills an input of that name when the flow declares one, which is what the field looked like it did all along. Pressing Run on a batch flow now submits a run rather than taking the old non-durable path — that button is the first thing anyone evaluating will press, and it was quietly doing something else. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
This commit is contained in:
@@ -13,6 +13,7 @@ should reopen it.
|
||||
|
||||
### To be sorted
|
||||
|
||||
- BUG/UI find a good way to represent the dictionary style of messages (truncated test, click to enlarge.. ). Show dtype instead of actual values -> actual values when clicking expand
|
||||
- BUG/UI loop-lag in home view can't have a value when there are no flows (push back if I'm wrong here)
|
||||
- BUG/UI brain view in the home view should be hidden when there are no flows/nodes
|
||||
- FEAT/UI in addition to the two issues above we should introduce a default flow and dashboard for every new user. These should be very minimal to quickly understand what's happening but still show off most of the features of the application. We could make this optional by offering a "Create Demo" button showing at the location of the brain view if there are no existing flows. This demo could then also serve as a demo.fluksio.com site later to play around without signup
|
||||
@@ -220,6 +221,9 @@ Open on purpose. Each names what should bring it back.
|
||||
- CHORE/INFRA: NATS JetStream as the work-queue backend — durable streams whose consumer semantics match the `WorkQueue` interface, in one small binary. Reopen with M5 remote workers, when the queue crosses hosts. NOTE: remote workers landed without it — a worker dials the engine's own socket and never touches Redis, so the queue still does not cross a host. Reopen if a second engine ever pulls from the same stream.
|
||||
- FEAT/RUNS: stage caching. `run_node.cache_key` is written on every run and the artifact store is content-addressed, so the pieces are in place; what is missing is computing the key from the node's source digest plus its input values and skipping a node whose key already has an `ok` row with its artifacts still present. The two research repos want this more than they want resume — neither persists checkpoints, and both re-run unchanged preprocessing every time.
|
||||
- FEAT/RUNS: per-label requirements overlays (`requirements-gpu.txt`) synced into a remote worker's venv, with drift surfaced against the engine's manifest. Today a worker's environment is whatever `--python` points at, which is fine for one hand-managed GPU box and not for several. `venv_digest` already arrives at attach and is shown on `/workers`, so the reporting half exists.
|
||||
- FEAT/UI: a dashboard shows a run's curve only while it is running. Emissions reach the socket live, but a run's values live in its own state namespace, so reloading the panel afterwards leaves the chart empty — the durable series is on the run (`/runs/{id}/metrics`) and nothing binds a widget to it. A chart variant that reads a run's series, or the existing querying chart pointed at `/runs/series/compare`, is what would close it. This is also what a demo needs to show a finished experiment rather than only a live one.
|
||||
- FEAT/UI: nothing submits a run with parameters from the UI. Pressing Run on a batch flow submits one with the declared defaults, which is enough to try it; choosing parameters, or launching a sweep, is API-only. A form generated from the flow's `inputs` is the obvious shape.
|
||||
- FEAT/UI: `scripts/seed_demo_training.py` (`make seed-demo`) builds a batch flow and dashboard covering runs, streaming outputs, artifacts, device routing and sweeps. It is the content a "Create Demo" button would seed; what it needs to become one is a route that runs it server-side for the signed-in user.
|
||||
- FEAT/RUNS: a run detail screen. The API answers everything — params, per-node status with logs and tracebacks, artifacts, metrics, and `/runs/series/compare` in the chart widget's own `series` shape — but nothing in the dashboard reads it yet, so a run is inspected over HTTP. Comparing curves is a widget binding once someone builds the page around it.
|
||||
- FEAT/RUNS: a thin client CLI (`fluksio run/runs/sweep/worker`) over the same API. The engine being resident is what makes runs cheap; a CLI is ergonomics on top, and `curl` covers it until someone is running sweeps daily.
|
||||
- FEAT/RUNS: the step on a run's series is the count of emissions on that message, so a node yielding every tenth training step records steps 0, 1, 2 rather than 0, 10, 20 — a faithful x-axis of its own emissions, not of the loop inside it. If a real step number ever matters, a `record`-typed streaming port carrying its own `step` is the shape to read it from; the column is already there.
|
||||
|
||||
Reference in New Issue
Block a user