A node's numbers leave through its ports, not a logging call
The first cut had node code call fluksio.log_metric, which was a second, undeclared way for data to leave a node: invisible to validation, absent from the canvas, and stored where the graph could not see it. That is precisely the MLflow discrepancy this framework exists to avoid, so it is gone. A node that produces values over time is a generator. Every yield is a dict keyed by output port, published the instant it happens — same port, same type check, same place on the canvas as any other value — and what it returns is its result. A port doing this declares stream: true, and a run keeps every number one takes, so experiment tracking is a consequence of the graph rather than an API beside it: a chart binds to a training curve the way it binds to a temperature. fluksio.emit writes the same ports imperatively, for where a yield cannot reach — inside a training framework's callback. In a live flow an emission also wakes what is downstream, as a subscriber publishing does; in a run it does not, because a run's graph is scheduled once and mid-node cascades would leave 'finished' with nothing to mean. The enqueued item carries no payload: the value is already in state, and one carrying it would re-apply an old emission after the node returned. Verified on the stack: 30 loss values arrived live on the flow socket during a run, attributed to the node that produced them, and the same node run on the remote worker streamed its curve back across the socket. Also caches remote compile results per worker, so attaching a GPU box does not put a network round trip in every rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
This commit is contained in:
@@ -222,6 +222,8 @@ Open on purpose. Each names what should bring it back.
|
||||
- FEAT/RUNS: per-label requirements overlays (`requirements-gpu.txt`) synced into a remote worker's venv, with drift surfaced against the engine's manifest. Today a worker's environment is whatever `--python` points at, which is fine for one hand-managed GPU box and not for several. `venv_digest` already arrives at attach and is shown on `/workers`, so the reporting half exists.
|
||||
- FEAT/RUNS: a run detail screen. The API answers everything — params, per-node status with logs and tracebacks, artifacts, metrics, and `/runs/series/compare` in the chart widget's own `series` shape — but nothing in the dashboard reads it yet, so a run is inspected over HTTP. Comparing curves is a widget binding once someone builds the page around it.
|
||||
- FEAT/RUNS: a thin client CLI (`fluksio run/runs/sweep/worker`) over the same API. The engine being resident is what makes runs cheap; a CLI is ergonomics on top, and `curl` covers it until someone is running sweeps daily.
|
||||
- FEAT/RUNS: the step on a run's series is the count of emissions on that message, so a node yielding every tenth training step records steps 0, 1, 2 rather than 0, 10, 20 — a faithful x-axis of its own emissions, not of the loop inside it. If a real step number ever matters, a `record`-typed streaming port carrying its own `step` is the shape to read it from; the column is already there.
|
||||
- CHORE/RUNS: an emission publishes on the node's port and, in a live flow, enqueues a cascade with no payload of its own — the value is already in state, and an item carrying it would re-apply that value whenever it was claimed, which is how a mid-node emission overwrites the one the node returned at the end. Downstream therefore reads what is current rather than the value that caused it to run. Right for a curve; worth revisiting if something ever needs every intermediate value delivered rather than sampled.
|
||||
- CHORE/RUNS: `run_metric` has no retention. Deliberately outside `OBS_RETENTION_DAYS` — an experiment nobody deleted should not vanish on a rollup window — but a few thousand runs at 3000 steps will want a policy eventually, probably per-flow rather than global.
|
||||
- CHORE/RUNS: a run holds one worker slot per node for its whole duration, and `MAX_PARALLEL` run drivers bound how many graphs are in flight. A sweep of 500 therefore queues behind the pool rather than the driver count. Fine — the GPU is the scarce thing — but the two limits are unrelated numbers that read as if they were one.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user