A node's numbers leave through its ports, not a logging call

The first cut had node code call fluksio.log_metric, which was a second,
undeclared way for data to leave a node: invisible to validation, absent from
the canvas, and stored where the graph could not see it. That is precisely the
MLflow discrepancy this framework exists to avoid, so it is gone.

A node that produces values over time is a generator. Every yield is a dict
keyed by output port, published the instant it happens — same port, same type
check, same place on the canvas as any other value — and what it returns is
its result. A port doing this declares stream: true, and a run keeps every
number one takes, so experiment tracking is a consequence of the graph rather
than an API beside it: a chart binds to a training curve the way it binds to a
temperature. fluksio.emit writes the same ports imperatively, for where a
yield cannot reach — inside a training framework's callback.

In a live flow an emission also wakes what is downstream, as a subscriber
publishing does; in a run it does not, because a run's graph is scheduled once
and mid-node cascades would leave 'finished' with nothing to mean. The
enqueued item carries no payload: the value is already in state, and one
carrying it would re-apply an old emission after the node returned.

Verified on the stack: 30 loss values arrived live on the flow socket during a
run, attributed to the node that produced them, and the same node run on the
remote worker streamed its curve back across the socket.

Also caches remote compile results per worker, so attaching a GPU box does not
put a network round trip in every rebuild.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
This commit is contained in:
2026-08-18 20:53:49 +02:00
co-authored by Claude Fable 5
parent 328e3c8e98
commit 774b03953a
14 changed files with 528 additions and 147 deletions
+9 -6
View File
@@ -135,12 +135,15 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend M
separate from the cascade rollups, which are pruned on a retention window
and an experiment must not be. `/runs`, `/runs/{id}`, `/runs/flows/{name}`,
`/sweep`, `/cancel`, `/metrics` and `/series/compare`
- [x] Reporting from inside a running node: node code imports `fluksio` and calls
`log_metric` / `progress` / `save_artifact` mid-call. The worker protocol
carries event lines before the reply, so the metrics of a two-hour training
arrive while it trains rather than with its result — and each event resets
the deadline, which turns `NodeDef.timeout` into an idle timeout for a node
that reports
- [x] Streaming outputs: a node that produces values over time is a generator,
and every `yield` is a dict keyed by output port, published the instant it
happens; what it returns is its result. A port doing this declares
`stream: true`, and a run keeps every number one takes — so a training
curve is an output of the graph rather than a log beside it, and a chart
binds to it like any message. `fluksio.emit` writes the same ports for the
case a yield cannot reach, inside a framework's callback. The worker
protocol carries each emission as a frame before the reply, which also
turns `NodeDef.timeout` into an idle timeout: silence, not duration
- [x] Artifacts: `DType.ARTIFACT` carries a reference (digest, size, media type,
name) into a content-addressed store on the data volume, so bytes never
enter a message, Redis or the queue. The digest is the future stage-cache