Benchmark the startup claim, and write down what a run is

The milestone is measured on being lighter than Kedro, so make bench-startup
measures it rather than asserting it: 61 ms from submit to result against
1110 ms for kedro run on a pipeline that does the same nothing. The difference
is not orchestration, it is that nothing is booted per run — on a 510-config
sweep that is about nine minutes of pure startup that never happens.

docs/flows/runs.md is the guide: batch flows, sweeps, reporting from inside a
node, artifacts, and the two sanctioned patterns for objects that cannot be
serialized — keep them in one node, or cross at a checkpoint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
This commit is contained in:
2026-08-18 18:00:36 +02:00
co-authored by Claude Fable 5
parent eb2d098d7c
commit 52d4de62c9
4 changed files with 240 additions and 2 deletions
+31
View File
@@ -32,6 +32,11 @@ existing flow engine reachable and persistent precedes new feature breadth.
stack's own containers. Two caveats: the redis scenario stops the container the
whole stack shares, and a cascade finishes fast enough that the engine kill
proves redelivery without stressing it
- [x] Startup benchmark: `backend/scripts/bench_startup.py` behind `make
bench-startup` times submitting a run against a Kedro project doing the same
nothing — 61 ms against 1110 ms, because nothing is booted per run. The
claim the ML-pipeline milestone is measured on, kept checkable rather than
asserted
## Phase 1 — Backend: management
@@ -120,6 +125,32 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend M
cascade — including the manual runs and previews that never went through the
queue. Read back through `/observability/*`, which always answers 200 so a
degraded engine still renders, and pruned on a retention window
- [x] Batch runs: a `mode: batch` flow taken from its declared inputs to its
declared outputs once, with parameters that identify it and a result kept.
Journaled to a Redis stream of its own and Postgres-authoritative from the
claim onwards, so a stale lease — not an unacked entry — marks a run whose
engine died. Each run executes an isolated pipeline over its own state
namespace, so a sweep's configs run in parallel without overwriting each
other's messages. `run`, `run_node`, `run_metric` and `run_artifact` are
separate from the cascade rollups, which are pruned on a retention window
and an experiment must not be. `/runs`, `/runs/{id}`, `/runs/flows/{name}`,
`/sweep`, `/cancel`, `/metrics` and `/series/compare`
- [x] Reporting from inside a running node: node code imports `fluksio` and calls
`log_metric` / `progress` / `save_artifact` mid-call. The worker protocol
carries event lines before the reply, so the metrics of a two-hour training
arrive while it trains rather than with its result — and each event resets
the deadline, which turns `NodeDef.timeout` into an idle timeout for a node
that reports
- [x] Artifacts: `DType.ARTIFACT` carries a reference (digest, size, media type,
name) into a content-addressed store on the data volume, so bytes never
enter a message, Redis or the queue. The digest is the future stage-cache
key, which is why it is content-addressed now rather than per-run
- [x] Remote workers: a worker dials out to `WS /workers/attach` with an RS256
worker-scope token, advertises labels, and answers the same JSON protocol
the local pool speaks. `NodeDef.device` routes a node to one, resolved per
call; a run whose labels nothing carries waits saying so. A device-bound
node is compiled on that machine. The agent is one file plus `worker_main`,
with `websockets` as its only dependency
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
deployment on failure
- [ ] User management scoped per flow and per data set