app/ROADMAP.md

# Roadmap

Component-level breakdown. The milestone-level master (M1–M5, with the vision
decisions behind it) is `docs/private/roadmap.md` in the docs submodule.

Implementation strategy and record of existing/planned features. Completed items are
terse checklists — the requirement detail lives in `docs/private/vision.md` (goals,
requirements, decisions) and `docs/architecture/structure.canvas` (the four-way component split).
Remaining tasks keep enough scope to be actionable.

Legend: `[x]` done · `[ ]` planned · sub-lists split done vs. remaining for partial items.

Within each phase, remaining `[ ]` items are listed in rough priority order: making the
existing flow engine reachable and persistent precedes new feature breadth.

## Phase 0 — Workspace and platform

- [x] Root orchestrator repo with `app`, `index` and `docs` as submodules
- [x] `make init` bootstrap: secrets generation, per-stack `.env` propagation, shared
      `proxy` docker network
- [x] Layered compose (`compose.yml` → `compose.dev.yml` → `compose.local.yml`) for both
      stacks, one Traefik serving `${DOMAIN}`, `app.${DOMAIN}`, `api.${DOMAIN}`
- [x] Design token contract: root `DESIGN-GUIDELINES.md`, per-repo `DESIGN.md`,
      byte-identical token blocks verified by `make design-check`
- [x] CI on Gitea (Forgejo Actions): pre-commit, backend tests, Playwright, compose smoke
      — the four workflows in `.gitea/workflows/`. Never observed running on a
      Gitea push from here, so treat the wiring as written but unproven
- [x] Soak and chaos harness: sustained load with the state backend, the broker and
      the engine itself taken away underneath it. `backend/scripts/soak.py` behind
      `make soak`, driving the durable webhook path through load, redis, engine and
      broker scenarios, with every docker verb checked against an allow-list of this
      stack's own containers. Two caveats: the redis scenario stops the container the
      whole stack shares, and a cascade finishes fast enough that the engine kill
      proves redelivery without stressing it
- [x] Startup benchmark: `backend/scripts/bench_startup.py` behind `make
      bench-startup` times submitting a run against a Kedro project doing the same
      nothing — 61 ms against 1110 ms, because nothing is booted per run. The
      claim the ML-pipeline milestone is measured on, kept checkable rather than
      asserted

## Phase 1 — Backend: management

Python, optimised for development speed. Owns the graph structure, persistence and the
external interfaces. See `docs/architecture/structure.canvas` → *Backend – Management*.

- [x] FastAPI + SQLModel + Alembic + Postgres base with JWT auth and user management
- [x] Flow engine in `backend/app/flow/`: `Node` / `Pipeline` / `StateBackend`
      (memory + Redis) / `FlowController`
- [x] Node types: HTTP, MQTT, InfluxDB, Delay, MLP
- [x] Flow-logic vocabulary as node types rather than repeated code: inject (manual,
      interval, cron or at startup), switch, change, filter-unchanged, join, trigger,
      command, file and ntfy. Each is configured by filling in a form the editor
      generates from its parameter schema
- [x] `app/flow` is an importable package with absolute `app.flow.*` imports
- [x] Typed, serializable node I/O: every port declares a `DType`, messages are
      JSON on the wire and in Redis, no pickle anywhere. Binary codecs are still
      open — `DType.JSON` carries everything non-scalar for now
- [x] Message namespacing per flow (`flow.message`), with several producers per
      message resolving to real fan-in
- [x] Secrets/credentials store for node integrations managed via the API/UI
      (encrypted at rest, referenced from node params as `{"$secret": "name"}`);
      `.env` bootstrap-only
- [x] Python modules for node code, managed from the UI: a pip manifest versioned
      with the flows, installed with `uv pip sync` into a venv of the user's own on
      the data volume. The worker processes run that interpreter, so an install
      takes effect without restarting the engine and can never shadow the app's
      own packages
- [x] Connector node contract: `ConnectorNode` with a declared contract version,
      a polling coordinator that deduplicates, `x-secret` parameters the editor
      renders as a secret picker, and health reporting. Connectors are installed
      packages found through the `fluksio.node_types` entry point group; the
      contract is documented in `docs/connectors/` with a working skeleton at
      `connector-skeleton/`. The registry follows later
- [x] First real connectors written against that contract from outside the engine:
      WF-RAC aircon, calendar, UniFi presence and Art-Net, in `connectors/`.
      Built by `make connectors` and installed into the image. Three of the four
      read only; the aircon package still cannot produce a command
- [x] The other direction of the contract: `ConnectorNode.write` receives the
      node's input ports, so a connector can command something rather than only
      read it. Additive, so `CONTRACT_VERSION` stays at 1 — before this the base
      class discarded every message reaching a connector, which made `artnet`'s
      packet builder unreachable. Art-Net now sends: a per-port `channels` map
      puts each input on its own DMX channel, `transmit` still gates the socket,
      and one node owns one universe because a frame carries all 512 levels
- [x] Node lifecycle as a protocol (`start`/`stop`/`report_health` on `Node`),
      replacing the controller's per-type isinstance chains — the same hooks a
      connector implements, validated on the built-in nodes first
- [x] Flow persistence: `flow.json` plus node sources per flow, replacing the
      watch-directory prototype
- [x] REST + WebSocket API over the engine: create/read/update flows, edit node
      source, run, and stream values, node status and execution events
- [x] Dependency-loop detection and graph validation surfaced as API errors
- [x] Per-flow start/stop, stored in a `runtime.json` beside the flow so it
      survives a restart and stays out of the autosaved document; pause/resume
      holds a flow's nodes while its values keep arriving
- [x] Node log streaming: what a node prints, and the traceback of one that
      fails, reach the editor as `node_log` events
- [x] MQTT broker / InfluxDB compose services for local development
      (`mosquitto` and `influxdb` in `docker/compose.dev.yml`)
- [x] Git-based versioning of the flow store (one commit per saved change)
- [x] Draft/publish split: edits autosave to `flow.draft.json` / `nodes.draft/`,
      the engine runs only the published files, and publishing promotes the
      draft. Saves carry the version they were based on, so a second client
      editing the same flow is refused rather than overwritten
- [ ] Import/export of a flow as human-readable code plus a JSON structure
- [x] Per-input/-output discretization interval setting: a port publishes, or
      wakes its node, at most every n seconds. State keeps the latest value, so
      only the delivery is skipped
- [x] Alert / notification handler: engine failures — a node raising, a connection
      dropping, a flow quarantined, the queue gone — reach ntfy, email or a webhook.
      Mostly it declines to send: the same fault repeating is one alert with a count,
      a flapping connection is muted, and there is a ceiling per hour. Configured
      through the API at `/alerts/config`, with a test send per channel
- [x] Deep health check (`GET /utils/health/`): reports event-loop lag and state-backend
      reachability and fails the container healthcheck, so a wedged engine is restarted
      rather than counted as up. One engine per deployment — the API image runs a single
      worker, because a second one would be a second engine
- [x] Supervised background tasks: a node's subscription, schedule or poll loop is
      restarted with growing delay when it dies, and a flow that spends its failure
      budget is quarantined and surfaced rather than left crash-looping. The loops
      themselves no longer carry private retry logic
- [x] Durable work queue: every external trigger is journaled to Redis Streams before
      anything runs and acknowledged once its cascade finishes, so an engine that dies
      mid-cascade picks the work up again instead of losing it. A reaper reclaims what
      a dead consumer never acknowledged; nodes that reach outside are skipped on a
      redelivery they already ran. Long-lived worker pools replace the per-wave
      executors, and a delay now waits in the queue rather than on a worker thread
- [x] Engine history in Postgres: a second bus subscriber folds executions, errors,
      timings and queue lag into per-minute rollups, keeps failures with their
      traceback and an audit trail of who published what, and records one row per
      cascade — including the manual runs and previews that never went through the
      queue. Read back through `/observability/*`, which always answers 200 so a
      degraded engine still renders, and pruned on a retention window
- [x] Batch runs: a `mode: batch` flow taken from its declared inputs to its
      declared outputs once, with parameters that identify it and a result kept.
      Journaled to a Redis stream of its own and Postgres-authoritative from the
      claim onwards, so a stale lease — not an unacked entry — marks a run whose
      engine died. Each run executes an isolated pipeline over its own state
      namespace, so a sweep's configs run in parallel without overwriting each
      other's messages. `run`, `run_node`, `run_metric` and `run_artifact` are
      separate from the cascade rollups, which are pruned on a retention window
      and an experiment must not be. `/runs`, `/runs/{id}`, `/runs/flows/{name}`,
      `/sweep`, `/cancel`, `/metrics` and `/series/compare`
- [x] Streaming outputs: a node that produces values over time is a generator,
      and every `yield` is a dict keyed by output port, published the instant it
      happens; what it returns is its result. A port doing this declares
      `stream: true`, and a run keeps every number one takes — so a training
      curve is an output of the graph rather than a log beside it, and a chart
      binds to it like any message. `fluksio.emit` writes the same ports for the
      case a yield cannot reach, inside a framework's callback. The worker
      protocol carries each emission as a frame before the reply, which also
      turns `NodeDef.timeout` into an idle timeout: silence, not duration
- [x] Artifacts: `DType.ARTIFACT` carries a reference (digest, size, media type,
      name) into a content-addressed store on the data volume, so bytes never
      enter a message, Redis or the queue. The digest is the future stage-cache
      key, which is why it is content-addressed now rather than per-run
- [x] Remote workers: a worker dials out to `WS /workers/attach` with an RS256
      worker-scope token, advertises labels, and answers the same JSON protocol
      the local pool speaks. `NodeDef.device` routes a node to one, resolved per
      call; a run whose labels nothing carries waits saying so. A device-bound
      node is compiled on that machine. The agent is one file plus `worker_main`,
      with `websockets` as its only dependency
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
      deployment on failure
- [ ] User management scoped per flow and per data set
- [x] MCP server over the same API: agents authenticate through a built-in
      OAuth 2.1 authorization server (dynamic registration, PKCE, rotating
      refresh tokens) and drive the flow API through 20 tools. Tokens are
      RS256, signed with their own keypair, so the set can be revoked on its
      own — and an additional issuer is one branch in `deps.decode_token`,
      which is the seam remote access needs later
- [ ] LLM interface for natural-language flow authoring beyond the MCP tools

## Phase 2 — Backend: processing

Rust, optimised for throughput. Executes nodes and distributes them across workers. See
`docs/architecture/structure.canvas` → *Backend – Processing*.

- [ ] Parallel invocation of stateless nodes over independent input sets, to
      keep I/O delay minimal (stateful I/O nodes keep serializing via the
      `synchronous` mechanism)
- [x] Run user Python nodes out of process: a pool of persistent worker subprocesses
      speaking one JSON object per line, entered through a proxy the controller
      installs as the node's function, so every execution path funnels through it
      unchanged. A crash costs one subprocess, a per-node timeout is a kill, and
      cancelling from the canvas is that same kill on request
- [ ] Extract node execution from the Python prototype into a Rust engine
- [ ] Worker distribution and load balancing across capable devices
- [ ] Input/output validation at the node boundary
- [ ] Data aggregation and discretization

## Phase 3 — Frontend: admin view

React + Vite, primarily desktop but usable on mobile. See `docs/architecture/structure.canvas` →
*Frontend – Admin View*.

- [x] Dashboard SPA shell: TanStack Router, floating frosted sidebar, auth flows,
      generated OpenAPI SDK
- [x] Node canvas (`@xyflow/react`) showing nodes and their connections, which
      are derived from message names rather than stored
- [x] Tab-style view of atomic flows, with a floating dock
- [x] Embedded code editor (Monaco) for node source
- [x] Live values on the edges, with the last payload and its time on click
- [x] Provenance: every value says what caused it, so an edge pulses for the producer
      that actually published rather than every producer of that message. A dashboard
      control, another flow or an API caller is drawn as a label on the canvas instead
      of being invisible — which also gives cross-flow wiring the link in/out it lacked
- [x] A node's settings are arguments of its function, next to its ports — one
      `process(...)` signature covering both, no `params` dict, and a setting
      sharing a port's name reported rather than shadowing it
- [x] A flow's boundary is on the canvas and in its panel: each declared input
      is a label feeding what reads it (unless a control or another flow already
      accounts for it), a batch flow's outputs hang off the end as its result,
      and the panel edits mode, inputs and result — with the value a live flow
      currently holds, and a way to put a new one in
- [x] Pressing Run on a batch flow asks for its parameters, prefilled from what
      each input starts from, rather than quietly submitting the defaults
- [x] Validation shown on the node it belongs to, and summarised in the dock
- [x] Publish control and draft markers in the flow bar, discard in the flow
      panel, and a conflict dialog when another client got there first
- [x] Marking a node reusable, and placing a shared one from the palette
- [x] Secret picker for credential parameters, so a password never lands in
      `flow.json`
- [x] Dashboard showing which flows run, which are stopped and which have
      errors, with a switch per flow
- [x] Logs panel in the canvas dock, pause/resume beside Run, and replaying an
      edge's last message from the inspector
- [x] The canvas lays itself out — a layered graph, left to right on a desktop
      and top to bottom on a phone, with room reserved for the value each edge
      carries. Nodes cannot be dragged and a flow document holds no positions:
      a graph nobody can arrange is one worth keeping small, which is what
      keeps flows atomic
- [x] Usable on a phone, and written down so it stays that way: one breakpoint
      (`md`), a stacked dashboard instead of a shrunken wall panel, a dock that
      wraps rather than overflows, and a Playwright project that fails the
      build when a screen no longer fits. See DESIGN-GUIDELINES.md → Responsive
- [ ] Device assignment per node, selectable from compatible devices
- [ ] Test-node affordance on the canvas
- [ ] User management screens
- [x] Screens for what the API used to own alone: the secrets store and the alert
      channels/rules each get a sidebar page, and the OAuth clients an agent
      registers are listed and revocable under Admin — which needed its
      management endpoints written first
- [x] Health sections: how the engine is doing now (nodes, flows, queue, loop lag)
      over what it has been doing all day — throughput and failure charts, a
      per-flow table, the recent cascades, failures that expand to their
      traceback, dead-lettered work and the audit trail. Hovering a chart
      filters the list beside it to that minute and a click pins it
- [x] Brain graph: every published flow at once, with nodes that talk to the
      same outside thing — a broker topic, a URL, a bucket — drawn as a single
      neuron, so the wiring that runs between flows through a broker is visible at
      all. Laid out by a force simulation settled once and then frozen, lit by the
      same socket the editor listens to, and read-only: a neuron leads back to the
      flow it came from
- [x] Both of the above sit on Home rather than at routes of their own: the brain
      flat across the top, the health sections under the flow switches. One
      overview instead of three
- [x] Mobile-friendly canvas: touch connect, full-screen node panel
- [ ] Installable as a PWA (`vite-plugin-pwa`)

## Phase 4 — Frontend: dashboard view

Shares components with the admin view. See `docs/architecture/structure.canvas` →
*Frontend – Dashboard View*.

- [x] User-defined dashboard layout with edit and view modes: dashboards are their own
      documents, widgets bind to message names, and the input ones publish back. View
      mode is plain CSS grid, so a panel that only displays loads no editing code
- [x] Chart widget drawing a message's history through uPlot, with `--chart-1…5`
      as one lightness ramp of the brand hue; a widget bound to the wrong dtype,
      or to nothing, is flagged the way a failing node is
- [x] Layout by dragging and resizing (react-grid-layout), a grid size per
      dashboard, and `/view/{name}` — a full-bleed route that loads neither the
      editor nor the grid library, which is what a wall panel is pointed at
- [x] Draft and publish, as flows have it: the editor autosaves
      `dashboard.draft.json` and a panel reads only the published document, so a
      half-arranged page never reaches the wall until someone publishes it
- [x] Declared structured payloads — `series`, `record` and `list` (with its item
      type) join the scalars in `DType` and are checked the same way, so a widget
      binds a shape rather than "some JSON" and a wrong binding is refused before
      anything runs. An agenda over a `list` and a notification over a `record`
      are the first two to read them, the latter fed either by a flow or by the
      `dashboard` alert channel
- [x] Charts that query: a chart publishes a request — the window and the
      resolution — exactly as a slider publishes a value, and draws the `series`
      a flow answers with. The answer says what it was computed for and one
      computed for another window is ignored, so two charts on a node cost a
      duplicate query rather than the wrong picture. Database nodes stay
      transport and credentials only: the InfluxDB node runs Flux handed to it
      and echoes the rest, and Python nodes either side build the query and shape
      the answer — which is what keeps the widget ignorant of the database
- [x] Per-device view: a *panel* is one screen and the ordered set of whole
      dashboards it shows, so a hallway tablet and a workshop tablet carry
      different sets without either dashboard knowing about the other. More than
      one and the device draws a rail to switch between them — the same rail the
      editor puts on screen, because the wall has it and it takes room off the
      canvas. A screen has no keyboard, so it pairs instead of logging in: it
      shows a six-character code, somebody approves it against a panel from the
      dashboards overview, and the credential that mints is scoped to that
      panel's dashboards and the message endpoints its widgets speak. Deleting
      the panel revokes it
- [ ] Pair a panel through the portal, for a screen hanging somewhere the
      installation is not reachable from. Three gates today: the hub's app shell
      redirects a browser carrying no portal session, its proxy authorizes before
      forwarding so the unauthenticated pairing call never crosses the tunnel,
      and the app under a portal takes its bearer from the injected config rather
      than from storage. A hub token with a `panel` scope opens all three —
      `decode_portal_token` already branches on scope, and `_panel_may` then
      applies unchanged, so the installation stays the authority on what the
      panel reads. The owner check in the hub's `_authorize` is what a panel
      token must not fall through, and the rate-limit key beside it assumes a
      user id too. Milestone-level rationale, and the alternative worth weighing
      first, are in `docs/private/roadmap.md` § M5

## Phase 5 — Website and docs

- [x] Marketing site with a live node-graph demo, shared design system
- [ ] Published documentation site fed from the `docs` submodule
- [ ] Umami analytics configured (the site still ships the placeholder script)