app/ROADMAP.md

Roadmap

Component-level breakdown. The milestone-level master (M1–M5, with the vision decisions behind it) is docs/private/roadmap.md in the docs submodule.

Implementation strategy and record of existing/planned features. Completed items are terse checklists — the requirement detail lives in docs/private/vision.md (goals, requirements, decisions) and docs/architecture/structure.canvas (the four-way component split). Remaining tasks keep enough scope to be actionable.

Legend: [x] done · [ ] planned · sub-lists split done vs. remaining for partial items.

Within each phase, remaining [ ] items are listed in rough priority order: making the existing flow engine reachable and persistent precedes new feature breadth.

Phase 0 — Workspace and platform

• Root orchestrator repo with app, index and docs as submodules
• make init bootstrap: secrets generation, per-stack .env propagation, shared proxy docker network
• Layered compose (compose.yml → compose.dev.yml → compose.local.yml) for both stacks, one Traefik serving ${DOMAIN}, app.${DOMAIN}, api.${DOMAIN}
• Design token contract: root DESIGN-GUIDELINES.md, per-repo DESIGN.md, byte-identical token blocks verified by make design-check
• CI on Codeberg (Forgejo Actions): pre-commit, backend tests, Playwright, compose smoke — the four workflows in .gitea/workflows/. Never observed running on a Codeberg push from here, so treat the wiring as written but unproven
• Soak and chaos harness: sustained load with the state backend, the broker and the engine itself taken away underneath it. backend/scripts/soak.py behind make soak, driving the durable webhook path through load, redis, engine and broker scenarios, with every docker verb checked against an allow-list of this stack's own containers. Two caveats: the redis scenario stops the container the whole stack shares, and a cascade finishes fast enough that the engine kill proves redelivery without stressing it

Phase 1 — Backend: management

Python, optimised for development speed. Owns the graph structure, persistence and the external interfaces. See docs/architecture/structure.canvas → Backend – Management.

• FastAPI + SQLModel + Alembic + Postgres base with JWT auth and user management
• Flow engine in backend/app/flow/: Node / Pipeline / StateBackend (memory + Redis) / FlowController
• Node types: HTTP, MQTT, InfluxDB, Delay, MLP
• Flow-logic vocabulary as node types rather than repeated code: inject (manual, interval, cron or at startup), switch, change, filter-unchanged, join, trigger, command, file and ntfy. Each is configured by filling in a form the editor generates from its parameter schema
• app/flow is an importable package with absolute app.flow.* imports
• Typed, serializable node I/O: every port declares a DType, messages are JSON on the wire and in Redis, no pickle anywhere. Binary codecs are still open — DType.JSON carries everything non-scalar for now
• Message namespacing per flow (flow.message), with several producers per message resolving to real fan-in
• Secrets/credentials store for node integrations managed via the API/UI (encrypted at rest, referenced from node params as {"$secret": "name"}); .env bootstrap-only
• Python modules for node code, managed from the UI: a pip manifest versioned with the flows, installed with uv pip sync into a venv of the user's own on the data volume. The worker processes run that interpreter, so an install takes effect without restarting the engine and can never shadow the app's own packages
• Connector node contract: ConnectorNode with a declared contract version, a polling coordinator that deduplicates, x-secret parameters the editor renders as a secret picker, and health reporting. Connectors are installed packages found through the fluksio.node_types entry point group; the contract is documented in docs/connectors/ with a working skeleton at connector-skeleton/. The registry follows later
• First real connectors written against that contract from outside the engine: WF-RAC aircon, calendar, UniFi presence and Art-Net, in connectors/. Built by make connectors and installed into the image. Reading only for now — the aircon package cannot produce a command and Art-Net keeps its packets off the wire until transmit is switched on
• Node lifecycle as a protocol (start/stop/report_health on Node), replacing the controller's per-type isinstance chains — the same hooks a connector implements, validated on the built-in nodes first
• Flow persistence: flow.json plus node sources per flow, replacing the watch-directory prototype
• REST + WebSocket API over the engine: create/read/update flows, edit node source, run, and stream values, node status and execution events
• Dependency-loop detection and graph validation surfaced as API errors
• Per-flow start/stop, stored in a runtime.json beside the flow so it survives a restart and stays out of the autosaved document; pause/resume holds a flow's nodes while its values keep arriving
• Node log streaming: what a node prints, and the traceback of one that fails, reach the editor as node_log events
• MQTT broker / InfluxDB compose services for local development (mosquitto and influxdb in docker/compose.dev.yml)
• Git-based versioning of the flow store (one commit per saved change)
• Draft/publish split: edits autosave to flow.draft.json / nodes.draft/, the engine runs only the published files, and publishing promotes the draft. Saves carry the version they were based on, so a second client editing the same flow is refused rather than overwritten
• Import/export of a flow as human-readable code plus a JSON structure
• Per-input/-output discretization interval setting: a port publishes, or wakes its node, at most every n seconds. State keeps the latest value, so only the delivery is skipped
• Alert / notification handler: engine failures — a node raising, a connection dropping, a flow quarantined, the queue gone — reach ntfy, email or a webhook. Mostly it declines to send: the same fault repeating is one alert with a count, a flapping connection is muted, and there is a ceiling per hour. Configured through the API at /alerts/config, with a test send per channel
• Deep health check (GET /utils/health/): reports event-loop lag and state-backend reachability and fails the container healthcheck, so a wedged engine is restarted rather than counted as up. One engine per deployment — the API image runs a single worker, because a second one would be a second engine
• Supervised background tasks: a node's subscription, schedule or poll loop is restarted with growing delay when it dies, and a flow that spends its failure budget is quarantined and surfaced rather than left crash-looping. The loops themselves no longer carry private retry logic
• Durable work queue: every external trigger is journaled to Redis Streams before anything runs and acknowledged once its cascade finishes, so an engine that dies mid-cascade picks the work up again instead of losing it. A reaper reclaims what a dead consumer never acknowledged; nodes that reach outside are skipped on a redelivery they already ran. Long-lived worker pools replace the per-wave executors, and a delay now waits in the queue rather than on a worker thread
• Engine history in Postgres: a second bus subscriber folds executions, errors, timings and queue lag into per-minute rollups, keeps failures with their traceback and an audit trail of who published what, and records one row per cascade — including the manual runs and previews that never went through the queue. Read back through /observability/*, which always answers 200 so a degraded engine still renders, and pruned on a retention window
• Test nodes: a small node dragged onto an existing one, smoke or unit, blocking deployment on failure
• User management scoped per flow and per data set
• MCP server over the same API: agents authenticate through a built-in OAuth 2.1 authorization server (dynamic registration, PKCE, rotating refresh tokens) and drive the flow API through 20 tools. Tokens are RS256, signed with their own keypair, so the set can be revoked on its own — and an additional issuer is one branch in deps.decode_token, which is the seam remote access needs later
• LLM interface for natural-language flow authoring beyond the MCP tools

Phase 2 — Backend: processing

Rust, optimised for throughput. Executes nodes and distributes them across workers. See docs/architecture/structure.canvas → Backend – Processing.

• Parallel invocation of stateless nodes over independent input sets, to keep I/O delay minimal (stateful I/O nodes keep serializing via the synchronous mechanism)
• Run user Python nodes out of process: a pool of persistent worker subprocesses speaking one JSON object per line, entered through a proxy the controller installs as the node's function, so every execution path funnels through it unchanged. A crash costs one subprocess, a per-node timeout is a kill, and cancelling from the canvas is that same kill on request
• Extract node execution from the Python prototype into a Rust engine
• Worker distribution and load balancing across capable devices
• Input/output validation at the node boundary
• Data aggregation and discretization

Phase 3 — Frontend: admin view

React + Vite, primarily desktop but usable on mobile. See docs/architecture/structure.canvas → Frontend – Admin View.

• Dashboard SPA shell: TanStack Router, floating frosted sidebar, auth flows, generated OpenAPI SDK
• Node canvas (@xyflow/react) showing nodes and their connections, which are derived from message names rather than stored
• Tab-style view of atomic flows, with a floating dock
• Embedded code editor (Monaco) for node source
• Live values on the edges, with the last payload and its time on click
• Provenance: every value says what caused it, so an edge pulses for the producer that actually published rather than every producer of that message. A dashboard control, another flow or an API caller is drawn as a label on the canvas instead of being invisible — which also gives cross-flow wiring the link in/out it lacked
• Validation shown on the node it belongs to, and summarised in the dock
• Publish control and draft markers in the flow bar, discard in the flow panel, and a conflict dialog when another client got there first
• Marking a node reusable, and placing a shared one from the palette
• Secret picker for credential parameters, so a password never lands in flow.json
• Dashboard showing which flows run, which are stopped and which have errors, with a switch per flow
• Logs panel in the canvas dock, pause/resume beside Run, and replaying an edge's last message from the inspector
• Device assignment per node, selectable from compatible devices
• Test-node affordance on the canvas
• User management screens
• Screens for what the API used to own alone: the secrets store and the alert channels/rules each get a sidebar page, and the OAuth clients an agent registers are listed and revocable under Admin — which needed its management endpoints written first
• Health sections: how the engine is doing now (nodes, flows, queue, loop lag) over what it has been doing all day — throughput and failure charts, a per-flow table, the recent cascades, failures that expand to their traceback, dead-lettered work and the audit trail. Hovering a chart filters the list beside it to that minute and a click pins it
• Brain graph: every published flow at once, with nodes that talk to the same outside thing — a broker topic, a URL, a bucket — drawn as a single neuron, so the wiring that runs between flows through a broker is visible at all. Laid out by a force simulation settled once and then frozen, lit by the same socket the editor listens to, and read-only: a neuron leads back to the flow it came from
• Both of the above sit on Home rather than at routes of their own: the brain flat across the top, the health sections under the flow switches. One overview instead of three
• Mobile-friendly canvas: touch connect, full-screen node panel
• Installable as a PWA (vite-plugin-pwa)

Phase 4 — Frontend: dashboard view

Shares components with the admin view. See docs/architecture/structure.canvas → Frontend – Dashboard View.

• User-defined dashboard layout with edit and view modes: dashboards are their own documents, widgets bind to message names, and the input ones publish back. View mode is plain CSS grid, so a panel that only displays loads no editing code
• Chart widget drawing a message's history through uPlot, with --chart-1…5 as one lightness ramp of the brand hue; a widget bound to the wrong dtype, or to nothing, is flagged the way a failing node is
• Layout by dragging and resizing (react-grid-layout), a grid size per dashboard, and /view/{name} — a full-bleed route that loads neither the editor nor the grid library, which is what a wall panel is pointed at
• Draft and publish, as flows have it: the editor autosaves dashboard.draft.json and a panel reads only the published document, so a half-arranged page never reaches the wall until someone publishes it
• Declared structured payloads — series, record and list (with its item type) join the scalars in DType and are checked the same way, so a widget binds a shape rather than "some JSON" and a wrong binding is refused before anything runs. An agenda over a list and a notification over a record are the first two to read them, the latter fed either by a flow or by the dashboard alert channel
• Charts that query: a chart publishes a request — the window and the resolution — exactly as a slider publishes a value, and draws the series a flow answers with. The answer says what it was computed for and one computed for another window is ignored, so two charts on a node cost a duplicate query rather than the wrong picture. Database nodes stay transport and credentials only: the InfluxDB node runs Flux handed to it and echoes the rest, and Python nodes either side build the query and shape the answer — which is what keeps the widget ignorant of the database
• Per-device view

Phase 5 — Website and docs

• Marketing site with a live node-graph demo, shared design system
• Published documentation site fed from the docs submodule
• Umami analytics configured (the site still ships the placeholder script)