app/ROADMAP.md

Roadmap

Component-level breakdown. The milestone-level master (M1–M5, with the vision decisions behind it) is docs/private/roadmap.md in the docs submodule.

Implementation strategy and record of existing/planned features. Completed items are terse checklists — the requirement detail lives in docs/private/vision.md (goals, requirements, decisions) and docs/architecture/structure.canvas (the four-way component split). Remaining tasks keep enough scope to be actionable.

Legend: [x] done · [ ] planned · sub-lists split done vs. remaining for partial items.

Within each phase, remaining [ ] items are listed in rough priority order: making the existing flow engine reachable and persistent precedes new feature breadth.

Phase 0 — Workspace and platform

• Root orchestrator repo with app, index and docs as submodules
• make init bootstrap: secrets generation, per-stack .env propagation, shared proxy docker network
• Layered compose (compose.yml → compose.dev.yml → compose.local.yml) for both stacks, one Traefik serving ${DOMAIN}, app.${DOMAIN}, api.${DOMAIN}
• Design token contract: root DESIGN-GUIDELINES.md, per-repo DESIGN.md, byte-identical token blocks verified by make design-check
• CI on Codeberg (Forgejo Actions): pre-commit, backend tests, Playwright, compose smoke

Phase 1 — Backend: management

Python, optimised for development speed. Owns the graph structure, persistence and the external interfaces. See docs/architecture/structure.canvas → Backend – Management.

• FastAPI + SQLModel + Alembic + Postgres base with JWT auth and user management
• Flow engine in backend/app/flow/: Node / Pipeline / StateBackend (memory + Redis) / FlowController
• Node types: HTTP, MQTT, InfluxDB, Delay, MLP
• app/flow is an importable package with absolute app.flow.* imports
• Typed, serializable node I/O: every port declares a DType, messages are JSON on the wire and in Redis, no pickle anywhere. Binary codecs are still open — DType.JSON carries everything non-scalar for now
• Message namespacing per flow (flow.message), with several producers per message resolving to real fan-in
• Secrets/credentials store for node integrations managed via the API/UI (encrypted at rest, referenced from node params as {"$secret": "name"}); .env bootstrap-only
• Connector node contract: ConnectorNode with a declared contract version, a polling coordinator that deduplicates, x-secret parameters the editor renders as a secret picker, and health reporting. Connectors are installed packages found through the fluksio.node_types entry point group; the contract is documented in docs/connectors/ with a working skeleton at connector-skeleton/. The registry follows later
• Node lifecycle as a protocol (start/stop/report_health on Node), replacing the controller's per-type isinstance chains — the same hooks a connector implements, validated on the built-in nodes first
• Flow persistence: flow.json plus node sources per flow, replacing the watch-directory prototype
• REST + WebSocket API over the engine: create/read/update flows, edit node source, run, and stream values, node status and execution events
• Dependency-loop detection and graph validation surfaced as API errors
• Per-flow start/stop, stored in a runtime.json beside the flow so it survives a restart and stays out of the autosaved document; pause/resume holds a flow's nodes while its values keep arriving
• Node log streaming: what a node prints, and the traceback of one that fails, reach the editor as node_log events
• MQTT broker / InfluxDB compose services for local development
• Git-based versioning of the flow store (one commit per saved change)
• Draft/publish split: edits autosave to flow.draft.json / nodes.draft/, the engine runs only the published files, and publishing promotes the draft. Saves carry the version they were based on, so a second client editing the same flow is refused rather than overwritten
• Import/export of a flow as human-readable code plus a JSON structure
• Per-input/-output discretization interval setting: a port publishes, or wakes its node, at most every n seconds. State keeps the latest value, so only the delivery is skipped
• Alert / notification handler
• Test nodes: a small node dragged onto an existing one, smoke or unit, blocking deployment on failure
• User management scoped per flow and per data set
• LLM interface for natural-language flow authoring

Phase 2 — Backend: processing

Rust, optimised for throughput. Executes nodes and distributes them across workers. See docs/architecture/structure.canvas → Backend – Processing.

• Parallel invocation of stateless nodes over independent input sets, to keep I/O delay minimal (stateful I/O nodes keep serializing via the synchronous mechanism)
• Extract node execution from the Python prototype into a Rust engine
• Worker distribution and load balancing across capable devices
• Input/output validation at the node boundary
• Data aggregation and discretization

Phase 3 — Frontend: admin view

React + Vite, primarily desktop but usable on mobile. See docs/architecture/structure.canvas → Frontend – Admin View.

• Dashboard SPA shell: TanStack Router, floating frosted sidebar, auth flows, generated OpenAPI SDK
• Node canvas (@xyflow/react) showing nodes and their connections, which are derived from message names rather than stored
• Tab-style view of atomic flows, with a floating dock
• Embedded code editor (Monaco) for node source
• Live values on the edges, with the last payload and its time on click
• Validation shown on the node it belongs to, and summarised in the dock
• Publish control and draft markers in the flow bar, discard in the flow panel, and a conflict dialog when another client got there first
• Marking a node reusable, and placing a shared one from the palette
• Secret picker for credential parameters, so a password never lands in flow.json
• Dashboard showing which flows run, which are stopped and which have errors, with a switch per flow
• Logs panel in the canvas dock, pause/resume beside Run, and replaying an edge's last message from the inspector
• Device assignment per node, selectable from compatible devices
• Test-node affordance on the canvas
• User management screens
• Mobile-friendly canvas: touch connect, full-screen node panel
• Installable as a PWA (vite-plugin-pwa)

Phase 4 — Frontend: dashboard view

Shares components with the admin view. See docs/architecture/structure.canvas → Frontend – Dashboard View.

• User-defined dashboard layout with edit and view modes
• Responsive layout targeting wall panels, mobile and desktop
• Per-device view

Phase 5 — Website and docs

• Marketing site with a live node-graph demo, shared design system
• Published documentation site fed from the docs submodule
• Umami analytics configured (the site still ships the placeholder script)