Melvin StroblandClaude Fable 5
3724b68f23
Connectors are the device-facing node class third parties write, so the surface they build against is versioned and documented: ConnectorNode carries a declared contract version, a polling loop that publishes only what changed and reports health around it, and parameters whose credential fields are marked x-secret so the editor offers the secrets store instead of a text box. They are found through the fluksio.node_types entry point group, with the package's own metadata as the manifest. docs/connectors/ has the contract and the authoring guide; connector-skeleton/ is a working one to copy. The controller no longer knows what any node type is: start, stop and report_health are protocol methods on Node, and the built-ins were migrated to them first, so the hooks a connector implements are the ones the engine has been driving all along. Marking a node reusable moves its source to _lib/ and points the node at it by name. Other flows instantiate it with their own ports and settings, one fix reaches all of them, and a shared source still in use cannot be deleted. Ports gained an interval: an output publishes, and an input wakes its node, at most every n seconds. State keeps the latest value, so only the delivery is skipped, and pressing Run is never throttled. Also fixes autosave sending no version on its first save of a session, which made every flow saved more than once conflict with itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
135 lines
7.3 KiB
Markdown
135 lines
7.3 KiB
Markdown
# Roadmap
|
||
|
||
Component-level breakdown. The milestone-level master (M1–M5, with the vision
|
||
decisions behind it) is `docs/private/roadmap.md` in the docs submodule.
|
||
|
||
Implementation strategy and record of existing/planned features. Completed items are
|
||
terse checklists — the requirement detail lives in `docs/private/vision.md` (goals,
|
||
requirements, decisions) and `docs/architecture/structure.canvas` (the four-way component split).
|
||
Remaining tasks keep enough scope to be actionable.
|
||
|
||
Legend: `[x]` done · `[ ]` planned · sub-lists split done vs. remaining for partial items.
|
||
|
||
Within each phase, remaining `[ ]` items are listed in rough priority order: making the
|
||
existing flow engine reachable and persistent precedes new feature breadth.
|
||
|
||
## Phase 0 — Workspace and platform
|
||
|
||
- [x] Root orchestrator repo with `app`, `index` and `docs` as submodules
|
||
- [x] `make init` bootstrap: secrets generation, per-stack `.env` propagation, shared
|
||
`proxy` docker network
|
||
- [x] Layered compose (`compose.yml` → `compose.dev.yml` → `compose.local.yml`) for both
|
||
stacks, one Traefik serving `${DOMAIN}`, `app.${DOMAIN}`, `api.${DOMAIN}`
|
||
- [x] Design token contract: root `DESIGN-GUIDELINES.md`, per-repo `DESIGN.md`,
|
||
byte-identical token blocks verified by `make design-check`
|
||
- [ ] CI on Codeberg (Forgejo Actions): pre-commit, backend tests, Playwright, compose smoke
|
||
|
||
## Phase 1 — Backend: management
|
||
|
||
Python, optimised for development speed. Owns the graph structure, persistence and the
|
||
external interfaces. See `docs/architecture/structure.canvas` → *Backend – Management*.
|
||
|
||
- [x] FastAPI + SQLModel + Alembic + Postgres base with JWT auth and user management
|
||
- [x] Flow engine in `backend/app/flow/`: `Node` / `Pipeline` / `StateBackend`
|
||
(memory + Redis) / `FlowController`
|
||
- [x] Node types: HTTP, MQTT, InfluxDB, Delay, MLP
|
||
- [x] `app/flow` is an importable package with absolute `app.flow.*` imports
|
||
- [x] Typed, serializable node I/O: every port declares a `DType`, messages are
|
||
JSON on the wire and in Redis, no pickle anywhere. Binary codecs are still
|
||
open — `DType.JSON` carries everything non-scalar for now
|
||
- [x] Message namespacing per flow (`flow.message`), with several producers per
|
||
message resolving to real fan-in
|
||
- [x] Secrets/credentials store for node integrations managed via the API/UI
|
||
(encrypted at rest, referenced from node params as `{"$secret": "name"}`);
|
||
`.env` bootstrap-only
|
||
- [x] Connector node contract: `ConnectorNode` with a declared contract version,
|
||
a polling coordinator that deduplicates, `x-secret` parameters the editor
|
||
renders as a secret picker, and health reporting. Connectors are installed
|
||
packages found through the `fluksio.node_types` entry point group; the
|
||
contract is documented in `docs/connectors/` with a working skeleton at
|
||
`connector-skeleton/`. The registry follows later
|
||
- [x] Node lifecycle as a protocol (`start`/`stop`/`report_health` on `Node`),
|
||
replacing the controller's per-type isinstance chains — the same hooks a
|
||
connector implements, validated on the built-in nodes first
|
||
- [x] Flow persistence: `flow.json` plus node sources per flow, replacing the
|
||
watch-directory prototype
|
||
- [x] REST + WebSocket API over the engine: create/read/update flows, edit node
|
||
source, run, and stream values, node status and execution events
|
||
- [x] Dependency-loop detection and graph validation surfaced as API errors
|
||
- [x] Per-flow start/stop, stored in a `runtime.json` beside the flow so it
|
||
survives a restart and stays out of the autosaved document; pause/resume
|
||
holds a flow's nodes while its values keep arriving
|
||
- [x] Node log streaming: what a node prints, and the traceback of one that
|
||
fails, reach the editor as `node_log` events
|
||
- [ ] MQTT broker / InfluxDB compose services for local development
|
||
- [x] Git-based versioning of the flow store (one commit per saved change)
|
||
- [x] Draft/publish split: edits autosave to `flow.draft.json` / `nodes.draft/`,
|
||
the engine runs only the published files, and publishing promotes the
|
||
draft. Saves carry the version they were based on, so a second client
|
||
editing the same flow is refused rather than overwritten
|
||
- [ ] Import/export of a flow as human-readable code plus a JSON structure
|
||
- [x] Per-input/-output discretization interval setting: a port publishes, or
|
||
wakes its node, at most every n seconds. State keeps the latest value, so
|
||
only the delivery is skipped
|
||
- [ ] Alert / notification handler
|
||
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
|
||
deployment on failure
|
||
- [ ] User management scoped per flow and per data set
|
||
- [ ] LLM interface for natural-language flow authoring
|
||
|
||
## Phase 2 — Backend: processing
|
||
|
||
Rust, optimised for throughput. Executes nodes and distributes them across workers. See
|
||
`docs/architecture/structure.canvas` → *Backend – Processing*.
|
||
|
||
- [ ] Parallel invocation of stateless nodes over independent input sets, to
|
||
keep I/O delay minimal (stateful I/O nodes keep serializing via the
|
||
`synchronous` mechanism)
|
||
- [ ] Extract node execution from the Python prototype into a Rust engine
|
||
- [ ] Worker distribution and load balancing across capable devices
|
||
- [ ] Input/output validation at the node boundary
|
||
- [ ] Data aggregation and discretization
|
||
|
||
## Phase 3 — Frontend: admin view
|
||
|
||
React + Vite, primarily desktop but usable on mobile. See `docs/architecture/structure.canvas` →
|
||
*Frontend – Admin View*.
|
||
|
||
- [x] Dashboard SPA shell: TanStack Router, floating frosted sidebar, auth flows,
|
||
generated OpenAPI SDK
|
||
- [x] Node canvas (`@xyflow/react`) showing nodes and their connections, which
|
||
are derived from message names rather than stored
|
||
- [x] Tab-style view of atomic flows, with a floating dock
|
||
- [x] Embedded code editor (Monaco) for node source
|
||
- [x] Live values on the edges, with the last payload and its time on click
|
||
- [x] Validation shown on the node it belongs to, and summarised in the dock
|
||
- [x] Publish control and draft markers in the flow bar, discard in the flow
|
||
panel, and a conflict dialog when another client got there first
|
||
- [x] Marking a node reusable, and placing a shared one from the palette
|
||
- [x] Secret picker for credential parameters, so a password never lands in
|
||
`flow.json`
|
||
- [x] Dashboard showing which flows run, which are stopped and which have
|
||
errors, with a switch per flow
|
||
- [x] Logs panel in the canvas dock, pause/resume beside Run, and replaying an
|
||
edge's last message from the inspector
|
||
- [ ] Device assignment per node, selectable from compatible devices
|
||
- [ ] Test-node affordance on the canvas
|
||
- [ ] User management screens
|
||
- [x] Mobile-friendly canvas: touch connect, full-screen node panel
|
||
- [ ] Installable as a PWA (`vite-plugin-pwa`)
|
||
|
||
## Phase 4 — Frontend: dashboard view
|
||
|
||
Shares components with the admin view. See `docs/architecture/structure.canvas` →
|
||
*Frontend – Dashboard View*.
|
||
|
||
- [ ] User-defined dashboard layout with edit and view modes
|
||
- [ ] Responsive layout targeting wall panels, mobile and desktop
|
||
- [ ] Per-device view
|
||
|
||
## Phase 5 — Website and docs
|
||
|
||
- [x] Marketing site with a live node-graph demo, shared design system
|
||
- [ ] Published documentation site fed from the `docs` submodule
|
||
- [ ] Umami analytics configured (the site still ships the placeholder script)
|