stroblmeandClaude Fable 5
75c26ef000
Moving a dashboard slider lit up an edge between two nodes that had done nothing. The canvas pulsed on the message's timestamp alone, and a message has no idea who published it — so it credited whichever node happened to be drawn as a producer. That was never only about dashboards. Two nodes producing one message pulsed both their edges whichever fired, and a message produced in another flow changed with nothing on screen to account for it at all. Values now carry their cause: a node, a dashboard widget, another flow, an agent or an API caller. An edge pulses only for the producer that actually published, and the edge inspector says where a value came from when it did not come from a node. What is not a node in this flow is now drawn as one — a label rather than a card, because a dashboard with twenty tiles would otherwise bury the logic the canvas exists to show. That covers cross-flow wiring too, which is the link in/out affordance that has been missing. They are never part of the document. They join at render, after everything that reads or writes the canvas nodes, so an autosave, an undo or a delete cannot reach them — with a Playwright test that drags a node and asserts the stored flow still holds exactly what it did. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011LF61rxW1FG5YCD2J9YqjY
174 lines
10 KiB
Markdown
174 lines
10 KiB
Markdown
# Roadmap
|
||
|
||
Component-level breakdown. The milestone-level master (M1–M5, with the vision
|
||
decisions behind it) is `docs/private/roadmap.md` in the docs submodule.
|
||
|
||
Implementation strategy and record of existing/planned features. Completed items are
|
||
terse checklists — the requirement detail lives in `docs/private/vision.md` (goals,
|
||
requirements, decisions) and `docs/architecture/structure.canvas` (the four-way component split).
|
||
Remaining tasks keep enough scope to be actionable.
|
||
|
||
Legend: `[x]` done · `[ ]` planned · sub-lists split done vs. remaining for partial items.
|
||
|
||
Within each phase, remaining `[ ]` items are listed in rough priority order: making the
|
||
existing flow engine reachable and persistent precedes new feature breadth.
|
||
|
||
## Phase 0 — Workspace and platform
|
||
|
||
- [x] Root orchestrator repo with `app`, `index` and `docs` as submodules
|
||
- [x] `make init` bootstrap: secrets generation, per-stack `.env` propagation, shared
|
||
`proxy` docker network
|
||
- [x] Layered compose (`compose.yml` → `compose.dev.yml` → `compose.local.yml`) for both
|
||
stacks, one Traefik serving `${DOMAIN}`, `app.${DOMAIN}`, `api.${DOMAIN}`
|
||
- [x] Design token contract: root `DESIGN-GUIDELINES.md`, per-repo `DESIGN.md`,
|
||
byte-identical token blocks verified by `make design-check`
|
||
- [ ] CI on Codeberg (Forgejo Actions): pre-commit, backend tests, Playwright, compose smoke
|
||
|
||
## Phase 1 — Backend: management
|
||
|
||
Python, optimised for development speed. Owns the graph structure, persistence and the
|
||
external interfaces. See `docs/architecture/structure.canvas` → *Backend – Management*.
|
||
|
||
- [x] FastAPI + SQLModel + Alembic + Postgres base with JWT auth and user management
|
||
- [x] Flow engine in `backend/app/flow/`: `Node` / `Pipeline` / `StateBackend`
|
||
(memory + Redis) / `FlowController`
|
||
- [x] Node types: HTTP, MQTT, InfluxDB, Delay, MLP
|
||
- [x] Flow-logic vocabulary as node types rather than repeated code: inject (manual,
|
||
interval, cron or at startup), switch, change, filter-unchanged, join, trigger,
|
||
command, file and ntfy. Each is configured by filling in a form the editor
|
||
generates from its parameter schema
|
||
- [x] `app/flow` is an importable package with absolute `app.flow.*` imports
|
||
- [x] Typed, serializable node I/O: every port declares a `DType`, messages are
|
||
JSON on the wire and in Redis, no pickle anywhere. Binary codecs are still
|
||
open — `DType.JSON` carries everything non-scalar for now
|
||
- [x] Message namespacing per flow (`flow.message`), with several producers per
|
||
message resolving to real fan-in
|
||
- [x] Secrets/credentials store for node integrations managed via the API/UI
|
||
(encrypted at rest, referenced from node params as `{"$secret": "name"}`);
|
||
`.env` bootstrap-only
|
||
- [x] Connector node contract: `ConnectorNode` with a declared contract version,
|
||
a polling coordinator that deduplicates, `x-secret` parameters the editor
|
||
renders as a secret picker, and health reporting. Connectors are installed
|
||
packages found through the `fluksio.node_types` entry point group; the
|
||
contract is documented in `docs/connectors/` with a working skeleton at
|
||
`connector-skeleton/`. The registry follows later
|
||
- [x] First real connectors written against that contract from outside the engine:
|
||
WF-RAC aircon, calendar, UniFi presence and Art-Net, in `connectors/`.
|
||
Built by `make connectors` and installed into the image. Reading only for
|
||
now — the aircon package cannot produce a command and Art-Net keeps its
|
||
packets off the wire until `transmit` is switched on
|
||
- [x] Node lifecycle as a protocol (`start`/`stop`/`report_health` on `Node`),
|
||
replacing the controller's per-type isinstance chains — the same hooks a
|
||
connector implements, validated on the built-in nodes first
|
||
- [x] Flow persistence: `flow.json` plus node sources per flow, replacing the
|
||
watch-directory prototype
|
||
- [x] REST + WebSocket API over the engine: create/read/update flows, edit node
|
||
source, run, and stream values, node status and execution events
|
||
- [x] Dependency-loop detection and graph validation surfaced as API errors
|
||
- [x] Per-flow start/stop, stored in a `runtime.json` beside the flow so it
|
||
survives a restart and stays out of the autosaved document; pause/resume
|
||
holds a flow's nodes while its values keep arriving
|
||
- [x] Node log streaming: what a node prints, and the traceback of one that
|
||
fails, reach the editor as `node_log` events
|
||
- [ ] MQTT broker / InfluxDB compose services for local development
|
||
- [x] Git-based versioning of the flow store (one commit per saved change)
|
||
- [x] Draft/publish split: edits autosave to `flow.draft.json` / `nodes.draft/`,
|
||
the engine runs only the published files, and publishing promotes the
|
||
draft. Saves carry the version they were based on, so a second client
|
||
editing the same flow is refused rather than overwritten
|
||
- [ ] Import/export of a flow as human-readable code plus a JSON structure
|
||
- [x] Per-input/-output discretization interval setting: a port publishes, or
|
||
wakes its node, at most every n seconds. State keeps the latest value, so
|
||
only the delivery is skipped
|
||
- [x] Alert / notification handler: engine failures — a node raising, a connection
|
||
dropping, a flow quarantined, the queue gone — reach ntfy, email or a webhook.
|
||
Mostly it declines to send: the same fault repeating is one alert with a count,
|
||
a flapping connection is muted, and there is a ceiling per hour. Configured
|
||
through the API at `/alerts/config`, with a test send per channel
|
||
- [x] Deep health check (`GET /utils/health/`): reports event-loop lag and state-backend
|
||
reachability and fails the container healthcheck, so a wedged engine is restarted
|
||
rather than counted as up. One engine per deployment — the API image runs a single
|
||
worker, because a second one would be a second engine
|
||
- [x] Supervised background tasks: a node's subscription, schedule or poll loop is
|
||
restarted with growing delay when it dies, and a flow that spends its failure
|
||
budget is quarantined and surfaced rather than left crash-looping. The loops
|
||
themselves no longer carry private retry logic
|
||
- [x] Durable work queue: every external trigger is journaled to Redis Streams before
|
||
anything runs and acknowledged once its cascade finishes, so an engine that dies
|
||
mid-cascade picks the work up again instead of losing it. A reaper reclaims what
|
||
a dead consumer never acknowledged; nodes that reach outside are skipped on a
|
||
redelivery they already ran. Long-lived worker pools replace the per-wave
|
||
executors, and a delay now waits in the queue rather than on a worker thread
|
||
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
|
||
deployment on failure
|
||
- [ ] User management scoped per flow and per data set
|
||
- [x] MCP server over the same API: agents authenticate through a built-in
|
||
OAuth 2.1 authorization server (dynamic registration, PKCE, rotating
|
||
refresh tokens) and drive the flow API through 20 tools. Tokens are
|
||
RS256, signed with their own keypair, so the set can be revoked on its
|
||
own — and an additional issuer is one branch in `deps.decode_token`,
|
||
which is the seam remote access needs later
|
||
- [ ] LLM interface for natural-language flow authoring beyond the MCP tools
|
||
|
||
## Phase 2 — Backend: processing
|
||
|
||
Rust, optimised for throughput. Executes nodes and distributes them across workers. See
|
||
`docs/architecture/structure.canvas` → *Backend – Processing*.
|
||
|
||
- [ ] Parallel invocation of stateless nodes over independent input sets, to
|
||
keep I/O delay minimal (stateful I/O nodes keep serializing via the
|
||
`synchronous` mechanism)
|
||
- [ ] Extract node execution from the Python prototype into a Rust engine
|
||
- [ ] Worker distribution and load balancing across capable devices
|
||
- [ ] Input/output validation at the node boundary
|
||
- [ ] Data aggregation and discretization
|
||
|
||
## Phase 3 — Frontend: admin view
|
||
|
||
React + Vite, primarily desktop but usable on mobile. See `docs/architecture/structure.canvas` →
|
||
*Frontend – Admin View*.
|
||
|
||
- [x] Dashboard SPA shell: TanStack Router, floating frosted sidebar, auth flows,
|
||
generated OpenAPI SDK
|
||
- [x] Node canvas (`@xyflow/react`) showing nodes and their connections, which
|
||
are derived from message names rather than stored
|
||
- [x] Tab-style view of atomic flows, with a floating dock
|
||
- [x] Embedded code editor (Monaco) for node source
|
||
- [x] Live values on the edges, with the last payload and its time on click
|
||
- [x] Provenance: every value says what caused it, so an edge pulses for the producer
|
||
that actually published rather than every producer of that message. A dashboard
|
||
control, another flow or an API caller is drawn as a label on the canvas instead
|
||
of being invisible — which also gives cross-flow wiring the link in/out it lacked
|
||
- [x] Validation shown on the node it belongs to, and summarised in the dock
|
||
- [x] Publish control and draft markers in the flow bar, discard in the flow
|
||
panel, and a conflict dialog when another client got there first
|
||
- [x] Marking a node reusable, and placing a shared one from the palette
|
||
- [x] Secret picker for credential parameters, so a password never lands in
|
||
`flow.json`
|
||
- [x] Dashboard showing which flows run, which are stopped and which have
|
||
errors, with a switch per flow
|
||
- [x] Logs panel in the canvas dock, pause/resume beside Run, and replaying an
|
||
edge's last message from the inspector
|
||
- [ ] Device assignment per node, selectable from compatible devices
|
||
- [ ] Test-node affordance on the canvas
|
||
- [ ] User management screens
|
||
- [x] Mobile-friendly canvas: touch connect, full-screen node panel
|
||
- [ ] Installable as a PWA (`vite-plugin-pwa`)
|
||
|
||
## Phase 4 — Frontend: dashboard view
|
||
|
||
Shares components with the admin view. See `docs/architecture/structure.canvas` →
|
||
*Frontend – Dashboard View*.
|
||
|
||
- [x] User-defined dashboard layout with edit and view modes: dashboards are their own
|
||
documents, widgets bind to message names, and the input ones publish back. View
|
||
mode is plain CSS grid, so a panel that only displays loads no editing code
|
||
- [ ] Responsive layout targeting wall panels, mobile and desktop
|
||
- [ ] Per-device view
|
||
|
||
## Phase 5 — Website and docs
|
||
|
||
- [x] Marketing site with a live node-graph demo, shared design system
|
||
- [ ] Published documentation site fed from the `docs` submodule
|
||
- [ ] Umami analytics configured (the site still ships the placeholder script)
|