Fluksio/app
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move NOTEPAD.md and ROADMAP.md to the workspace root

Browse Source 
Consolidates deferred-work tracking now that app is going public, so no
strategy/planning content ships with it.
This commit is contained in:
stroblme
2026-08-24 19:16:39 +02:00
parent 0926dabb99
commit 00a91c19b8
2 changed files with 0 additions and 800 deletions
Download Patch File Download Diff File Expand all files Collapse all files
NOTEPAD.md
-406
View File
@@ -1,406 +0,0 @@
This file captures tasks which derive from roadmap tasks (unfinished, deferred), bugs encountered during usage and feature requests/improvements which are not fitting directly in the roadmap.
Always sort by priority and put tasks blocked by other tasks/features at the dedicated section.
When working on a task, check for other, similar tasks that could be resolved on the way.
Use following pattern to classify tasks: TYPE/SCOPE
Where TYPE could be BUG, FEAT, PERF, CHORE and SCOPE could be UX, UI, FLOW, NODE, API, INFRA, DOCS appended by MOBILE if only for mobile use case.
Don't write temporary reasons for deferring a task in the task description (only strategical reasons should be noted).
Deferring because out of scope is fine, but don't mention deferring than.
`Deferred` holds what stays open on purpose, each with the condition that should reopen it.
## Open
### To be sorted
- FEAT/UI the CLI of fluksio should be improved by using colors and styling as well as ctrl-c handling. Consider a small TUI which serves like a mini-dashboard showing current flow status and error messages
- FEAT/UI when a dashboard has more widgets than the canvas allows for, the canvas should become scrollable (temporarily) to allow scrolling to the widgets outside the canvas and reordering them
- BUG/UI some nodes (especially smaller ones) in the brain graph still don't have a pixel perfect centered ring and node background
- FEAT/UI: make the "Settings" button in the dashboard panel a toggle; such that it becomes stateful and pressing again closes the settings panel
- FEAT/UI: we promise testing, but currently don't provide an UI for testing e.g. mock values or probing edge cases of a flow. I'm thinking of a "Labs" page, which allows simulating an installation with all the flows (using their draft states) and which allows injecting values or mocking values based on events in the past. This should then use the draft version of flows / dashboards. For the flows we could make a tab-style view and re-embedd the flow canvas. For the dashboards we could reuse the icon rail
- FEAT/UI: check if PWA (https://whatpwacando.today/) notifications could be used to have a panel sending notifications to the device event bus (or generally using PWA to retrieve e.g. location etc). We could introduce a general concept of having a panel (a device, like a wall panel or a phone where the pwa (dashboard) runs) being effectively a node with various outputs. Then various inputs could trigger actions like authentification (i.e. you get home and get a notification which allows you to authenticate the door unlock), get notified on alarms (native alarm connector) or to query geolocation (check where the user is before turning of all lights) etc. Another example would be a wall panel with speech in- and output; we could use pwa as an interface here for any tts/stt instance managed through fluksio
- BUG/UI sync the theme state between panels and installations
- CHORE/UI: a dashboard forced to one theme inside a shell on the other still
matches `dark:` utilities, because the variant is `&:is(.dark *)` and the
shell is an ancestor. The dashboard's own components no longer use those
utilities — both looks read the tokens and their own `[data-look]` rules — so
what is left is the shadcn pieces still drawn inside a tile (tooltip,
skeleton). A real fix is `.theme-light` / `.theme-dark` namespacing, which is
a change to both repos' byte-identical token blocks.
- CHORE/UI: a stacked dashboard on a phone gets no ground — no blobs, no
background image, just the palette's own colour. The canvas is what carries
it, and a stacked panel is not drawn on one.
- CHORE/UI: the Glass look's ground is three compositor-animated gradients, each
wandering a closed path on its own clock. If a low-end wall panel stutters,
drop one or set `LOOK.glass.drift` to 0.
- CHORE/UI: the app still has three copies of the segmented pill
(`ui/segmented.tsx`, `Common/RangePicker.tsx`, and the flow screens). The
dashboard no longer shares them — it has its own, one per look — so
unifying the remaining three is now purely an app-side job.
- CHORE/UI: the house panels are laid out for 1280x800 — twelve columns, twelve
rows. A chart's fixed chrome is now its legend alone: the range picker moved
to a column down the right-hand edge and gave its row back, so a third chart
may now fit — worth measuring against the panel before adding one. The
temperature history was the one dropped; `history.climate_*` still answers,
so it is a tile away.
- FEAT/UI: a bar row is captioned by its port name, which is chosen for the
graph rather than for somebody reading it across a room. A row takes an
optional `label`; the house dashboard sets one, and it shows on the next
frontend build.
- BUG/NODE: **an MQTT client can hang forever on the way out.** `MqttNode` builds
`aiomqtt.Client` without a `timeout`, so `Client.__aexit__` waits for the broker's
disconnect acknowledgement with no deadline — `_wait_for(..., timeout=None)` falls
through to `self.timeout`, which is `None` too. A subscriber cancelled while its
socket is dead never finishes unwinding. This was what wedged the rebuild lock;
the teardown now abandons such a task after five seconds rather than waiting on
it, so what is left is the task itself, which is stopped only by the second
cancellation it is sent on the way out. Passing a `timeout` to the client fixes it
at the source, but the same number also bounds `subscribe` and `publish`, so it
wants choosing deliberately.
- BUG/FLOW: `MqttNode.stop_publisher`, `MqttNode.stop_subscription` and
`DelayNode.stop_cron` catch `CancelledError` around the task they have just
cancelled, which swallows a cancellation meant for the caller — the trap
`Supervisor.cancel_all` was just fixed for. Latent rather than live: a supervised
node leaves those handles `None`, so only a node built on its own (a test, a
preview) awaits there.
- BUG/INFRA: **zombie `python` processes in the API container**, twelve of them in
groups of four with matching elapsed times — one group per worker pool. `_Worker.kill()`
waits on the process it kills, so these are workers orphaned when the process holding
their `Popen` went away without stopping the pool, which is what uvicorn `--reload`
does on every source edit. They reparent onto PID 1, and PID 1 in the container is the
reloader, which reaps nothing that is not its own child. `init: true` on the backend
service hands PID 1 to an init that does reap orphans, and would close the whole class
rather than one source of it.
- CHORE/INFRA: the engine and the worker speak a version-matched protocol, but only the
engine hot-reloads — uvicorn watches `/app/backend`, and a worker process holds the code
it started with until it is respawned. So a protocol change reaches one side and not the
other, which is how every python node came to fail with `RemoteError: the worker answered
request None`. `compose.dev.yml` now mounts `worker/fluksio_worker` beside
`backend/fluksio`, which closes the image half. The reload half is open: editing worker
source restarts nothing, and the running pool keeps the old code.
- CHORE/UI: the portal's aggregate tiles have no not-reporting state. A figure an
installation never sends renders through `?? 0`, so a missing field reads as a confident
zero rather than as unknown — which is how a deleted `failures_24h` showed as "0 failures"
unnoticed. The per-card half is one line; Home sums across installations and has nowhere to
say "3 of 4 reporting", so the two want doing together.
- FEAT/SEC: `locked` is a read-only surface, not a permission — the server accepts a publish
from a panel whose dashboard says locked. Making it real means carrying the flag into
`_panel_may`.
- FEAT/UI: a dashboard forced light inside a dark app shell gets correct tokens but still
matches `dark:` variant utilities, because those compile to `&:is(.dark *)` and `<html>`
always carries a theme class. Cosmetic (a few faint shadcn backgrounds); fixing it properly
needs a theme-scope mechanism CSS ancestor selectors cannot express today.
Shares a root with a bug fixed since: the token scopes are keyed on the bare class names
`light`/`dark`, so React Flow's own `colorMode` class silently re-themed an entire canvas
to light until the prop was passed explicitly. Namespacing them (`.theme-light` /
`.theme-dark`) would sit under both. Two repos — the token blocks are a byte-identical
duplication contract with `index/src/index.css` — plus `useDashboardTheme`.
- CHORE/UI: the lock notice is on `PanelSurface` only, so `/dashboards/{name}` in read mode
shows disabled controls without the pill.
- CHORE/DOC: the docs call a cron `inject` node "a node publishing on a schedule". The node
type literally named `trigger` is a debounce/hold node, not a scheduler — worth renaming one
of the two eventually.
- CHORE/DEMO: the demo panel now fills all 17 rows its 2560×1600 canvas holds, every row
across all 16 columns. Another tile needs a rearrangement first — worth doing together
with re-authoring `PAGES` as a single section.
- CHORE/TEST: no Playwright coverage for the colour tile — `tests/widgets.spec.ts` covers the
other controls. Wants a spec that drags the disc and asserts one publish on release. The
maths is checked (`color.check.ts`); what is not is the gesture.
- CHORE/DEMO: `seed_demo.py` still writes the Home dashboard as three titled sections
("Right now", "Energy and comfort", "Yield model"). The editor and the panel now read a
page's sections as one grid, so those headings are no longer drawn and the first editor
save collapses them into one section. Re-author `PAGES` as a single section, and decide
whether the three headings come back as markdown widgets or go for good.
- CHORE/UI: the widget-side segmented control in `widgets.tsx` (`DropdownWidget`,
`style: "segmented"`) and `Common/RangePicker.tsx` still carry their own copy of the shape
now in `ui/segmented.tsx`. Pointing them there needs a size prop first — the widget one is
a full-width pill with a 44px touch target, the editor one is `w-fit`/`text-xs`.
- CHORE/UI: `IconPicker` in `panels.tsx` has no filter field; the grid shows all of `ICONS`
at once. Add one when the map outgrows a popover.
- CHORE/UI: the delete-panel button in `PanelsDialog` has no confirmation, while unpairing —
the strictly less destructive action — now does.
- CHORE/UI: `DashboardMosaic.blocksOf` offsets sections unconditionally, while
`DashboardView.flatWidgets` skips the offset for an unarranged document. Cosmetic, in the
schematic preview only; the two want to be one function once the mosaic may import the
dashboard chunk.
- FEAT/API: `DashboardSummary` carries no `icon`, so the panels dialog cannot show which
glyph each assigned dashboard draws on the rail — the icon is only visible from that
dashboard's own settings panel. Setting it from the panels dialog would need a decision
about publishing an icon change on its own, since that dialog has no draft concept.
- CHORE/TEST: nothing guards "a dashboard with two pages or sections survives an editor
save". Verified by hand; the only runner is Playwright, so it wants a spec that PUTs a
two-page document, opens the editor, drags one widget and asserts the second page came
back untouched.
- CHORE/UI: the transmit pulse and the editor's selection ring are both a 2px inset
`--primary` ring, so a control selected in edit mode and one mid-publish look the same.
Only visible while editing.
- CHORE/UI: slider tick labels are laid out by percentage without measuring, so five
labels of four characters can crowd on a tile narrower than its default four columns.
A width-aware count needs a `ResizeObserver`.
- FEAT/API: neither `FlowSummary` nor `DashboardSummary` carries a modified time, so
Home's "recently modified" order is a proxy — drafts first, then version counter, then
name (`byRecency`, `Common/DashboardMosaic.tsx`). The store is git-backed, so an
`updated_at` on both summaries would make it exact.
- PERF/UI: the home mosaic reads each dashboard's document for its footprint, capped at
eight. An installation with dozens shows name-only tiles past that; a placement digest
on `DashboardSummary` is the fix.
- PERF/API: `_panel_may` re-reads `panels.json` and every published dashboard document of
the panel on each request a screen makes, to resolve the message allowlist. Marked
`# ponytail:` in `api/deps.py`; cache behind the dashboard store's version if it shows
up in a profile.
- CHORE/TEST: the frontend has no unit-test runner (Playwright only), so pure helpers like
the slider's `tickIntervals` have nowhere to be checked cheaply. `color.check.ts` is a
worked pattern — `bun run` over plain `node:assert`, no framework, imported by nothing so
it never bundles. What is left is a make target that runs every `*.check.ts`.
- FEAT/UI when an installation is added to the hub, the dialog which shows the access code should disappear automatically
- BUG/UI the "Connect" button in "Remote Access" when adding an installtion to the hub is invisible and only shows upon hovering (could also be a local browser issue)
- CHORE/PKG the SPA is not in the wheel: `fluksio serve` serves no UI, on the
assumption that a pip install is paired with a portal. Bundling `dist/` and
mounting it with `StaticFiles` would give a local dashboard — it needs a hatch
build hook running bun, `VITE_API_URL=""` for the same-origin case, and a
decision about the MCP `mount("/")` it would collide with.
- CHORE/PKG the Docker image still starts with `fastapi run`; `fluksio serve`
now does the same thing plus the bootstrap. Switching would give the container
and a pip install one code path.
- CHORE/DEPS `sentry-sdk` went to 2.x and the `requires-python` cap came off with
it. Nothing exercises Python 3.13/3.14 in CI — the matrix is one version.
- BUG/UI assimilate the design of the settings in the app to mirror the design of the settings in the portal
- CHORE/UI: pending states here are lucide's `Loader2` on `animate-spin`; the portal draws the brand mark instead (`index/frontend/src/components/ui/fluksio-loader.tsx` plus the `.fluksio-loader-line` rule at the foot of its `index.css`). Copying both across — the shadcn model, so the CSS travels with the component — would make one loading mark across the two shells.
- CHORE/INFRA: the `generate-frontend-sdk` pre-commit hook runs `scripts/generate-client.sh`
on any `backend/**` change, and that script ends by formatting the whole frontend tree —
while `biome.json` excludes `src/client`, so it formats nothing the generator wrote. Every
backend commit therefore rewrites files it never touched. Dropping the trailing format call
is the fix; it was kept this wave only to preserve behaviour.
- CHORE/UI: acknowledging a node's failure clears it on the engine but publishes no event, so
another browser watching the same flow keeps the marker until its next snapshot or rebuild.
One event on the bus would close it, the way `dashboard_changed` does for panels.
- CHORE/API: `save_dashboard` still catches `DashboardNotFound` from `write_draft`, which can
no longer raise it. Harmless, and the same shape `saveFlow` has: a PUT to an unknown name now
creates that dashboard's first draft rather than answering 404.
- FEAT/UX add an option to the settings of an installation to configure automatic updates. If enabled, the installation would send a request e.g. every 1h to the hub at fluksio.com and the hub then checks if a new version is available. The settings should include a second toggle for automatically installing an update (which might cause a short outage). Later this mechanism should be extended to check if updating would cause things to break.
- CHORE/UI: two of the mobile-overflow floors are over-determined. Removing
`.widget-grid { min-width: 0 }`, or the segmented fieldset's `min-w-0`, leaves the mobile
suite green — the grid tracks are already `minmax(0, 1fr)` and the fieldset became a grid.
The uPlot legend is the one offender the assertion actually catches. Both are cheap
insurance for a future widget, but nothing would notice if they regressed.
- CHORE/API: artifact blobs are content-addressed and have no GC, so deleting a flow drops
its `run_artifact` rows and leaves the bytes on the data volume.
- CHORE/API: no run table carries a foreign key to `run.id` — the baseline migration declares
none for `run_node`, `run_metric` or `run_artifact` — so nothing at the database level stops
a late write from orphaning rows. `DELETE /flows/{name}` now refuses while a run is
`running` or `queued`, which closes the path that was known; a constraint would need a
migration and would turn `_record_node`'s late insert into a hard error rather than the
no-op it is today.
- CHORE/INFRA: `backend/htmlcov` can be left root-owned by a container run, and
`make test-backend` then fails at the coverage HTML step *after* every test has passed —
which reads like a test failure and is not one. `make clean` is the fix.
- CHORE/UI: the edge popover shows the same value twice — `MessageSparkline` falls through to a collapsed `ValuePreview` for a non-numeric value, and `EdgeInspector` then renders its own `ValuePreview defaultOpen` below it. Cosmetic; one of the two is redundant.
- FEAT/UI: a settings-and-inputs overview page, so what every node of an installation is configured with can be read and searched in one place rather than one panel at a time.
- FEAT/UI: an input endpoint opens the flow panel, which is right for editing but not for reading one value. A panel of its own — the declaration, the current value, its history — is what clicking a label wants to give.
- FEAT/UI: sync between the header of the python function and the node configuration. The config→header half exists for ports *and* settings — `scaffoldFor` writes `def process(<ports>, <settings>)` and `editNode` keeps it in step — but only while the source is still exactly the generated scaffold (`SCAFFOLD_SHAPE`), and never for shared code. What is missing is the same for code someone has edited, and the reverse direction: nothing parses a `def process(...)` header back into ports and settings.
- BUG/UI on flows like "House history" where the widget sets the range for the "draw the window" node to generate some data, the edges overlap the nodes. We should adjust the flow visualization to account for these cyclic behaviors
- CHORE/UI: loop lag on Home reads a real number with no flows, and that is right — `LoopWatchdog` times how late `asyncio.sleep(1.0)` wakes on the API's event loop and is started unconditionally, so it measures the engine process rather than any flow, and it is what turns the health badge `degraded`. Nothing to fix; recorded so it is not reopened.
- BUG/UI auto node placement on flows should be improved in regards to least crossing edges and a more vertical layout on mobile devices
- INFRA: ensure that all the packages/ dependencies needed to run fluksio are available on arm to make this software runnable on e.g. raspbian
- INFRA: merge the philosophy statement at the beginning of vision.md into the rest of the document. Dissolve the decision dates and fold the decisions into a clean structure
- FEAT/UI introduce a graph panel which renders at the top right next to the graph view (to make more use of the horizontal space) and which allows (de-) selecting flows to be excluded from the graph view or search for individual nodes where only the flows containing this node should be shown (like slicing the brain)
- FEAT/UI labels in flows (indicating dashboard widget connections) naturally can't pulse. Instead add an animation (enlightning fade) from either ltr or rtl depending if the label is in- or outbound
- CHORE/UI: `layoutGraph` treats every node as 220 wide rather than measuring, because feeding a measurement back into the layout oscillates. A node wider than that crowds its neighbours; take the width from `node.measured` once it has settled if it shows. Height is no longer a guess — `nodeHeight()` derives it from the port count before dagre runs, so it cannot oscillate.
- FEAT/UI/MOBILE: a rank of many nodes — a connector feeding eight dashboard tiles — is thousands of pixels wide however the graph is turned, so on a phone the fit shrinks it past reading. The layout is right and the flow is simply too big for the screen; a "one rank at a time" reading mode, or wrapping a wide rank, is what would make it legible.
- CHORE/UI: an edge's value chip sits at the bezier midpoint while the layout reserves its room at dagre's label rank. The two agree closely enough today; if chips ever pile up, take the position from the layout instead.
- FEAT/UI (deferred until MCP lands): add a "bot" icon button to the home view (graph panel) which opens a chat window (reuse general concept of a side panel like in flows/nodes to make it a chat panel which can open on any screen (stacks below any other existing panel -> introduce stacking) to give support on errors/write code, generate dashboards etc) to explain the error(s)
- FEAT/UI make the header (Fluksio - YEAR) and the logo in the sidebar link to the main page (fluksio.com)
- FEAT/UI consider adding a diagram to the Home view which shows a histogram of the different classes of nodes and which time it takes to execute (logarithmic scale); this should give a hint on the load and help to detect bottle necks/hotspots
- CHORE/API: revoking an OAuth client does not invalidate access tokens already issued; they are stateless JWTs valid up to `MCP_TOKEN_EXPIRE_MINUTES`. Immediate revocation means `app/mcp/http.py` checking the client row still exists.
- CHORE/FLOW: `Pipeline.trigger`'s docstring says a paused flow still publishes so the value shows on the canvas. True only without a queue; with one the item parks before `apply_outputs` and nothing shows. Docstring and behaviour disagree.
### Persistence and databases
- CHORE/INFRA: Redis AOF runs at `appendfsync everysec`, so up to ~1 s of journaled work-queue entries can vanish on a crash — softer than "journaled before it runs" reads. Queue write volume is low, so `appendfsync always` is likely affordable; otherwise document the loss window.
- CHORE/INFRA: Redis has no auth (`requirepass` unset). Fine on the compose-internal network; a blocker for M5 remote workers, which turn Redis into a network-exposed shared bus.
### Connector write paths
- CHORE/NODE: the operatorId worry was unfounded — the reference Node-RED `setstat` node for this unit is configured with an empty operatorId and deviceId, so a command needs no registration. The second unit may still differ.
- PERF/NODE: a `wfrac` command is two round trips (read, then set) on the scheduler's thread, so at the default timeout a command can hold a cascade for several seconds. Fine for a person pressing a button; a flow commanding it on a schedule would want the work off that thread.
- CHORE/NODE: `wfrac` writes carry the unit's whole state, so two flows commanding one unit will each undo whatever the other set between their read and their write. One writer per unit, the same rule Art-Net has for a universe.
- FEAT/NODE: the second WF-RAC unit (the one Node-RED addresses with operatorId "0") closes the connection on an anonymous read. It likely wants an account registered; the first unit answers without one.
- CHORE/NODE: `wfrac` sometimes reports `mode` as "unknown" while the unit is off, because the mode bits hold a value outside the known set. Narrower than it first looked: a unit switched off *by a command* keeps its last mode in those bits and reads back correctly, so this is about however the remote turns it off. "off" would still read better than "unknown".
- PERF/NODE: `ArtNetOut.write` sends one frame per input port, so a node with two ports emits two frames per run. The last one carries both channels, so the end state is right; folding them into one send would halve the traffic.
- CHORE/NODE: the Art-Net node starts from an all-zero universe and has no way to learn what the fixtures are currently at — Art-Net has no read-back. Taking over a universe therefore blanks everything the flow does not drive. A baseline setting, or driving every channel, is what a real cutover needs.
### Porting the Node-RED flows
- CHORE/FLOW: 1CH values are still used raw, so those fixtures still never go
above 100/255, and the 4CH master channel still takes `v` on a different
scale from its colour channels. Both are `scale` and `master_raw` settings on
the encoders now, so fixing one is a decision about one fixture rather than a
surprise across all of them.
- FEAT/FLOW: the deferred half of the port — the media plug and the radio at
the media plug's address, the alarm clock, audio through `thgui/display/audio`, and the
fire alarm. The alarm's hook exists: every arbiter takes a `force_at` pulse,
so an alarm flow provides `safety.fire` and each actuator binds it with its
own safe value.
- CHORE/FLOW: the 433 MHz weather stations retain `status = dead`. Nothing
waits on them, but every humidity rule is inert until they are back.
- CHORE/INFRA: `TZ` now reaches the api container and is set to Europe/Berlin.
The container has to be recreated once before any cron means local time.
- CHORE/UI: a rollershutter is three buttons, because there is no cover widget
and no position to bind one to. A cover widget with an up/stop/down control
would be one tile instead of three, on four motors.
- PERF/FLOW: the MQTT node opens one connection per node, while
`docs/reference/node-types.md` says nodes sharing a broker share one. Eight
nodes speak to the house broker here. Either a shared client registry or a
correction to the sentence.
### Out-of-process nodes and modules
- CHORE/FLOW: a node's return value now round-trips through JSON, so tuples arrive downstream as lists and anything non-JSON is an explicit error. That is the message contract, but flows written before this may notice.
- CHORE/FLOW: a remote worker's agent spawns a fresh subprocess per call, so a node on a device pays for its imports on *every* call and pays them out of the node's own timeout — the cold-start problem local workers no longer have. Masked above ten seconds, because the agent's heartbeat resets the hub's silence deadline; a node with a shorter timeout and slow imports cannot run there at all. The fix is a persistent process on the agent, which is where its module cache would live.
- CHORE/FLOW: `worker_main.load_function` keeps every source it ever compiled in `sys.modules`, keyed by digest, and never drops one. A worker that has served many edits of one node holds all of them. Bounded in practice by `respawn_all` on a module change.
- CHORE/FLOW: `_remote_class` caps how many exception classes it caches but not how long a name may be, and for a remote worker that name comes off the wire. Local workers can only send a name their own code defined.
### Engine history
- CHORE/FLOW: a rate-limit flush gets no run record — it is the tail of the run that scheduled it, and there is no id linking the two. A flush that fails therefore shows as a failure with no run beside it.
- CHORE/FLOW: `Pipeline.flush` releasing a held value runs its cascade without a run id, so those executions land in the minute rollups but in no run. Threading the scheduling run's id through the queue item would close it.
- CHORE/FLOW: `EventBus.emits` counts a node's publishes so a reconnecting client can restore what it missed. Two deliberate shortcuts: the increment is a read-modify-write, so two threads emitting from one node can lose a count — `publish` is documented as never blocking, and a dropped increment is invisible in an animation — and the dict is never pruned, so a deleted flow's node ids sit there until restart. Bounded by distinct ids seen in the process, and orphans are never read, since lookups go through `brain_graph` members.
- CHORE/API: the metrics collector is a bus subscriber, so a storm that overflows the bus queue undercounts. The events dropped are the same ones the websocket drops; exact accounting would need the collector to be fed from the engine rather than the bus.
- CHORE/UI: the Home block's "Changes" list is the newest 15 audit rows whatever range is selected. Deliberate — an audit trail is worth reading past the window — but it sits under a control that governs everything else on the screen.
- CHORE/FLOW: run records for a deleted flow stay until the retention window passes, so a flow that no longer exists keeps appearing in the history. Deliberate — it is a record of what ran — but `forget_flow` could offer to clear it.
- CHORE/API: nothing can ask the collector to flush now, so anything needing the tables to be current has to wait out `FLUSH_INTERVAL_S` — which is what the soak harness does before clearing its own rows.
- CHORE/FLOW: `RedisWorkQueue.clear_flow` deletes only `pipeline:__parked__:{flow}`, so a deleted or renamed flow's `__queue__` stream entries, `__delayed__` zset members and `__done__:*` markers stay behind. The stream is capped and the entries are dropped when they reach a node that no longer exists, so it costs work rather than correctness.
- CHORE/FLOW: `MemoryWorkQueue`'s in-flight count is a counter around claim/ack, and claiming already removed the item — so an item a handler leaves unacknowledged (no pipeline bound) counts as in flight until the process ends. Nothing can hand it back either way, which is what the memory queue is.
- CHORE/API: audit rows ride the same drop-oldest bus as telemetry, so a storm can lose one. Writing a node's source is not audited either; publishing is.
- PERF/API: `queue.stats()` does a keyspace `scan_iter` on every call while two endpoints poll it.
- CHORE/INFRA: dev only — memory-queue ids (`mem-{seq}`) restart at 0 each boot and `FlowRun.id` is the primary key, so a restart without Redis upserts over the previous boot's run rows.
### Wall-panel
- CHORE/UI: identical in-flight chart requests are deduplicated per browser tab,
so two wall panels showing the same tile still run the query twice. An
`interval` on the request port is the backstop, and it belongs to the flow
serving the request rather than to the widget asking.
- CHORE/FLOW: one request/answer pair per InfluxDB node — the first input
carrying a `flux` key is the request and the answer leaves on the first output
port. A second query stream through one node needs a second node.
- CHORE/FLOW: porting the controls needs a declared writable message per control,
since an input widget can only target what a flow declares. Declaring one is no
longer API-only — the flow panel edits a flow's inputs and the canvas draws each
as a label — so the dashboard-input node this asked for has largely been
answered by flow inputs. What is left is the naming: an input a panel writes
looks the same as one a run passes in.
### Dashboard
- FEAT/NODE: the hosted demo places six of the fifteen built-in node types (`python`, `inject`, `change`, `join`, `rbe`, `trigger`); it does cover all fifteen dashboard widget types. `switch` and `delay` are the awkward ones — a `switch` branch needs either a dead-end port or trivial nodes to turn a branch back into a label, and neither reads as something a person would hang — while the I/O types (`mqtt`, `http`, `influxdb`, `exec`, `file`, `ntfy`, `mlp`) are unplaced because the demo has nothing real to talk to. Worth revisiting when the demo grows a second page.
- CHORE/UI: multi-page and multi-section dashboards have no UI and need none — a panel carries
several whole dashboards instead, each with its own canvas and its own publish. `PageDef`/
`SectionDef` stay in the schema and still round-trip; the editor and the panel both read a
page's sections as one grid, and section headings are no longer drawn. Recorded, not open.
- CHORE/API: a panel paired through the portal is revoked here the moment the panel is deleted — `_panel_may` finds nothing and answers 401 — but the hub's copy of the token stays valid until it expires or the installation's generation counter is bumped ("New code"). The hub has no per-panel revocation, and giving it one means telling it which panels exist, which is exactly what this design avoids. The generation bump is the lever; it is blunt, cutting every credential the portal minted for the installation. The per-panel nonce does not reach it either: `pnc` is only checked on a token this installation signed.
- CHORE/UI: the device line under a pairing code is the raw user agent plus the address the request came from. Both are self-reported and neither is proof; it is there so an admin can tell the screen they just hung from one they were not expecting, not to authenticate anything.
- CHORE/API: `POST /panels/pair` is reachable from the internet once an installation is enrolled — the hub forwards it without a session, since a device with no credential is the point of it. Bounded three ways (the hub's per-installation and per-address limits, and the fifty-code cap here), but it is the first unauthenticated surface this installation exposes outward.
- CHORE/UI: only `layout.lg` is ever written, and `md`/`sm` stay unwritten by decision — a phone stacks the widgets (`.widget-stacked`) rather than carrying an arrangement of its own, since arranging is not a phone feature. The keys stay in the schema for a panel that one day wants a second size.
- PERF/UI: `ChartWidget`'s cost per live value is the `uPlot.join` in `UplotChart`, not the tail append — the fetched half comes from React Query and is replaced wholesale on every refetch, so a ring buffer over the live tail would leave the dominant cost untouched. If this is ever profiled and fixed, the `setData` effect must stay dependency-free: a mutable buffer's identity never changes, so keying the effect on it reintroduces the staleness that the point-count dependency used to cause, and more quietly.
- CHORE/UI: opening edit mode on a dashboard whose widgets predate placement writes the migrated positions immediately, bumping the version once.
### Flow editor follow-ups
- PERF/FLOW: `FlowController._teardown` stops nodes one after another, each under `NODE_STOP_TIMEOUT`, so one flow's teardown is `k * 5.0` worst case for its `k` nodes. `asyncio.gather` over the flow's nodes would bound it at five seconds flat.
- CHORE/FLOW: a work item claimed just before a `replace_flow`, whose node belongs to the replaced flow, still applies its outputs and publishes `node_executed` for a node that has been torn down. Cosmetic — its cascade finds no edges and stops there. The fix is a flow-keyed in-flight counter in `ExecutionService` and a `pause_flow_intake(flow)` to go with it, which is only worth building if the stale event is ever actually seen.
- CHORE/FLOW: `FlowController.reload_lib_users` resolves who runs a shared node through `store.usages`, which reads *drafts* — so a flow whose draft references the library but whose published copy does not gets rebuilt for nothing. Harmless; it wants a published-only usage lookup rather than a second caller-side filter.
- CHORE/API: `POST /flows/{name}/rename` is no longer reachable from the UI. A flow's title is what the panel edits, matching how nodes work; the canonical name is fixed at creation, so either the endpoint goes or renaming comes back deliberately.
- CHORE/UI: ⌘C/⌘V `preventDefault` on the canvas blocks the native clipboard there (fields are guarded). The node clipboard is `localStorage`, so it does not cross browsers or profiles.
- PERF/UI: `useParamSuggestions` fetches every flow's detail to build the suggestion list. An aggregate endpoint if an installation ever has many flows.
- CHORE/UX: free-form params (python nodes) get no suggestions, since there is no schema to key them off.
- PERF/UI: `BrainView` runs 300 force-layout ticks synchronously inside a `useMemo`, so the graph is laid out on the render thread.
- FEAT/UI: the brain is a band on a scrolling page now, so it neither pans nor zooms — the fit keeps the whole graph in view instead. An installation with enough flows to make the labels unreadable at that fit needs a way to open the graph larger.
- CHORE/UI: React Flow measures a node's handle bounds out of the DOM once and never again, and in the brain that one measurement falls inside the graph's `scaleIn` entrance — so every `sourceX`/`targetX` it hands an edge there is the entrance's 4% short of the centre, permanently. `BrainEdge` takes both ends from the layout instead (position + radius). Any future view that mounts a canvas inside a transform and reads node internals meets the same thing.
### Infrastructure
- CHORE/INFRA: `make soak`'s redis scenario stops the container the whole stack shares, so every flow briefly fails to journal, not just the soak fixtures. They recover on their own — nothing was dead-lettered or quarantined in the run this note comes from — but it is not a thing to run against a stack someone is relying on.
- CHORE/INFRA: the soak harness's engine kill only catches a couple of items unacknowledged, because a cascade finishes in about four milliseconds. Redelivery is proven but barely stressed; a fixture node with a deliberate sleep would widen the window enough to test it properly.
## Deferred
- CHORE/OPS: the Postgres `app` database is an orphan on every installation
that predates 73eeec2 (2026-08-21, "Keep the engine's state in SQLite, not
Postgres"). Nothing reads it — the engine's database is `$DATA_DIR/fluksio.db`
and `docker/compose.yml` already puts `db` behind the analytics profile as
"Umami's database, and nothing else's" — but it still holds the pre-cutover
history (dev 17 MB / 43 runs, the hosted instance 15 MB / 4 runs) and is
stamped at `c4e8b2170f93`, a revision the squash deleted. Harmless, and
alarming to read: `alembic_version` says a revision that no longer exists.
Dump it if the history is wanted, then drop the database; on the hosted
instance keep the *container*, because Umami's database is in it. The `db`
service also still bootstraps `POSTGRES_DB=app` by default, which is what
recreates the confusion — `umami` would be the honest default now.
- FEAT/SDK: a traced flow body — `@Flow` over a function whose calls to other
nodes build the graph, the way Covalent's lattice does. Rejected for the first
version with reasons in `docs/private/python-api.md` (2.5× the code, two
meanings per decorated function, and several things it cannot express). The
registry holds `Flow` objects rather than modules, so a tracer producing the
same `FlowDef` would slot in without changing anything stored.
- CHORE/SDK: two `use()`s of one function in one flow store two identical node
bodies. `share_node` already models this — `_lib/<name>.py` plus a
`source_ref` — and sync could write the shared shim once. Not done because
editing a shared source bypasses draft/publish, which a generated body should
not.
- FEAT/SDK: `fluksio sync` puts the repository on `sys.path` from inside the
generated body, by absolute path. Right for the machine that synced, and
meaningless in a container — the note in the docs says to install the package
where the node runs. A `--repo-as` flag, or reading the installed location
back from the engine, would close it.
Open on purpose. Each names what should bring it back.
- PERF/UI: the app's entry chunk exceeds the warning threshold. React Flow and Monaco are already lazy; a manualChunks split measured no better, so this needs route-level work on the shell rather than chunking config.
- PERF/UI: the Monaco chunk is 2.6 MB. It only loads when a node panel opens, but the editor could be trimmed further or swapped for CodeMirror if that becomes a problem. NOTE: switch to codemirror; loading speed is definitely an issue.
- CHORE/API: node source saves carry no version precondition, so two clients editing the same node's code are last-writer-wins. The flow document is what the optimistic lock protects; code files would need their own, and an exact-match one produces false conflicts against a single client's own interleaved flow and source saves. Revisit with the M5 multi-user work.
- CHORE/FLOW: shared node sources bypass the draft/publish split. Editing one writes the library copy and reloads immediately, since the code is not any single flow's to hold back. Deliberate, but it means a shared node is the one thing publish does not gate.
- CHORE/INFRA: `requires-python` is capped below 3.14 because the MCP SDK wants a newer starlette there than the pinned `sentry-sdk<2` allows. Lift the cap when sentry-sdk moves to 2.x.
- FEAT/UI: the node-panel and edge trend curves take no range, unlike the health block. They are drawn from a Redis ring of the last 120 values per message, which has no window to ask for — a hover caption names what the curve covers instead of a picker promising a span nothing can serve. Reopen if per-message history ever gains a time window.
- FEAT/UI: an e-ink rendering profile for a dashboard — motion off, hover-only affordances resolved to something visible, high-contrast palette, thick strokes, and a repaint cadence low enough for a display that takes a second to settle. Reopen when a panel with such a display is actually hung.
- CHORE/INFRA: Postgres stays. The 2026-08 review rejected YugabyteDB/CockroachDB (multi-node cluster systems, ~4 GB+ RAM per node, against the small-server target — the scaling story is remote workers, not a distributed DB) and found merging Postgres into Redis or vice versa buys little: the stores hold disjoint data and both sit behind abstractions. SQLite would fit the single-instance design and drop a container; reopen if the home-install footprint becomes a product concern.
- CHORE/INFRA: NATS JetStream as the work-queue backend — durable streams whose consumer semantics match the `WorkQueue` interface, in one small binary. Reopen with M5 remote workers, when the queue crosses hosts. NOTE: remote workers landed without it — a worker dials the engine's own socket and never touches Redis, so the queue still does not cross a host. Reopen if a second engine ever pulls from the same stream.
- FEAT/RUNS: stage caching. `run_node.cache_key` is written on every run and the artifact store is content-addressed, so the pieces are in place; what is missing is computing the key from the node's source digest plus its input values and skipping a node whose key already has an `ok` row with its artifacts still present. The two research repos want this more than they want resume — neither persists checkpoints, and both re-run unchanged preprocessing every time.
- FEAT/RUNS: per-label requirements overlays (`requirements-gpu.txt`) synced into a remote worker's venv, with drift surfaced against the engine's manifest. Today a worker's environment is whatever `--python` points at, which is fine for one hand-managed GPU box and not for several. `venv_digest` already arrives at attach and is shown on `/workers`, so the reporting half exists.
- FEAT/UI: a dashboard shows a run's curve only while it is running. Emissions reach the socket live, but a run's values live in its own state namespace, so reloading the panel afterwards leaves the chart empty — the durable series is on the run (`/runs/{id}/metrics`) and nothing binds a widget to it. A chart variant that reads a run's series, or the existing querying chart pointed at `/runs/series/compare`, is what would close it. This is also what a demo needs to show a finished experiment rather than only a live one.
- FEAT/UI: launching a sweep is API-only. Pressing Run on a batch flow asks for its parameters, but the many-runs-at-once shape has no UI; a run detail screen is what it wants to land next to.
- FEAT/RUNS: a run detail screen. The API answers everything — params, per-node status with logs and tracebacks, artifacts, metrics, and `/runs/series/compare` in the chart widget's own `series` shape — but nothing in the dashboard reads it yet, so a run is inspected over HTTP. Comparing curves is a widget binding once someone builds the page around it.
- FEAT/RUNS: a thin client CLI (`fluksio run/runs/sweep/worker`) over the same API. The engine being resident is what makes runs cheap; a CLI is ergonomics on top, and `curl` covers it until someone is running sweeps daily.
- FEAT/RUNS: the step on a run's series is the count of emissions on that message, so a node yielding every tenth training step records steps 0, 1, 2 rather than 0, 10, 20 — a faithful x-axis of its own emissions, not of the loop inside it. If a real step number ever matters, a `record`-typed streaming port carrying its own `step` is the shape to read it from; the column is already there.
- CHORE/RUNS: an emission publishes on the node's port and, in a live flow, enqueues a cascade with no payload of its own — the value is already in state, and an item carrying it would re-apply that value whenever it was claimed, which is how a mid-node emission overwrites the one the node returned at the end. Downstream therefore reads what is current rather than the value that caused it to run. Right for a curve; worth revisiting if something ever needs every intermediate value delivered rather than sampled.
- CHORE/RUNS: `run_metric` has no retention. Deliberately outside `OBS_RETENTION_DAYS` — an experiment nobody deleted should not vanish on a rollup window — but a few thousand runs at 3000 steps will want a policy eventually, probably per-flow rather than global.
- CHORE/RUNS: a run holds one worker slot per node for its whole duration, and `MAX_PARALLEL` run drivers bound how many graphs are in flight. A sweep of 500 therefore queues behind the pool rather than the driver count. Fine — the GPU is the scarce thing — but the two limits are unrelated numbers that read as if they were one.
## Blocked
- CHORE/INFRA: `bun install` inside the frontend Docker build intermittently fails with
"Fail extracting tarball" for several packages at once, and succeeds on a plain rebuild.
It looks like concurrent extraction under memory pressure. Pin down or retry in the
Dockerfile if it starts costing CI time. NOTE: memory lifted; retry and close if stale
- CHORE/DOCS: `docs/` (the published site) documents `device` and `device_policy` as
API-only, because the node panel has no field for either. That is the one place the
public docs have to say "use the API instead of the UI". Adding a Device section to
`NodePanel.tsx` — a label field plus a require/prefer toggle — would close it.
- CHORE/DOCS: the site's node-type reference is hand-written from `NODE_TYPES` and each
node's `Params`. It will drift. `GET /flows/node-types` already returns the whole thing
with its schemas, so a generator (the way n3xd generates its command catalog) is the
obvious fix once the type list stops moving.
- CHORE: `frontend/vite.config.js` and `frontend/vite.config.d.ts` are committed build
output of `vite.config.ts` and fail `biome check` as tracked (semicolons, 4-space
indent). The pre-commit hook therefore fails for anyone who stages any other file
under `frontend/`. Either gitignore the two or run the formatter over them once.
- CHORE: the pre-commit biome hook selects files by the `frontend/` prefix rather than by
extension, so a non-JS file placed there is handed to biome and blows up with an
internal error on a doubled path (`frontend/frontend/...`).
ROADMAP.md
-394
View File
@@ -1,394 +0,0 @@
# Roadmap
Component-level breakdown. The milestone-level master (M1–M5, with the vision
decisions behind it) is `docs/private/roadmap.md` in the docs submodule.
Implementation strategy and record of existing/planned features. Completed items are
terse checklists — the requirement detail lives in `docs/private/vision.md` (goals,
requirements, decisions) and `docs/architecture/structure.canvas` (the four-way component split).
Remaining tasks keep enough scope to be actionable.
Legend: `[x]` done · `[ ]` planned · sub-lists split done vs. remaining for partial items.
Within each phase, remaining `[ ]` items are listed in rough priority order: making the
existing flow engine reachable and persistent precedes new feature breadth.
## Phase 0 — Workspace and platform
- [x] Root orchestrator repo with `app`, `index` and `docs` as submodules
- [x] `make init` bootstrap: secrets generation, per-stack `.env` propagation, shared
`proxy` docker network
- [x] Layered compose (`compose.yml` → `compose.dev.yml` → `compose.local.yml`) for both
stacks, one Traefik serving `${DOMAIN}`, `app.${DOMAIN}`, `api.${DOMAIN}`
- [x] Design token contract: root `DESIGN-GUIDELINES.md`, per-repo `DESIGN.md`,
byte-identical token blocks verified by `make design-check`
- [x] CI on Gitea (Forgejo Actions): pre-commit, backend tests, Playwright, compose smoke
— the four workflows in `.gitea/workflows/`. Never observed running on a
Gitea push from here, so treat the wiring as written but unproven
- [x] Soak and chaos harness: sustained load with the state backend, the broker and
the engine itself taken away underneath it. `backend/scripts/soak.py` behind
`make soak`, driving the durable webhook path through load, redis, engine and
broker scenarios, with every docker verb checked against an allow-list of this
stack's own containers. Two caveats: the redis scenario stops the container the
whole stack shares, and a cascade finishes fast enough that the engine kill
proves redelivery without stressing it
- [x] Startup benchmark: `backend/scripts/bench_startup.py` behind `make
bench-startup` times submitting a run against a Kedro project doing the same
nothing — 61 ms against 1110 ms, because nothing is booted per run. The
claim the ML-pipeline milestone is measured on, kept checkable rather than
asserted
## Phase 1 — Backend: management
Python, optimised for development speed. Owns the graph structure, persistence and the
external interfaces. See `docs/architecture/structure.canvas` → *Backend – Management*.
- [x] FastAPI + SQLModel + Alembic + Postgres base with JWT auth and user management
- [x] Flow engine in `backend/fluksio/flow/`: `Node` / `Pipeline` / `StateBackend`
(memory + Redis) / `FlowController`
- [x] Node types: HTTP, MQTT, InfluxDB, Delay, MLP
- [x] Flow-logic vocabulary as node types rather than repeated code: inject (manual,
interval, cron or at startup), switch, change, filter-unchanged, join, trigger,
command, file and ntfy. Each is configured by filling in a form the editor
generates from its parameter schema
- [x] `fluksio/flow` is an importable package with absolute `fluksio.flow.*` imports
- [x] Typed, serializable node I/O: every port declares a `DType`, messages are
JSON on the wire and in Redis, no pickle anywhere. Binary codecs are still
open — `DType.JSON` carries everything non-scalar for now
- [x] Message namespacing per flow (`flow.message`), with several producers per
message resolving to real fan-in
- [x] Secrets/credentials store for node integrations managed via the API/UI
(encrypted at rest, referenced from node params as `{"$secret": "name"}`);
`.env` bootstrap-only
- [x] Python modules for node code, managed from the UI: a pip manifest versioned
with the flows, installed with `uv pip sync` into a venv of the user's own on
the data volume. The worker processes run that interpreter, so an install
takes effect without restarting the engine and can never shadow the app's
own packages
- [x] Connector node contract: `ConnectorNode` with a declared contract version,
a polling coordinator that deduplicates, `x-secret` parameters the editor
renders as a secret picker, and health reporting. Connectors are installed
packages found through the `fluksio.node_types` entry point group; the
contract is documented at `docs/reference/connector-contract.md` with a
working skeleton at `connectors/skeleton/`. The registry follows later
- [x] First real connectors written against that contract from outside the engine:
WF-RAC aircon, calendar, UniFi presence and Art-Net, in `connectors/`.
Built by `make connectors` and installed into the image. Calendar and
UniFi read only; Art-Net and the aircon write, each behind a setting that
starts off
- [x] The other direction of the contract: `ConnectorNode.write` receives the
node's input ports, so a connector can command something rather than only
read it. Additive, so `CONTRACT_VERSION` stays at 1 — before this the base
class discarded every message reaching a connector, which made `artnet`'s
packet builder unreachable. Art-Net now sends: a per-port `channels` map
puts each input on its own DMX channel, `transmit` still gates the socket,
and one node owns one universe because a frame carries all 512 levels
- [x] The aircon writes too: power, mode, setpoint and fan speed, behind a
`commands` setting that starts off. A WF-RAC command carries the whole
state, so the node reads the unit and applies the change on top — the
encoder is a port of the same reference the decoder came from, and it
round-trips the unit's own live reading field for field. No operatorId
registration turned out to be needed: the unit this instance talks to
accepts an anonymous command, which is what the reference Node-RED node
does as well. Exercised against the real unit — fan speed, mode, setpoint
to the half degree, power off and back on — each checked against the
unit's own answer, and it was left as it was found
- [x] The aircon reaches both firmware generations: adapters shipped from 2025
report `firmType` `WF-RAC-HTTPS` and wrap port 51443 in TLS with a
self-signed certificate naming their MAC, where older ones answer plain
HTTP there and drop the handshake. Nothing announces which before the
first exchange, so the node tries TLS, falls back, and remembers — a unit
that gets updated costs one retry rather than a reconfiguration. The
payload is identical across both, so only the transport moved. Verified
against one unit of each generation, read and command
- [x] The aircon rig covers both units and both firmware generations: one
`wfrac` node each in `aircon_control`, polling as well as commanding, so
the panel shows what a unit answers next to what was asked of it. Three
things had to be fixed before a panel could actually drive one. The
seeded `commands` catch is off, which is the usual reason a fresh panel
looks dead — `AIRCON_COMMANDS=1` arms it at seed time. A unit that is off
names no mode and reports `unknown`, and a flow redelivers every bound
port on each run, so rejecting it made an idle unit impossible to start;
it now travels through untouched. And the old firmware serves one
connection at a time, answering an overlapping request with 501 — the
connector serialises per adapter and retries a busy one, which it must,
because other clients on the network are outside that lock
- [x] Node lifecycle as a protocol (`start`/`stop`/`report_health` on `Node`),
replacing the controller's per-type isinstance chains — the same hooks a
connector implements, validated on the built-in nodes first
- [x] Flow persistence: `flow.json` plus node sources per flow, replacing the
watch-directory prototype
- [x] REST + WebSocket API over the engine: create/read/update flows, edit node
source, run, and stream values, node status and execution events
- [x] Dependency-loop detection and graph validation surfaced as API errors
- [x] Per-flow start/stop, stored in a `runtime.json` beside the flow so it
survives a restart and stays out of the autosaved document; pause/resume
holds a flow's nodes while its values keep arriving
- [x] Node log streaming: what a node prints, and the traceback of one that
fails, reach the editor as `node_log` events
- [x] MQTT broker / InfluxDB compose services for local development
(`mosquitto` and `influxdb` in `docker/compose.dev.yml`)
- [x] Git-based versioning of the flow store (one commit per saved change)
- [x] Draft/publish split: edits autosave to `flow.draft.json` / `nodes.draft/`,
the engine runs only the published files, and publishing promotes the
draft. Saves carry the version they were based on, so a second client
editing the same flow is refused rather than overwritten
- [x] Installable with pip, and startable with no infrastructure: `fluksio` and
`fluksio-worker` are two distributions, `fluksio serve` makes its own data
directory, key, SQLite database and admin account, and `fluksio enroll
<code> --portal …` pairs a machine with no browser pointed at it. Proven
in a bare `python:3.12-slim`: installed, paired, dialled out, and the hub
proxied requests back through the tunnel. `git` and a Redis are both
optional — the flows are files and the state falls back to memory
- [x] SQLite as the database, on the data volume beside the flows. One process
owns it, which the single uvicorn worker already said; Postgres stays in
the stack only for the optional Umami profile. `DATA_DIR` is the one
setting the rest derive from, the schema is prepared in-process at
startup, and the ten Postgres-only revisions are one portable baseline
- [ ] Import/export of a flow as human-readable code plus a JSON structure
- [x] Per-input/-output discretization interval setting: a port publishes, or
wakes its node, at most every n seconds. State keeps the latest value, so
only the delivery is skipped
- [x] Alert / notification handler: engine failures — a node raising, a connection
dropping, a flow quarantined, the queue gone — reach ntfy, email or a webhook.
Mostly it declines to send: the same fault repeating is one alert with a count,
a flapping connection is muted, and there is a ceiling per hour. Configured
through the API at `/alerts/config`, with a test send per channel
- [x] Deep health check (`GET /utils/health/`): reports event-loop lag and state-backend
reachability and fails the container healthcheck, so a wedged engine is restarted
rather than counted as up. One engine per deployment — the API image runs a single
worker, because a second one would be a second engine
- [x] Supervised background tasks: a node's subscription, schedule or poll loop is
restarted with growing delay when it dies, and a flow that spends its failure
budget is quarantined and surfaced rather than left crash-looping. The loops
themselves no longer carry private retry logic
- [x] Durable work queue: every external trigger is journaled to Redis Streams before
anything runs and acknowledged once its cascade finishes, so an engine that dies
mid-cascade picks the work up again instead of losing it. A reaper reclaims what
a dead consumer never acknowledged; nodes that reach outside are skipped on a
redelivery they already ran. Long-lived worker pools replace the per-wave
executors, and a delay now waits in the queue rather than on a worker thread
- [x] Engine history in Postgres: a second bus subscriber folds executions, errors,
timings and queue lag into per-minute rollups, keeps failures with their
traceback and an audit trail of who published what, and records one row per
cascade — including the manual runs and previews that never went through the
queue. Read back through `/observability/*`, which always answers 200 so a
degraded engine still renders, and pruned on a retention window
- [x] Batch runs: a `mode: batch` flow taken from its declared inputs to its
declared outputs once, with parameters that identify it and a result kept.
Journaled to a Redis stream of its own and Postgres-authoritative from the
claim onwards, so a stale lease — not an unacked entry — marks a run whose
engine died. Each run executes an isolated pipeline over its own state
namespace, so a sweep's configs run in parallel without overwriting each
other's messages. `run`, `run_node`, `run_metric` and `run_artifact` are
separate from the cascade rollups, which are pruned on a retention window
and an experiment must not be. `/runs`, `/runs/{id}`, `/runs/flows/{name}`,
`/sweep`, `/cancel`, `/metrics` and `/series/compare`
- [x] A Python SDK, so a research repository is the source of a flow rather than
a place code is copied from: `@node` declares a function's ports where the
function is, `Flow(name, nodes=[...])` says which of them make a flow, and
`use(fn, wire=…, **settings)` rebinds one for a single flow. `fluksio
sync` uploads the document plus a generated import shim per node — so the
store still holds a complete, runnable, git-versioned definition — stamps
it with the repository's commit, and retires the workers so the next run
imports the code as it is now. `fluksio login|run|runs` and
`flow.submit().wait()` are the client half. The decorators return the
function untouched, which is the whole point: it is still your code, still
callable, and `device="gpu"` on the same decorator is where it runs
- [x] Streaming outputs: a node that produces values over time is a generator,
and every `yield` is a dict keyed by output port, published the instant it
happens; what it returns is its result. A port doing this declares
`stream: true`, and a run keeps every number one takes — so a training
curve is an output of the graph rather than a log beside it, and a chart
binds to it like any message. `fluksio.emit` writes the same ports for the
case a yield cannot reach, inside a framework's callback. The worker
protocol carries each emission as a frame before the reply, which also
turns `NodeDef.timeout` into an idle timeout: silence, not duration
- [x] Artifacts: `DType.ARTIFACT` carries a reference (digest, size, media type,
name) into a content-addressed store on the data volume, so bytes never
enter a message, Redis or the queue. The digest is the future stage-cache
key, which is why it is content-addressed now rather than per-run
- [x] Remote workers: a worker dials out to `WS /workers/attach` with an RS256
worker-scope token, advertises labels, and answers the same JSON protocol
the local pool speaks. `NodeDef.device` routes a node to one, resolved per
call; a run whose labels nothing carries waits saying so. A device-bound
node is compiled on that machine. The agent is one file plus `worker_main`,
with `websockets` as its only dependency
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
deployment on failure
- [ ] User management scoped per flow and per data set
- [x] MCP server over the same API: agents authenticate through a built-in
OAuth 2.1 authorization server (dynamic registration, PKCE, rotating
refresh tokens) and drive the flow API through 20 tools. Tokens are
RS256, signed with their own keypair, so the set can be revoked on its
own — and an additional issuer is one branch in `deps.decode_token`,
which is the seam remote access needs later
- [ ] LLM interface for natural-language flow authoring beyond the MCP tools
## Phase 2 — Backend: processing
Rust, optimised for throughput. Executes nodes and distributes them across workers. See
`docs/architecture/structure.canvas` → *Backend – Processing*.
- [ ] Parallel invocation of stateless nodes over independent input sets, to
keep I/O delay minimal (stateful I/O nodes keep serializing via the
`synchronous` mechanism)
- [x] Run user Python nodes out of process: a pool of persistent worker subprocesses
speaking one JSON object per line, entered through a proxy the controller
installs as the node's function, so every execution path funnels through it
unchanged. A crash costs one subprocess, a per-node timeout is a kill, and
cancelling from the canvas is that same kill on request
- [ ] Extract node execution from the Python prototype into a Rust engine
- [ ] Worker distribution and load balancing across capable devices
- [ ] Input/output validation at the node boundary
- [ ] Data aggregation and discretization
## Phase 3 — Frontend: admin view
React + Vite, primarily desktop but usable on mobile. See `docs/architecture/structure.canvas` →
*Frontend – Admin View*.
- [x] Dashboard SPA shell: TanStack Router, floating frosted sidebar, auth flows,
generated OpenAPI SDK
- [x] Node canvas (`@xyflow/react`) showing nodes and their connections, which
are derived from message names rather than stored
- [x] Tab-style view of atomic flows, with a floating dock
- [x] Embedded code editor (Monaco) for node source
- [x] Live values on the edges, with the last payload and its time on click
- [x] Provenance: every value says what caused it, so an edge pulses for the producer
that actually published rather than every producer of that message. A dashboard
control, another flow or an API caller is drawn as a label on the canvas instead
of being invisible — which also gives cross-flow wiring the link in/out it lacked
- [x] A node's settings are arguments of its function, next to its ports — one
`process(...)` signature covering both, no `params` dict, and a setting
sharing a port's name reported rather than shadowing it
- [x] A flow's boundary is on the canvas and in its panel: each declared input
is a label feeding what reads it (unless a control or another flow already
accounts for it), a batch flow's outputs hang off the end as its result,
and the panel edits mode, inputs and result — with the value a live flow
currently holds, and a way to put a new one in
- [x] Pressing Run on a batch flow asks for its parameters, prefilled from what
each input starts from, rather than quietly submitting the defaults
- [x] Validation shown on the node it belongs to, and summarised in the dock
- [x] Publish control and draft markers in the flow bar, discard in the flow
panel, and a conflict dialog when another client got there first
- [x] Marking a node reusable, and placing a shared one from the palette
- [x] Secret picker for credential parameters, so a password never lands in
`flow.json`
- [x] Dashboard showing which flows run, which are stopped and which have
errors, with a switch per flow
- [x] Logs panel in the canvas dock, pause/resume beside Run, and replaying an
edge's last message from the inspector
- [x] The canvas lays itself out — a layered graph, left to right on a desktop
and top to bottom on a phone, with room reserved for the value each edge
carries. Nodes cannot be dragged and a flow document holds no positions:
a graph nobody can arrange is one worth keeping small, which is what
keeps flows atomic
- [x] Usable on a phone, and written down so it stays that way: one breakpoint
(`md`), a stacked dashboard instead of a shrunken wall panel, a dock that
wraps rather than overflows, and a Playwright project that fails the
build when a screen no longer fits. See DESIGN-GUIDELINES.md → Responsive
- [ ] Device assignment per node, selectable from compatible devices
- [ ] Test-node affordance on the canvas
- [ ] User management screens
- [x] Screens for what the API used to own alone: the secrets store and the alert
channels/rules each get a sidebar page, and the OAuth clients an agent
registers are listed and revocable under Admin — which needed its
management endpoints written first
- [x] Health sections: how the engine is doing now (nodes, flows, queue, loop lag)
over what it has been doing all day — throughput and failure charts, a
per-flow table, the recent cascades, failures that expand to their
traceback, dead-lettered work and the audit trail. Hovering a chart
filters the list beside it to that minute and a click pins it
- [x] Brain graph: every published flow at once, with nodes that talk to the
same outside thing — a broker topic, a URL, a bucket — drawn as a single
neuron, so the wiring that runs between flows through a broker is visible at
all. Laid out by a force simulation settled once and then frozen, lit by the
same socket the editor listens to, and read-only: a neuron leads back to the
flow it came from
- [x] Both of the above sit on Home rather than at routes of their own: the brain
flat across the top, the health sections under the flow switches. One
overview instead of three
- [x] Mobile-friendly canvas: touch connect, full-screen node panel
- [ ] Installable as a PWA (`vite-plugin-pwa`)
## Phase 4 — Frontend: dashboard view
Shares components with the admin view. See `docs/architecture/structure.canvas` →
*Frontend – Dashboard View*.
- [x] User-defined dashboard layout with edit and view modes: dashboards are their own
documents, widgets bind to message names, and the input ones publish back. View
mode is plain CSS grid, so a panel that only displays loads no editing code
- [x] Chart widget drawing a message's history through uPlot, with `--chart-1…5`
as one lightness ramp of the brand hue; a widget bound to the wrong dtype,
or to nothing, is flagged the way a failing node is
- [x] Layout by dragging and resizing (react-grid-layout), a grid size per
dashboard, and `/view/{name}` — a full-bleed route that loads neither the
editor nor the grid library, which is what a wall panel is pointed at
- [x] Draft and publish, as flows have it: the editor autosaves
`dashboard.draft.json` and a panel reads only the published document, so a
half-arranged page never reaches the wall until someone publishes it
- [x] Declared structured payloads — `series`, `record` and `list` (with its item
type) join the scalars in `DType` and are checked the same way, so a widget
binds a shape rather than "some JSON" and a wrong binding is refused before
anything runs. An agenda over a `list` and a notification over a `record`
are the first two to read them, the latter fed either by a flow or by the
`dashboard` alert channel
- [x] Charts that query: a chart publishes a request — the window and the
resolution — exactly as a slider publishes a value, and draws the `series`
a flow answers with. The answer says what it was computed for and one
computed for another window is ignored, so two charts on a node cost a
duplicate query rather than the wrong picture. Database nodes stay
transport and credentials only: the InfluxDB node runs Flux handed to it
and echoes the rest, and Python nodes either side build the query and shape
the answer — which is what keeps the widget ignorant of the database
- [x] Per-device view: a *panel* is one screen and the ordered set of whole
dashboards it shows, so a hallway tablet and a workshop tablet carry
different sets without either dashboard knowing about the other. More than
one and the device draws a rail to switch between them — the same rail the
editor puts on screen, because the wall has it and it takes room off the
canvas. A screen has no keyboard, so it pairs instead of logging in: it
shows a six-character code, somebody approves it against a panel from the
dashboards overview, and the credential that mints is scoped to that
panel's dashboards and the message endpoints its widgets speak. Deleting
the panel revokes it
- [x] Pair a panel through the portal, for a screen hanging somewhere the
installation is not reachable from. A fourth hub token class, `scope=panel`,
named by panel instead of by person: it passes the hub's `_authorize`
without the owner check no panel could satisfy, and arrives here through
`decode_portal_token` carrying the panel, where the same `_panel_may` that
bounds a locally paired screen bounds it. The installation asks for it
holding the credential it dials the tunnel with, so the portal decides
nothing but which installation it is for. The three gates opened one each:
the shell serves `/panel` alone without a session, the proxy forwards the
two pairing calls without one — rate-limited per installation and per
address, and stripped of any bearer the browser tried to send — and the
credential is traded for the hub's cookie rather than carried in the URL a
year-long token must never sit in. What is waiting on a code is named
before anyone approves it
- [x] Remote users: a portal account other than the installation's owner reaches
it, as a local user of its own. The mirror of the claim code — the person
wanting in mints a code on their portal account, and a superuser here
redeems it, so admission is a local decision made by someone who already
had the right to grant remote access at all. The installation redeems it
holding its tunnel credential rather than a portal session, which is what
stops an admitted user from admitting anyone else; the account they get is
never a superuser, so the same is true from this side. `user.portal_sub`
is where a portal identity meets a local account, set for the enrolling
superuser at enrolment and for each admitted user after. A proxy token now
names a person rather than resolving to whoever enrolled, and an unmapped
identity resolves to no user — so deleting the local row is the whole of
the revocation, immediate even against a credential already in flight and
even when the portal cannot be reached to be told. The hub's `_authorize`
grew one member lookup beside the owner comparison, which is the only
thing that changed about the tunnel. Enrolments made before the mapping
existed adopt it from the handshake — the hub names the owner in its
welcome frame — because the alternative was locking an owner out of a
machine they could only have fixed by standing in front of it
## Phase 5 — Website and docs
- [x] Marketing site with a live node-graph demo, shared design system
- [ ] Published documentation site fed from the `docs` submodule
- [ ] Umami analytics configured (the site still ships the placeholder script)