From 3e084c06cfdae1150cd9a94a12ffbede1ced7ff7 Mon Sep 17 00:00:00 2001 From: stroblme Date: Fri, 21 Aug 2026 10:14:06 +0200 Subject: [PATCH] NOTEPAD: drop what this wave closed, file what it turned up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Twenty-one items are done: the canvas viewport and logs panel, the widget overflow and stacking work, the backup and queue-namespace changes, and the observability fold. What replaced them is mostly the residue of doing them — a legend styled in the wrong CSS chunk, two overflow floors that nothing would notice regressing, and artifact blobs that outlive the rows pointing at them. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Uq8mtNb97A7praJLyeEYgs --- NOTEPAD.md | 47 +++++++++++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/NOTEPAD.md b/NOTEPAD.md index 44a0cc4..3b9803a 100644 --- a/NOTEPAD.md +++ b/NOTEPAD.md @@ -12,35 +12,49 @@ Deferring because out of scope is fine, but don't mention deferring than. ### To be sorted -- BUG/UI on mobile, the width of the widgets in a dasboard should resize such that the screen is not horizontally scrollable. It usually works for most of the widgets, but some still render scrollable (tested on Pixel 8). Furthermore, long text in text widgets should become vertically scrollabe, as the text content just renders over the widget title -- FEAT/UI center a node when being selected, so that the panel does not overdraw the node +- CHORE/UI: `.u-legend` is styled in the dashboard's own CSS chunk, so a uPlot legend on a + Health or Home page renders unstyled until a dashboard has been visited in that session. + `RangePicker` already side-effect-imports `dashboard.css` for the segmented thumb; the + legend rules want the same treatment, or a shared chart stylesheet. +- CHORE/UI: the bar widget's readout swaps between an inline `right` and an inline `left` as + the value crosses 30%, and the side it stops setting resets to `auto`, which does not + interpolate — so the label jumps once at that threshold while everything else animates. + Positioning it always by `left` plus a `translateX(-100%)` would put the whole travel on + one property. +- CHORE/UI: two of the mobile-overflow floors are over-determined. Removing + `.widget-grid { min-width: 0 }`, or the segmented fieldset's `min-w-0`, leaves the mobile + suite green — the grid tracks are already `minmax(0, 1fr)` and the fieldset became a grid. + The uPlot legend is the one offender the assertion actually catches. Both are cheap + insurance for a future widget, but nothing would notice if they regressed. +- CHORE/API: artifact blobs are content-addressed and have no GC, so deleting a flow drops + its `run_artifact` rows and leaves the bytes on the data volume. +- CHORE/API: `DELETE /flows/{name}` does not refuse while the flow has a `running` or + `queued` Run. `RunService._record_node` can then insert `run_node` rows for a run that no + longer exists; `_finish` is an UPDATE, so it degrades to a harmless 0-row no-op. +- CHORE/INFRA: `backend/htmlcov` can be left root-owned by a container run, and + `make test-backend` then fails at the coverage HTML step *after* every test has passed — + which reads like a test failure and is not one. +- CHORE/INFRA: `.gitea/workflows/playwright.yml` says it shards "Two spec files + (tests/flows.spec.ts, tests/admin.spec.ts)". There are nine. - FEAT/UI add (multi-)select to the flows and dashboards view to allow deleting (multiple) items; long press to select -> "Add" button should change into "Trash" icon button -- BUG/UI remove the zoom in/zoom out buttons from the toolbar; either we have a mouse or use touch gesture for that - BUG/UI in the brain view: make the chasing circle animation running entirely in the gap between the ring and the node (using the full width) -- FEAT/UI add animation to widgets; i.e. status of bars, gauges etc. should fade from one state to another. Multi-buttons (like "Mode" in the "Home" dashboard of the demo) should transition from one state to another; use inspiration for animations based on the google material guidelines - BUG when clicking "edit" in the "Home" dashboard of the demo on hub.fluksio.com, most of the panels disappear (only a handfull is left for actual edit) - CHORE/UI: the edge popover shows the same value twice — `MessageSparkline` falls through to a collapsed `ValuePreview` for a non-numeric value, and `EdgeInspector` then renders its own `ValuePreview defaultOpen` below it. Cosmetic; one of the two is redundant. - FEAT/UI: a settings-and-inputs overview page, so what every node of an installation is configured with can be read and searched in one place rather than one panel at a time. - FEAT/UI: an input endpoint opens the flow panel, which is right for editing but not for reading one value. A panel of its own — the declaration, the current value, its history — is what clicking a label wants to give. - FEAT/UI: sync between the header of the python function and the node configuration. The config→header half exists for ports *and* settings — `scaffoldFor` writes `def process(, )` and `editNode` keeps it in step — but only while the source is still exactly the generated scaffold (`SCAFFOLD_SHAPE`), and never for shared code. What is missing is the same for code someone has edited, and the reverse direction: nothing parses a `def process(...)` header back into ports and settings. - BUG/UI on flows like "House history" where the widget sets the range for the "draw the window" node to generate some data, the edges overlap the nodes. We should adjust the flow visualization to account for these cyclic behaviors -- BUG/UI when enlarging the code editor of a node, the code editor should enlarge to the left (node settings remain on the right) so that the code editor fills the center of the screen with the node properties available next to it (still in the same panel, essentially just a second column). Note that this overdraws the toolbar and flow name box which should move out of the screen towards the bottom and top respectively -- BUG/UX the console/log panel should be moved up slightly (to separate from the toolbar) and kept open (console button becomes stateful) and center over the toolbar - CHORE/UI: loop lag on Home reads a real number with no flows, and that is right — `LoopWatchdog` times how late `asyncio.sleep(1.0)` wakes on the API's event loop and is started unconditionally, so it measures the engine process rather than any flow, and it is what turns the health badge `degraded`. Nothing to fix; recorded so it is not reopened. - BUG/UI auto node placement on flows should be improved in regards to least crossing edges and a more vertical layout on mobile devices - FEAT/UI we should highlight failing nodes accordingly in the flow view to facilitate easier tracking of mis-configurations - INFRA: ensure that all the packages/ dependencies needed to run fluksio are available on arm to make this software runnable on e.g. raspbian - INFRA: merge the philosophy statement at the beginning of vision.md into the rest of the document. Dissolve the decision dates and fold the decisions into a clean structure -- CHORE/INFRA: `make test-backend` reads `app/.env`, so a checkout configured for a deployment fails two tests that assume the development defaults — `test_private.py::test_create_user` (the `/private` router is off unless `ENVIRONMENT=local`) and `test_mcp_http.py::test_an_agent_can_list_and_call_tools` (the MCP host allow-list is built from `DOMAIN` while the test speaks to `api.localhost`). Both pass under `ENVIRONMENT=local DOMAIN=localhost`; the suite should pin its own environment rather than inherit the deployment's. -- CHORE/DOCS: `app/development.md` is still the upstream template's text — `compose.override.yml`, `localhost.tiangolo.com`, `docker compose watch` as the dev flow — so it disagrees with the Makefile about how the stack is started and says nothing about the Playwright suite. The containerised run above is what it should carry. - FEAT/UI add a loading animation for the initial app load and when loading individual pages; make sure that elements e.g. in the home dashboard load independently to ensure a fast loading of the initial site but figures charts, tables, graph etc. follow after that - FEAT/UI introduce a graph panel which renders at the top right next to the graph view (to make more use of the horizontal space) and which allows (de-) selecting flows to be excluded from the graph view or search for individual nodes where only the flows containing this node should be shown (like slicing the brain) - FEAT/UI labels in flows (indicating dashboard widget connections) naturally can't pulse. Instead add an animation (enlightning fade) from either ltr or rtl depending if the label is in- or outbound -- CHORE/UI: `UplotChart` pushes new readings only when the *number* of points changes (`useEffect(..., [points, key])`), so a refetch that returns the same count with different values leaves the chart drawing the old ones. Harmless while a window is still filling up — each minute adds a bucket — but a full rolling window keeps a constant count, and the chart then stops following the data. - CHORE/UI: `layoutGraph` treats every node as 220×56 rather than measuring, because feeding a measurement back into the layout oscillates. A node wider than that crowds its neighbours; take the sizes from `node.measured` once they have settled if it shows. - FEAT/UI/MOBILE: a rank of many nodes — a connector feeding eight dashboard tiles — is thousands of pixels wide however the graph is turned, so on a phone the fit shrinks it past reading. The layout is right and the flow is simply too big for the screen; a "one rank at a time" reading mode, or wrapping a wide rank, is what would make it legible. - CHORE/UI: an edge's value chip sits at the bezier midpoint while the layout reserves its room at dagre's label rank. The two agree closely enough today; if chips ever pile up, take the position from the layout instead. -- CHORE/UI: the bar widget's nested fill is `--chart-5` against a `--primary` outer fill, which measures 2.53:1 in light mode — under the 3:1 guideline for non-text. The `inset-y-1` gutter supplies the edge, and every other slot of the ramp collapses in dark mode instead. - FEAT/UI (deferred until MCP lands): add a "bot" icon button to the home view (graph panel) which opens a chat window (reuse general concept of a side panel like in flows/nodes to make it a chat panel which can open on any screen (stacks below any other existing panel -> introduce stacking) to give support on errors/write code, generate dashboards etc) to explain the error(s) - FEAT/UI make the header (Fluksio - YEAR) and the logo in the sidebar link to the main page (fluksio.com) - FEAT/UI consider adding a diagram to the Home view which shows a histogram of the different classes of nodes and which time it takes to execute (logarithmic scale); this should give a hint on the load and help to detect bottle necks/hotspots @@ -50,11 +64,8 @@ Deferring because out of scope is fine, but don't mention deferring than. From the 2026-08 database review. Verdict recorded under Deferred: the Postgres + Redis + git-files split stays; the actionable part is durability. -- BUG/INFRA: `scripts/backup.sh` dumps Postgres only — the flow git repo, `secrets.enc`, `oauth-key.pem` (losing it invalidates every issued MCP token), `alerts.json` and the Redis AOF are not backed up. Extend it to tar `app-flow-data` and copy a `BGSAVE`'d Redis snapshot. Its header comment still claims "the flow engine keeps no on-disk state", which stopped being true when flows became a git repo. -- CHORE/FLOW: state backend and work queue share the `pipeline:` prefix in db 0, so `RedisState.clear()` would DEL the work-queue stream and `RedisState.keys()` enumerates queue keys — only callers filtering `__`-prefixed names keep it safe. A separate prefix (or db index) for the queue removes the hazard. - CHORE/INFRA: Redis AOF runs at `appendfsync everysec`, so up to ~1 s of journaled work-queue entries can vanish on a crash — softer than "journaled before it runs" reads. Queue write volume is low, so `appendfsync always` is likely affordable; otherwise document the loss window. - CHORE/INFRA: Redis has no auth (`requirepass` unset). Fine on the compose-internal network; a blocker for M5 remote workers, which turn Redis into a network-exposed shared bus. -- CHORE/INFRA: the SQLAlchemy engine sets no `pool_pre_ping`, so long-idle connections throw once after a Postgres restart before the pool recovers. ### Connector write paths @@ -95,7 +106,6 @@ is lost on its restart. - CHORE/API: revoking an OAuth client does not invalidate access tokens already issued; they are stateless JWTs valid up to `MCP_TOKEN_EXPIRE_MINUTES`. Immediate revocation means `app/mcp/http.py` checking the client row still exists. - CHORE/FLOW: `Pipeline.trigger`'s docstring says a paused flow still publishes so the value shows on the canvas. True only without a queue; with one the item parks before `apply_outputs` and nothing shows. Docstring and behaviour disagree. -- CHORE/API: `GET /observability/runs` caps at 200 rows, so pinning a minute busier than that on the Home chart shows its newest 200 with nothing saying more exist. Fine at ~60 runs/min; a count alongside the rows, or paging, is what a busier instance needs. ### Out-of-process nodes and modules @@ -110,18 +120,14 @@ is lost on its restart. - CHORE/FLOW: a rate-limit flush gets no run record — it is the tail of the run that scheduled it, and there is no id linking the two. A flush that fails therefore shows as a failure with no run beside it. - CHORE/FLOW: `Pipeline.flush` releasing a held value runs its cascade without a run id, so those executions land in the minute rollups but in no run. Threading the scheduling run's id through the queue item would close it. - CHORE/FLOW: `EventBus.emits` counts a node's publishes so a reconnecting client can restore what it missed. Two deliberate shortcuts: the increment is a read-modify-write, so two threads emitting from one node can lose a count — `publish` is documented as never blocking, and a dropped increment is invisible in an animation — and the dict is never pruned, so a deleted flow's node ids sit there until restart. Bounded by distinct ids seen in the process, and orphans are never read, since lookups go through `brain_graph` members. -- CHORE/API: nothing deletes a flow's `Run` rows. Deleting a flow clears its values and its history but leaves the runs, so a demo that is reseeded accumulates them. - CHORE/API: the metrics collector is a bus subscriber, so a storm that overflows the bus queue undercounts. The events dropped are the same ones the websocket drops; exact accounting would need the collector to be fed from the engine rather than the bus. -- CHORE/API: `/observability/summary` reports the work queue's `depth` as the Redis stream length, which is the journal size (capped at `STREAM_MAXLEN`) rather than a backlog. The health screen shows `pending` instead; the field name still invites the wrong reading. -- PERF/API: the health block picks its window now, but `/observability/timeseries` and `/observability/flows` still read every `metric_minute` row in it and fold them in Python. `bucket_s` only coarsens what comes back, so the 7d preset pulls a week of rows on each 30 s poll. `date_bin()` is what makes the long windows cheap. -- CHORE/API: `/observability/summary` still returns `failures_24h`, which nothing reads any more — the Home tile counts errors over the selected window from the rollups instead. Drop the field, or let the summary take a window. - CHORE/UI: the Home block's "Changes" list is the newest 15 audit rows whatever range is selected. Deliberate — an audit trail is worth reading past the window — but it sits under a control that governs everything else on the screen. - CHORE/FLOW: run records for a deleted flow stay until the retention window passes, so a flow that no longer exists keeps appearing in the history. Deliberate — it is a record of what ran — but `forget_flow` could offer to clear it. - CHORE/API: nothing can ask the collector to flush now, so anything needing the tables to be current has to wait out `FLUSH_INTERVAL_S` — which is what the soak harness does before clearing its own rows. - CHORE/FLOW: `RedisWorkQueue.clear_flow` deletes only `pipeline:__parked__:{flow}`, so a deleted or renamed flow's `__queue__` stream entries, `__delayed__` zset members and `__done__:*` markers stay behind. The stream is capped and the entries are dropped when they reach a node that no longer exists, so it costs work rather than correctness. - CHORE/FLOW: `MemoryWorkQueue`'s in-flight count is a counter around claim/ack, and claiming already removed the item — so an item a handler leaves unacknowledged (no pipeline bound) counts as in flight until the process ends. Nothing can hand it back either way, which is what the memory queue is. - CHORE/API: audit rows ride the same drop-oldest bus as telemetry, so a storm can lose one. Writing a node's source is not audited either; publishing is. -- PERF/API: two unmarked ceilings — `/observability/flows` scans every `metric_minute` row in the window on each 30 s poll, and `queue.stats()` does a keyspace `scan_iter` on every call while two endpoints poll it. +- PERF/API: `queue.stats()` does a keyspace `scan_iter` on every call while two endpoints poll it. - CHORE/INFRA: dev only — memory-queue ids (`mem-{seq}`) restart at 0 each boot and `FlowRun.id` is the primary key, so a restart without Redis upserts over the previous boot's run rows. ### Wall-panel parity with the current home dashboard @@ -151,7 +157,6 @@ Decisions taken up front, because most items below depend on them: - Nothing e-ink-specific in the widgets. Panel access is a credential problem (see below); the display's demands are a rendering profile, deferred. -- BUG/UI multiple entries in the bar diagram should draw stacked instad of drawn over another - CHORE/UI: identical in-flight chart requests are deduplicated per browser tab, so two wall panels showing the same tile still run the query twice. An `interval` on the request port is the backstop, and it belongs to the flow @@ -204,10 +209,8 @@ as an em dash. - BUG/UI: the payload of the inject node should adapt to the type set in the "provides" arguments and the number of fields should grow with the number of "provides" arguments. So if e.g. one argument is boolean, the payload should be a dropdown for true/false. If it is int, the value entered should be parsed as int etc. `payload` is typed `Any`, so `ParamsForm` renders one plain text field and never sees the node's `provides` at all. The dtype→widget pieces to reuse are `parseByDtype` and the bool dropdown in `FlowBoundary.tsx`, written for flow inputs and the run dialog. - CHORE/UI: a node's error status clears as soon as it runs again, so a failure that genuinely fired an alert can leave no trace on the canvas by the time anyone looks. The logs panel keeps the traceback; the node itself reads as healthy. -- FEAT/UI: the flow graph should auto zoom-to fit when a panel is open and edges are created through assigning inputs/outputs. Declaring a *flow-level* input or output already refits, since it adds an endpoint and the effect watches `external.nodes.length`; editing a node's own ports does not, and neither does opening a panel. - PERF/FLOW: every save rebuilds the whole pipeline. Fine at the current flow count; rebuild only the touched flow when it starts to show. - CHORE/API: `POST /flows/{name}/rename` is no longer reachable from the UI. A flow's title is what the panel edits, matching how nodes work; the canonical name is fixed at creation, so either the endpoint goes or renaming comes back deliberately. -- BUG/UI: `renderedNodes` overwrites xyflow's own `selected` flag with `id === selectedId`, so a box-selection of several nodes is invisible even though delete and copy act on all of them. - CHORE/UI: ⌘C/⌘V `preventDefault` on the canvas blocks the native clipboard there (fields are guarded). The node clipboard is `localStorage`, so it does not cross browsers or profiles. - PERF/UI: `useParamSuggestions` fetches every flow's detail to build the suggestion list. An aggregate endpoint if an installation ever has many flows. - CHORE/UX: the derived-cron chip also appears on the delay node, where `interval` is a rate limit rather than a schedule. May want it inject-only.