Add the flow API: typed messages, git-backed store, REST and live events

Makes the flow engine reachable from the API, which is what M3 needs before
any of it can reach the browser.

- app/flow is a package now; the prototype's watch-dir scripts and the
  matplotlib/networkx visualiser are gone with their dependencies.
- Messages carry a serializable dtype instead of a live Python type, and a
  port name, so the graph can speak qualified names while node functions keep
  local arguments. Redis state is JSON, not pickle.
- Message names are namespaced per flow ("heating.temp"); a bare name resolves
  to its own flow, a dotted one crosses flows.
- Several nodes may provide the same message: producers are a list, so fan-in
  is a real edge instead of a silently dropped one.
- Flows are stored as flow.json plus node sources in a git repository, one
  commit per save, with identical saves skipped so autosave stays quiet.
- Node failures are isolated and reported per node; validate() returns cycles
  and unconnected inputs instead of raising deep in a run.
- Credentials live in an encrypted store and are referenced as {"$secret": …}.
- Engine events reach websocket clients through a bus, so values, node status
  and execution show up live.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016WzrvW7rjQbynnhF6pxh6i
This commit is contained in:
Melvin Strobl
2026-08-15 17:23:36 +02:00
co-authored by Claude Fable 5
parent 61be29827d
commit 06a4506767
54 changed files with 2586 additions and 4978 deletions
+16 -12
View File
@@ -8,29 +8,33 @@ Deferring because out of scope is fine, but don't mention deferring than.
## Open
- CHORE/FLOW: `backend/app/flow/` is not an importable package. There is no `__init__.py`
and siblings are imported top-level (`from util import …` in `nodes.py:6`, `from nodes
import …` in `controller.py:23-26` and `pipeline.py:17-19`), so the modules only run with
the cwd set to `app/flow/`. Convert to `from app.flow.util import …`.
- CHORE/FLOW: the flow test files live in `backend/app/flow/test*.py`, inside the package
rather than `backend/tests/`, so `coverage run -m pytest tests/` never collects them.
- CHORE/INFRA: port the n3xd CI workflows (pre-commit, backend tests, Playwright shards,
compose smoke) to `.gitea/workflows/`. The Gitea instance runs Gitea Actions and needs CI
enabled per repository.
- FEAT/UI: `@xyflow/react` and a Monaco editor belong here once the node canvas starts.
The website already demos `@xyflow/react` on its features page.
- CHORE/FLOW: `app/flow/nodes.py` and `state.py` are excluded from strict mypy
(`[[tool.mypy.overrides]]` in `pyproject.toml`). The node classes still carry prototype
typing and redis-py types its calls as sync/async unions. Shrink the exclusion as each
integration is revisited.
- FEAT/API: per-hook shared secrets for `HttpNode` webhooks. They are mounted unauthenticated
under `/hooks/{flow}/…` because devices cannot present a JWT.
- FEAT/FLOW: state does not survive a restart — the engine runs on `MemoryState` unless
`REDIS_HOST` is set, and there is no Redis compose service yet.
- CHORE/FLOW: rotating `SECRET_KEY` makes the stored secrets unreadable, since the Fernet key
is derived from it. Needs a re-encrypt path before anyone rotates in production.
- PERF/FLOW: every save rebuilds the whole pipeline. Fine at the current flow count; rebuild
only the touched flow when it starts to show.
- FEAT/UI: reintroduce `--chart-*` tokens as one designed sequential scale when the first
chart lands. The stock shadcn five were dropped because their light and dark values are
unrelated hues.
- CHORE/API: `backend/app/models.py` still holds only the template's `User` and `Item`.
`Item` should go once flow persistence replaces it.
- CHORE/API: `backend/app/models.py` still holds the template's `Item` model. It has no
purpose now that flows are persisted and can go with its routes and tests.
- PERF/UI: both frontend bundles exceed the 500 kB warning threshold. Revisit chunking
once the canvas and editor are in, since they will dominate.
## Blocked
- FEAT/INFRA: Redis, MQTT broker and InfluxDB compose services. Blocked on the flow engine
being reachable from the API — no runtime path imports them today.
- FEAT/INFRA: MQTT broker and InfluxDB compose services for local development. The node types
exist; a local broker would make them testable without external hardware.
- CHORE/INFRA: `bun install` inside the frontend Docker build intermittently fails with
"Fail extracting tarball" for several packages at once, and succeeds on a plain rebuild.
It looks like concurrent extraction under memory pressure. Pin down or retry in the