Add the flow API: typed messages, git-backed store, REST and live events

Makes the flow engine reachable from the API, which is what M3 needs before
any of it can reach the browser.

- app/flow is a package now; the prototype's watch-dir scripts and the
  matplotlib/networkx visualiser are gone with their dependencies.
- Messages carry a serializable dtype instead of a live Python type, and a
  port name, so the graph can speak qualified names while node functions keep
  local arguments. Redis state is JSON, not pickle.
- Message names are namespaced per flow ("heating.temp"); a bare name resolves
  to its own flow, a dotted one crosses flows.
- Several nodes may provide the same message: producers are a list, so fan-in
  is a real edge instead of a silently dropped one.
- Flows are stored as flow.json plus node sources in a git repository, one
  commit per save, with identical saves skipped so autosave stays quiet.
- Node failures are isolated and reported per node; validate() returns cycles
  and unconnected inputs instead of raising deep in a run.
- Credentials live in an encrypted store and are referenced as {"$secret": …}.
- Engine events reach websocket clients through a bus, so values, node status
  and execution show up live.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016WzrvW7rjQbynnhF6pxh6i
This commit is contained in:
Melvin Strobl
2026-08-15 17:23:36 +02:00
co-authored by Claude Fable 5
parent 61be29827d
commit 06a4506767
54 changed files with 2586 additions and 4978 deletions
+38
View File
@@ -1,3 +1,7 @@
import asyncio
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
import sentry_sdk
from fastapi import FastAPI
from fastapi.routing import APIRoute
@@ -5,6 +9,11 @@ from starlette.middleware.cors import CORSMiddleware
from app.api.main import api_router
from app.core.config import settings
from app.flow.controller import FlowController
from app.flow.events import event_bus
from app.flow.secrets import init_secrets
from app.flow.state import MemoryState, RedisState, StateBackend
from app.flow.store import FlowStore
def custom_generate_unique_id(route: APIRoute) -> str:
@@ -14,10 +23,39 @@ def custom_generate_unique_id(route: APIRoute) -> str:
if settings.SENTRY_DSN and settings.ENVIRONMENT != "local":
sentry_sdk.init(dsn=str(settings.SENTRY_DSN), enable_tracing=True)
def _state_backend() -> StateBackend:
if settings.REDIS_HOST:
return RedisState(host=settings.REDIS_HOST, port=settings.REDIS_PORT)
return MemoryState()
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
"""Start the flow engine alongside the API."""
event_bus.bind(asyncio.get_running_loop())
init_secrets(settings.SECRETS_FILE, settings.SECRET_KEY)
controller = FlowController(
store=FlowStore(settings.FLOWS_DIR),
state=_state_backend(),
events=event_bus,
max_workers=settings.FLOW_MAX_WORKERS,
fastapi_app=app,
)
app.state.flow_controller = controller
await controller.start()
try:
yield
finally:
await controller.stop()
app = FastAPI(
title=settings.PROJECT_NAME,
openapi_url=f"{settings.API_V1_STR}/openapi.json",
generate_unique_id_function=custom_generate_unique_id,
lifespan=lifespan,
)
# Set all CORS enabled origins