Add the connector contract, reusable nodes and per-port intervals
Connectors are the device-facing node class third parties write, so the surface they build against is versioned and documented: ConnectorNode carries a declared contract version, a polling loop that publishes only what changed and reports health around it, and parameters whose credential fields are marked x-secret so the editor offers the secrets store instead of a text box. They are found through the fluksio.node_types entry point group, with the package's own metadata as the manifest. docs/connectors/ has the contract and the authoring guide; connector-skeleton/ is a working one to copy. The controller no longer knows what any node type is: start, stop and report_health are protocol methods on Node, and the built-ins were migrated to them first, so the hooks a connector implements are the ones the engine has been driving all along. Marking a node reusable moves its source to _lib/ and points the node at it by name. Other flows instantiate it with their own ports and settings, one fix reaches all of them, and a shared source still in use cannot be deleted. Ports gained an interval: an output publishes, and an input wakes its node, at most every n seconds. State keeps the latest value, so only the delivery is skipped, and pressing Run is never throttled. Also fixes autosave sending no version on its first save of a session, which made every flow saved more than once conflict with itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
7344eac262
commit
3724b68f23
+8
-3
@@ -27,9 +27,14 @@ Deferring because out of scope is fine, but don't mention deferring than.
|
|||||||
- FEAT/UI: publishing and discarding are only reachable while no side panel is open, since the
|
- FEAT/UI: publishing and discarding are only reachable while no side panel is open, since the
|
||||||
floating chrome hides for the panel. Editing a node's code and publishing it is therefore
|
floating chrome hides for the panel. Editing a node's code and publishing it is therefore
|
||||||
close-panel-then-publish.
|
close-panel-then-publish.
|
||||||
- FEAT/UI: no way to point a node parameter at a stored secret. The store and the
|
- FEAT/UI: there is no screen for managing the secrets store itself. A node parameter marked
|
||||||
`{"$secret": "name"}` reference both work, but the params form only produces literals, so a
|
`x-secret` offers the stored secrets, but they can only be created through the API.
|
||||||
webhook secret typed there lands in `flow.json`.
|
- FEAT/FLOW: input discretization drops the trailing edge — if a producer goes quiet inside
|
||||||
|
the window, the consumer never sees the last value. Delivering it at the end of the window
|
||||||
|
needs a timer per (node, port).
|
||||||
|
- CHORE/FLOW: shared node sources bypass the draft/publish split. Editing one writes the
|
||||||
|
library copy and reloads immediately, since the code is not any single flow's to hold back.
|
||||||
|
Deliberate, but it means a shared node is the one thing publish does not gate.
|
||||||
- FEAT/UI: flow-level edits (title, inputs) go through `FlowPanel.onChange` and stay outside
|
- FEAT/UI: flow-level edits (title, inputs) go through `FlowPanel.onChange` and stay outside
|
||||||
the undo stack.
|
the undo stack.
|
||||||
- CHORE/UI: `make test-backend` cannot reach Postgres while the integrated stack is up —
|
- CHORE/UI: `make test-backend` cannot reach Postgres while the integrated stack is up —
|
||||||
|
|||||||
+15
-6
@@ -42,11 +42,15 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend – M
|
|||||||
- [x] Secrets/credentials store for node integrations managed via the API/UI
|
- [x] Secrets/credentials store for node integrations managed via the API/UI
|
||||||
(encrypted at rest, referenced from node params as `{"$secret": "name"}`);
|
(encrypted at rest, referenced from node params as `{"$secret": "name"}`);
|
||||||
`.env` bootstrap-only
|
`.env` bootstrap-only
|
||||||
- [ ] Connector node contract: declared I/O schemas, UI-driven credential and
|
- [x] Connector node contract: `ConnectorNode` with a declared contract version,
|
||||||
setup flow, polling/discretization behaviour, health reporting, packaging
|
a polling coordinator that deduplicates, `x-secret` parameters the editor
|
||||||
metadata — plus an authoring guide and skeleton repo. Device-specific
|
renders as a secret picker, and health reporting. Connectors are installed
|
||||||
nodes are written against it as external plugins; the registry follows
|
packages found through the `fluksio.node_types` entry point group; the
|
||||||
later
|
contract is documented in `docs/connectors/` with a working skeleton at
|
||||||
|
`connector-skeleton/`. The registry follows later
|
||||||
|
- [x] Node lifecycle as a protocol (`start`/`stop`/`report_health` on `Node`),
|
||||||
|
replacing the controller's per-type isinstance chains — the same hooks a
|
||||||
|
connector implements, validated on the built-in nodes first
|
||||||
- [x] Flow persistence: `flow.json` plus node sources per flow, replacing the
|
- [x] Flow persistence: `flow.json` plus node sources per flow, replacing the
|
||||||
watch-directory prototype
|
watch-directory prototype
|
||||||
- [x] REST + WebSocket API over the engine: create/read/update flows, edit node
|
- [x] REST + WebSocket API over the engine: create/read/update flows, edit node
|
||||||
@@ -64,7 +68,9 @@ external interfaces. See `docs/architecture/structure.canvas` → *Backend – M
|
|||||||
draft. Saves carry the version they were based on, so a second client
|
draft. Saves carry the version they were based on, so a second client
|
||||||
editing the same flow is refused rather than overwritten
|
editing the same flow is refused rather than overwritten
|
||||||
- [ ] Import/export of a flow as human-readable code plus a JSON structure
|
- [ ] Import/export of a flow as human-readable code plus a JSON structure
|
||||||
- [ ] Per-input/-output discretization interval setting
|
- [x] Per-input/-output discretization interval setting: a port publishes, or
|
||||||
|
wakes its node, at most every n seconds. State keeps the latest value, so
|
||||||
|
only the delivery is skipped
|
||||||
- [ ] Alert / notification handler
|
- [ ] Alert / notification handler
|
||||||
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
|
- [ ] Test nodes: a small node dragged onto an existing one, smoke or unit, blocking
|
||||||
deployment on failure
|
deployment on failure
|
||||||
@@ -99,6 +105,9 @@ React + Vite, primarily desktop but usable on mobile. See `docs/architecture/str
|
|||||||
- [x] Validation shown on the node it belongs to, and summarised in the dock
|
- [x] Validation shown on the node it belongs to, and summarised in the dock
|
||||||
- [x] Publish control and draft markers in the flow bar, discard in the flow
|
- [x] Publish control and draft markers in the flow bar, discard in the flow
|
||||||
panel, and a conflict dialog when another client got there first
|
panel, and a conflict dialog when another client got there first
|
||||||
|
- [x] Marking a node reusable, and placing a shared one from the palette
|
||||||
|
- [x] Secret picker for credential parameters, so a password never lands in
|
||||||
|
`flow.json`
|
||||||
- [x] Dashboard showing which flows run, which are stopped and which have
|
- [x] Dashboard showing which flows run, which are stopped and which have
|
||||||
errors, with a switch per flow
|
errors, with a switch per flow
|
||||||
- [x] Logs panel in the canvas dock, pause/resume beside Run, and replaying an
|
- [x] Logs panel in the canvas dock, pause/resume beside Run, and replaying an
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ from app.flow.schemas import (
|
|||||||
FlowStatePublic,
|
FlowStatePublic,
|
||||||
FlowSummary,
|
FlowSummary,
|
||||||
HistoryPoint,
|
HistoryPoint,
|
||||||
|
LibraryNode,
|
||||||
MessageHistory,
|
MessageHistory,
|
||||||
MessageValue,
|
MessageValue,
|
||||||
NodeSource,
|
NodeSource,
|
||||||
@@ -34,7 +35,13 @@ from app.flow.schemas import (
|
|||||||
NodeTypeInfo,
|
NodeTypeInfo,
|
||||||
)
|
)
|
||||||
from app.flow.state import as_number
|
from app.flow.state import as_number
|
||||||
from app.flow.store import FlowExists, FlowNotFound, StaleVersion
|
from app.flow.store import (
|
||||||
|
FlowExists,
|
||||||
|
FlowNotFound,
|
||||||
|
LibExists,
|
||||||
|
LibNotFound,
|
||||||
|
StaleVersion,
|
||||||
|
)
|
||||||
from app.models import Message
|
from app.models import Message
|
||||||
|
|
||||||
router = APIRouter(
|
router = APIRouter(
|
||||||
@@ -73,6 +80,10 @@ class PublishRequest(BaseModel):
|
|||||||
version: int
|
version: int
|
||||||
|
|
||||||
|
|
||||||
|
class ShareRequest(BaseModel):
|
||||||
|
lib_name: str
|
||||||
|
|
||||||
|
|
||||||
class RunRequest(BaseModel):
|
class RunRequest(BaseModel):
|
||||||
inputs: dict[str, Any] = {}
|
inputs: dict[str, Any] = {}
|
||||||
|
|
||||||
@@ -114,6 +125,12 @@ def _read_flow(controller: FlowController, name: str) -> FlowDef:
|
|||||||
raise HTTPException(status_code=404, detail=f"No flow named '{name}'")
|
raise HTTPException(status_code=404, detail=f"No flow named '{name}'")
|
||||||
|
|
||||||
|
|
||||||
|
def _source_ref(definition: FlowDef, node_id: str) -> str | None:
|
||||||
|
"""The library source this node runs, if it is a shared one."""
|
||||||
|
node = next((n for n in definition.nodes if n.id == node_id), None)
|
||||||
|
return node.source_ref if node else None
|
||||||
|
|
||||||
|
|
||||||
def _require_enabled(controller: FlowController, name: str) -> None:
|
def _require_enabled(controller: FlowController, name: str) -> None:
|
||||||
if not controller.is_enabled(name):
|
if not controller.is_enabled(name):
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
@@ -168,6 +185,39 @@ def read_node_types() -> Any:
|
|||||||
return node_type_info()
|
return node_type_info()
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Shared nodes
|
||||||
|
#
|
||||||
|
# Declared above the "/{name}" routes: "library" would otherwise be read as a
|
||||||
|
# flow name.
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/library", response_model=list[LibraryNode])
|
||||||
|
def read_library(controller: FlowControllerDep) -> Any:
|
||||||
|
"""The node sources shared across flows, and which nodes use each."""
|
||||||
|
return [
|
||||||
|
LibraryNode(name=name, used_by=controller.store.usages(name))
|
||||||
|
for name in controller.store.list_lib()
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/library/{lib_name}", response_model=Message)
|
||||||
|
async def delete_shared_node(lib_name: str, controller: FlowControllerDep) -> Any:
|
||||||
|
"""Remove a shared source, as long as no flow still runs it."""
|
||||||
|
used_by = controller.store.usages(lib_name)
|
||||||
|
if used_by:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=409,
|
||||||
|
detail=f"'{lib_name}' is still used by {', '.join(used_by)}",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
await run_in_threadpool(controller.store.delete_lib_source, lib_name)
|
||||||
|
except LibNotFound:
|
||||||
|
raise HTTPException(status_code=404, detail=f"No shared node '{lib_name}'")
|
||||||
|
return Message(message=f"Deleted shared node '{lib_name}'")
|
||||||
|
|
||||||
|
|
||||||
@router.get("/{name}", response_model=FlowDetail)
|
@router.get("/{name}", response_model=FlowDetail)
|
||||||
def read_flow(name: str, controller: FlowControllerDep) -> Any:
|
def read_flow(name: str, controller: FlowControllerDep) -> Any:
|
||||||
"""Read one flow, with the state of its nodes."""
|
"""Read one flow, with the state of its nodes."""
|
||||||
@@ -303,7 +353,13 @@ def read_node_source(
|
|||||||
controller: FlowControllerDep,
|
controller: FlowControllerDep,
|
||||||
) -> Any:
|
) -> Any:
|
||||||
"""Read a node's Python source, including unpublished edits."""
|
"""Read a node's Python source, including unpublished edits."""
|
||||||
_read_flow(controller, name)
|
definition = _read_flow(controller, name)
|
||||||
|
ref = _source_ref(definition, node_id)
|
||||||
|
if ref:
|
||||||
|
try:
|
||||||
|
return NodeSource(code=controller.store.read_lib_source(ref))
|
||||||
|
except LibNotFound:
|
||||||
|
raise HTTPException(status_code=404, detail=f"No shared node '{ref}'")
|
||||||
return NodeSource(code=controller.store.read_node_source(name, node_id, draft=True))
|
return NodeSource(code=controller.store.read_node_source(name, node_id, draft=True))
|
||||||
|
|
||||||
|
|
||||||
@@ -319,8 +375,20 @@ async def save_node_source(
|
|||||||
The answer comes from compiling the code rather than from the running
|
The answer comes from compiling the code rather than from the running
|
||||||
pipeline: a draft is not deployed, and compiling is both faster and more
|
pipeline: a draft is not deployed, and compiling is both faster and more
|
||||||
precise about what the author just typed.
|
precise about what the author just typed.
|
||||||
|
|
||||||
|
A shared node writes to the library, so the fix reaches every flow using
|
||||||
|
it — and that one is live immediately rather than waiting for a publish,
|
||||||
|
because the copy is not any single flow's to hold back.
|
||||||
"""
|
"""
|
||||||
_read_flow(controller, name)
|
definition = _read_flow(controller, name)
|
||||||
|
ref = _source_ref(definition, node_id)
|
||||||
|
if ref:
|
||||||
|
changed = await run_in_threadpool(
|
||||||
|
controller.store.write_lib_source, ref, source.code
|
||||||
|
)
|
||||||
|
if changed:
|
||||||
|
await controller.reload()
|
||||||
|
else:
|
||||||
await run_in_threadpool(
|
await run_in_threadpool(
|
||||||
controller.store.write_node_source, name, node_id, source.code, True
|
controller.store.write_node_source, name, node_id, source.code, True
|
||||||
)
|
)
|
||||||
@@ -334,6 +402,46 @@ async def save_node_source(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{name}/nodes/{node_id}/share", response_model=FlowDetail)
|
||||||
|
async def share_node(
|
||||||
|
name: str,
|
||||||
|
node_id: str,
|
||||||
|
body: ShareRequest,
|
||||||
|
controller: FlowControllerDep,
|
||||||
|
) -> Any:
|
||||||
|
"""Move this node's code into the library so other flows can run it too."""
|
||||||
|
_read_flow(controller, name)
|
||||||
|
if not NAME_PATTERN.match(body.lib_name):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=(
|
||||||
|
"Use lowercase letters, digits and underscores, starting with a letter"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
await run_in_threadpool(
|
||||||
|
controller.store.share_node, name, node_id, body.lib_name
|
||||||
|
)
|
||||||
|
except LibExists as exc:
|
||||||
|
raise HTTPException(status_code=409, detail=str(exc))
|
||||||
|
return _detail(controller, _read_flow(controller, name))
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{name}/nodes/{node_id}/unshare", response_model=FlowDetail)
|
||||||
|
async def unshare_node(
|
||||||
|
name: str,
|
||||||
|
node_id: str,
|
||||||
|
controller: FlowControllerDep,
|
||||||
|
) -> Any:
|
||||||
|
"""Take a private copy of the shared code back into this flow."""
|
||||||
|
_read_flow(controller, name)
|
||||||
|
try:
|
||||||
|
await run_in_threadpool(controller.store.unshare_node, name, node_id)
|
||||||
|
except LibNotFound as exc:
|
||||||
|
raise HTTPException(status_code=404, detail=str(exc))
|
||||||
|
return _detail(controller, _read_flow(controller, name))
|
||||||
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------
|
||||||
# Running, stopped, paused
|
# Running, stopped, paused
|
||||||
# -----------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -0,0 +1,154 @@
|
|||||||
|
"""The contract a connector node is written against.
|
||||||
|
|
||||||
|
A connector is the device-facing node class: it talks to something outside the
|
||||||
|
engine — a device, a service, a protocol — and publishes what it finds as
|
||||||
|
ordinary typed messages. Third parties write these, so this surface is the one
|
||||||
|
part of the engine that has to stay stable; it is versioned by
|
||||||
|
:data:`CONTRACT_VERSION` and a connector declares which version it was written
|
||||||
|
for.
|
||||||
|
|
||||||
|
What a connector gets from the base class:
|
||||||
|
|
||||||
|
* a polling loop that runs :meth:`ConnectorNode.poll` on a schedule, publishes
|
||||||
|
only the ports whose value changed, and reports health around it;
|
||||||
|
* :meth:`Node.report_health`, so a connection problem shows on the node rather
|
||||||
|
than only in the log;
|
||||||
|
* the lifecycle hooks the controller drives, so nothing device-specific has to
|
||||||
|
be known by the engine.
|
||||||
|
|
||||||
|
The message schemas and the parameter model are the rest of the contract, and
|
||||||
|
they are the same ones the built-in nodes use. See ``docs/connectors/`` for the
|
||||||
|
authoring guide.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
from typing import TYPE_CHECKING, Any, ClassVar
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from app.flow.nodes import Node
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from fastapi import FastAPI
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
#: Bumped when a change would break connectors written against the old surface.
|
||||||
|
#: The loader refuses a connector declaring anything else.
|
||||||
|
CONTRACT_VERSION = 1
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorNode(Node):
|
||||||
|
"""Base class for device- and service-facing nodes.
|
||||||
|
|
||||||
|
A subclass declares the contract version it was written for, describes
|
||||||
|
itself for the editor, and implements :meth:`poll`, :meth:`start`, or both::
|
||||||
|
|
||||||
|
class RandomSensor(ConnectorNode):
|
||||||
|
contract = CONTRACT_VERSION
|
||||||
|
title = "Random sensor"
|
||||||
|
description = "Emits a random reading, for trying the contract out."
|
||||||
|
|
||||||
|
class Params(ConnectorNode.Params):
|
||||||
|
ceiling: float = 1.0
|
||||||
|
|
||||||
|
async def poll(self):
|
||||||
|
return {"reading": random.random() * self.config.ceiling}
|
||||||
|
"""
|
||||||
|
|
||||||
|
#: Declared explicitly by every connector; inheriting it does not count.
|
||||||
|
contract: ClassVar[int]
|
||||||
|
title: ClassVar[str] = ""
|
||||||
|
description: ClassVar[str] = ""
|
||||||
|
|
||||||
|
class Params(BaseModel):
|
||||||
|
"""Settings the editor renders a form for.
|
||||||
|
|
||||||
|
Subclass it to add your own. A field holding a credential should carry
|
||||||
|
``json_schema_extra={"x-secret": True}``, which makes the editor offer
|
||||||
|
the stored secrets instead of a text box.
|
||||||
|
"""
|
||||||
|
|
||||||
|
poll_interval: float = Field(
|
||||||
|
default=0,
|
||||||
|
ge=0,
|
||||||
|
description="Seconds between polls; 0 polls never.",
|
||||||
|
)
|
||||||
|
|
||||||
|
__slots__ = ("config", "_poll_task", "_stop_event", "_last_published")
|
||||||
|
|
||||||
|
def __init__(self, **kwargs: Any) -> None:
|
||||||
|
super().__init__(f=self._unused, **kwargs)
|
||||||
|
self.config = type(self).Params(**self.params)
|
||||||
|
self._poll_task: asyncio.Task[None] | None = None
|
||||||
|
self._stop_event: asyncio.Event | None = None
|
||||||
|
self._last_published: dict[str, Any] = {}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _unused(**_: Any) -> None:
|
||||||
|
"""A connector publishes from its own loop, not from the scheduler."""
|
||||||
|
return None
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# What a connector implements
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
async def poll(self) -> dict[str, Any] | None:
|
||||||
|
"""Read the device once and return values keyed by output port.
|
||||||
|
|
||||||
|
Return ``None`` when there is nothing new. Raising is reported as a
|
||||||
|
health problem and retried on the next tick.
|
||||||
|
"""
|
||||||
|
return None
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# What the engine drives
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
async def start(self, app: FastAPI | None = None) -> None:
|
||||||
|
if self.config.poll_interval > 0 and self._poll_task is None:
|
||||||
|
self._stop_event = asyncio.Event()
|
||||||
|
self._poll_task = asyncio.create_task(self._poll_loop())
|
||||||
|
|
||||||
|
async def stop(self, app: FastAPI | None = None) -> None:
|
||||||
|
if self._poll_task is None:
|
||||||
|
return
|
||||||
|
if self._stop_event is not None:
|
||||||
|
self._stop_event.set()
|
||||||
|
self._poll_task.cancel()
|
||||||
|
try:
|
||||||
|
await self._poll_task
|
||||||
|
except (asyncio.CancelledError, Exception): # noqa: B014 - shutting down
|
||||||
|
pass
|
||||||
|
self._poll_task = None
|
||||||
|
self._last_published = {}
|
||||||
|
|
||||||
|
async def _poll_loop(self) -> None:
|
||||||
|
"""Poll, publish what changed, and say how the connection is doing.
|
||||||
|
|
||||||
|
Only changed ports are published: a device polled every few seconds is
|
||||||
|
usually saying the same thing, and every publication wakes everything
|
||||||
|
downstream of it.
|
||||||
|
"""
|
||||||
|
while not (self._stop_event and self._stop_event.is_set()):
|
||||||
|
try:
|
||||||
|
values = await self.poll()
|
||||||
|
self.report_health("ok")
|
||||||
|
changed = {
|
||||||
|
port: value
|
||||||
|
for port, value in (values or {}).items()
|
||||||
|
if self._last_published.get(port, object()) != value
|
||||||
|
}
|
||||||
|
if changed:
|
||||||
|
self._last_published.update(changed)
|
||||||
|
# inject runs the graph, which is blocking work.
|
||||||
|
await asyncio.to_thread(self.inject, changed)
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
break
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("Connector '%s' failed to poll: %s", self.id, exc)
|
||||||
|
self.report_health("down", f"{type(exc).__name__}: {exc}")
|
||||||
|
await asyncio.sleep(self.config.poll_interval)
|
||||||
@@ -33,10 +33,16 @@ from app.flow.nodes import (
|
|||||||
Node,
|
Node,
|
||||||
)
|
)
|
||||||
from app.flow.pipeline import Pipeline, ValidationIssue
|
from app.flow.pipeline import Pipeline, ValidationIssue
|
||||||
from app.flow.schemas import FlowDef, NodeDef, NodeStatusPublic, NodeTypeInfo
|
from app.flow.schemas import (
|
||||||
|
FlowDef,
|
||||||
|
Health,
|
||||||
|
NodeDef,
|
||||||
|
NodeStatusPublic,
|
||||||
|
NodeTypeInfo,
|
||||||
|
)
|
||||||
from app.flow.secrets import SecretNotFound, resolve_params
|
from app.flow.secrets import SecretNotFound, resolve_params
|
||||||
from app.flow.state import MemoryState, StateBackend
|
from app.flow.state import MemoryState, StateBackend
|
||||||
from app.flow.store import FlowNotFound, FlowStore
|
from app.flow.store import LIB_DIR, FlowNotFound, FlowStore, LibNotFound
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -50,13 +56,20 @@ class NodeStatus(str, Enum):
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class LoadedNode:
|
class LoadedNode:
|
||||||
"""A node as the editor sees it: built, or failed with a reason."""
|
"""A node as the editor sees it: built, or failed with a reason.
|
||||||
|
|
||||||
|
``status`` says whether it loaded; ``health`` says how its connection is
|
||||||
|
doing once it is running. A node can be perfectly loaded and unable to
|
||||||
|
reach its broker.
|
||||||
|
"""
|
||||||
|
|
||||||
id: str
|
id: str
|
||||||
flow: str
|
flow: str
|
||||||
status: NodeStatus = NodeStatus.ACTIVE
|
status: NodeStatus = NodeStatus.ACTIVE
|
||||||
node: Node | None = None
|
node: Node | None = None
|
||||||
error: str | None = None
|
error: str | None = None
|
||||||
|
health: Health = "ok"
|
||||||
|
health_detail: str | None = None
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -77,6 +90,9 @@ class NodeType:
|
|||||||
cls: Any
|
cls: Any
|
||||||
has_source: bool = False
|
has_source: bool = False
|
||||||
params_schema: dict[str, Any] = field(default_factory=dict)
|
params_schema: dict[str, Any] = field(default_factory=dict)
|
||||||
|
#: Which installed package supplied this type, for the ones that are not
|
||||||
|
#: built in.
|
||||||
|
plugin: str | None = None
|
||||||
|
|
||||||
|
|
||||||
def _schema_of(cls: Any) -> dict[str, Any]:
|
def _schema_of(cls: Any) -> dict[str, Any]:
|
||||||
@@ -132,6 +148,7 @@ def node_type_info() -> list[NodeTypeInfo]:
|
|||||||
description=spec.description,
|
description=spec.description,
|
||||||
params_schema=spec.params_schema,
|
params_schema=spec.params_schema,
|
||||||
has_source=spec.has_source,
|
has_source=spec.has_source,
|
||||||
|
plugin=spec.plugin,
|
||||||
)
|
)
|
||||||
for key, spec in NODE_TYPES.items()
|
for key, spec in NODE_TYPES.items()
|
||||||
]
|
]
|
||||||
@@ -252,21 +269,16 @@ class FlowController:
|
|||||||
if node is None:
|
if node is None:
|
||||||
continue
|
continue
|
||||||
try:
|
try:
|
||||||
if isinstance(node, MqttNode) and node.is_subscribed:
|
await node.stop(self.app)
|
||||||
await node.stop_subscription()
|
|
||||||
elif isinstance(node, DelayNode) and node.cron_expr:
|
|
||||||
await node.stop_cron()
|
|
||||||
if (
|
|
||||||
isinstance(node, HttpNode)
|
|
||||||
and node.mode == HttpNode.Mode.TRIGGER
|
|
||||||
and self.app is not None
|
|
||||||
):
|
|
||||||
node.unregister_route(self.app)
|
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.exception("Error stopping node '%s'", entry.id)
|
logger.exception("Error stopping node '%s'", entry.id)
|
||||||
|
|
||||||
async def _activate(self) -> None:
|
async def _activate(self) -> None:
|
||||||
"""Start subscriptions, schedules and webhooks of the new pipeline."""
|
"""Start subscriptions, schedules and webhooks of the new pipeline.
|
||||||
|
|
||||||
|
What that means per node type is the node's own business — a connector
|
||||||
|
written against the contract starts here the same way the built-ins do.
|
||||||
|
"""
|
||||||
for entry in self.loaded.values():
|
for entry in self.loaded.values():
|
||||||
node = entry.node
|
node = entry.node
|
||||||
if node is None:
|
if node is None:
|
||||||
@@ -276,16 +288,7 @@ class FlowController:
|
|||||||
if entry.flow in self.disabled:
|
if entry.flow in self.disabled:
|
||||||
continue
|
continue
|
||||||
try:
|
try:
|
||||||
if isinstance(node, MqttNode) and node.mode == MqttNode.Mode.SUBSCRIBER:
|
await node.start(self.app)
|
||||||
await node.start_subscription()
|
|
||||||
elif isinstance(node, DelayNode) and node.cron_expr:
|
|
||||||
await node.start_cron()
|
|
||||||
elif (
|
|
||||||
isinstance(node, HttpNode)
|
|
||||||
and node.mode == HttpNode.Mode.TRIGGER
|
|
||||||
and self.app is not None
|
|
||||||
):
|
|
||||||
node.register_route(self.app)
|
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
logger.exception("Error starting node '%s'", entry.id)
|
logger.exception("Error starting node '%s'", entry.id)
|
||||||
entry.status = NodeStatus.ERROR
|
entry.status = NodeStatus.ERROR
|
||||||
@@ -334,9 +337,16 @@ class FlowController:
|
|||||||
params = resolve_params(node_def.params)
|
params = resolve_params(node_def.params)
|
||||||
|
|
||||||
if node_type.has_source:
|
if node_type.has_source:
|
||||||
|
# A shared node runs the library's copy, compiled once under
|
||||||
|
# the library's own name so every flow using it agrees.
|
||||||
|
if node_def.source_ref:
|
||||||
|
code = self.store.read_lib_source(node_def.source_ref)
|
||||||
|
function = _load_function(LIB_DIR, node_def.source_ref, code)
|
||||||
|
else:
|
||||||
code = self.store.read_node_source(flow, node_def.id, draft=draft)
|
code = self.store.read_node_source(flow, node_def.id, draft=draft)
|
||||||
|
function = _load_function(flow, node_def.id, code)
|
||||||
node = Node(
|
node = Node(
|
||||||
f=_load_function(flow, node_def.id, code),
|
f=function,
|
||||||
requires=_bound(node_def.requires),
|
requires=_bound(node_def.requires),
|
||||||
provides=_bound(node_def.provides),
|
provides=_bound(node_def.provides),
|
||||||
params=params,
|
params=params,
|
||||||
@@ -351,13 +361,14 @@ class FlowController:
|
|||||||
)
|
)
|
||||||
|
|
||||||
node.assign_flow(flow, node_def.id)
|
node.assign_flow(flow, node_def.id)
|
||||||
|
node._on_health = self._health_changed
|
||||||
|
|
||||||
if isinstance(node, HttpNode) and node.mode == HttpNode.Mode.TRIGGER:
|
if isinstance(node, HttpNode) and node.mode == HttpNode.Mode.TRIGGER:
|
||||||
# Webhooks live under their flow, away from the JSON API.
|
# Webhooks live under their flow, away from the JSON API.
|
||||||
node.url = f"{HOOK_PREFIX}/{flow}/{node.url.lstrip('/')}"
|
node.url = f"{HOOK_PREFIX}/{flow}/{node.url.lstrip('/')}"
|
||||||
|
|
||||||
entry.node = node
|
entry.node = node
|
||||||
except SecretNotFound as exc:
|
except (SecretNotFound, LibNotFound) as exc:
|
||||||
entry.status = NodeStatus.ERROR
|
entry.status = NodeStatus.ERROR
|
||||||
entry.error = str(exc)
|
entry.error = str(exc)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
@@ -372,11 +383,34 @@ class FlowController:
|
|||||||
|
|
||||||
def node_statuses(self, flow: str | None = None) -> list[NodeStatusPublic]:
|
def node_statuses(self, flow: str | None = None) -> list[NodeStatusPublic]:
|
||||||
return [
|
return [
|
||||||
NodeStatusPublic(id=entry.id, status=entry.status.value, error=entry.error)
|
NodeStatusPublic(
|
||||||
|
id=entry.id,
|
||||||
|
status=entry.status.value,
|
||||||
|
error=entry.error,
|
||||||
|
health=entry.health,
|
||||||
|
health_detail=entry.health_detail,
|
||||||
|
)
|
||||||
for entry in self.loaded.values()
|
for entry in self.loaded.values()
|
||||||
if flow is None or entry.flow == flow
|
if flow is None or entry.flow == flow
|
||||||
]
|
]
|
||||||
|
|
||||||
|
def _health_changed(self, node: Node, status: str, detail: str | None) -> None:
|
||||||
|
"""A node saying how its connection is doing, from any thread."""
|
||||||
|
entry = self.loaded.get(node.id)
|
||||||
|
if entry is None or (entry.health == status and entry.health_detail == detail):
|
||||||
|
return
|
||||||
|
entry.health = cast(Health, status)
|
||||||
|
entry.health_detail = detail
|
||||||
|
self._publish(
|
||||||
|
{
|
||||||
|
"type": "node_health",
|
||||||
|
"flow": node.flow,
|
||||||
|
"node": node.id,
|
||||||
|
"health": status,
|
||||||
|
"detail": detail,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
def flow_issues(self, flow: str) -> list[ValidationIssue]:
|
def flow_issues(self, flow: str) -> list[ValidationIssue]:
|
||||||
return [issue for issue in self.issues if not issue.flow or issue.flow == flow]
|
return [issue for issue in self.issues if not issue.flow or issue.flow == flow]
|
||||||
|
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ import json
|
|||||||
from enum import Enum
|
from enum import Enum
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from pydantic import BaseModel, ConfigDict, model_validator
|
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||||
|
|
||||||
|
|
||||||
class DType(str, Enum):
|
class DType(str, Enum):
|
||||||
@@ -42,6 +42,10 @@ class MessageSpec(BaseModel):
|
|||||||
:param port: The identifier the node function sees. Defaults to the last
|
:param port: The identifier the node function sees. Defaults to the last
|
||||||
segment of ``name``, so unqualified flows read naturally.
|
segment of ``name``, so unqualified flows read naturally.
|
||||||
:param dtype: Payload type, validated on every message that passes through.
|
:param dtype: Payload type, validated on every message that passes through.
|
||||||
|
:param interval: Deliver at most every this many seconds; 0 is every time.
|
||||||
|
On an output it holds back publishing, on an input it holds back waking
|
||||||
|
the node. The value is never lost — state keeps the latest — only the
|
||||||
|
delivery is skipped.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
model_config = ConfigDict(frozen=True)
|
model_config = ConfigDict(frozen=True)
|
||||||
@@ -49,6 +53,7 @@ class MessageSpec(BaseModel):
|
|||||||
name: str = ""
|
name: str = ""
|
||||||
port: str = ""
|
port: str = ""
|
||||||
dtype: DType = DType.FLOAT
|
dtype: DType = DType.FLOAT
|
||||||
|
interval: float = Field(default=0, ge=0)
|
||||||
|
|
||||||
@model_validator(mode="after")
|
@model_validator(mode="after")
|
||||||
def _default_port(self) -> MessageSpec:
|
def _default_port(self) -> MessageSpec:
|
||||||
|
|||||||
@@ -74,6 +74,7 @@ class Node:
|
|||||||
"params",
|
"params",
|
||||||
"_pipeline",
|
"_pipeline",
|
||||||
"synchronous",
|
"synchronous",
|
||||||
|
"_on_health",
|
||||||
)
|
)
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@@ -86,6 +87,7 @@ class Node:
|
|||||||
):
|
):
|
||||||
self.f = f
|
self.f = f
|
||||||
self._pipeline: Pipeline | None = None
|
self._pipeline: Pipeline | None = None
|
||||||
|
self._on_health: Callable[[Node, str, str | None], None] | None = None
|
||||||
self.params = dict(params) if params else {}
|
self.params = dict(params) if params else {}
|
||||||
self.synchronous = bool(self.params.get("synchronous", False))
|
self.synchronous = bool(self.params.get("synchronous", False))
|
||||||
|
|
||||||
@@ -138,6 +140,26 @@ class Node:
|
|||||||
"""The node's id within its flow."""
|
"""The node's id within its flow."""
|
||||||
return self.name
|
return self.name
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Lifecycle
|
||||||
|
#
|
||||||
|
# A plain logic node has nothing to start or stop. The ones that talk to the
|
||||||
|
# outside — subscriptions, schedules, webhooks — override these, which is
|
||||||
|
# how the controller can bring a flow up or down without knowing what any
|
||||||
|
# particular node type is.
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
async def start(self, app: FastAPI | None = None) -> None:
|
||||||
|
"""Begin whatever this node listens to. Called when its flow starts."""
|
||||||
|
|
||||||
|
async def stop(self, app: FastAPI | None = None) -> None:
|
||||||
|
"""Undo :meth:`start`. Called before a rebuild, and must be idempotent."""
|
||||||
|
|
||||||
|
def report_health(self, status: str, detail: str | None = None) -> None:
|
||||||
|
"""Say how this node's connection is doing: ok, degraded or down."""
|
||||||
|
if self._on_health is not None:
|
||||||
|
self._on_health(self, status, detail)
|
||||||
|
|
||||||
def bind(self, pipeline: Pipeline) -> None:
|
def bind(self, pipeline: Pipeline) -> None:
|
||||||
"""
|
"""
|
||||||
Bind this node to a pipeline for external triggering.
|
Bind this node to a pipeline for external triggering.
|
||||||
@@ -428,8 +450,9 @@ class HttpNode(Node):
|
|||||||
description=(
|
description=(
|
||||||
"Shared secret for a webhook, appended to its URL: "
|
"Shared secret for a webhook, appended to its URL: "
|
||||||
"/hooks/<flow>/<url>/<secret>. Empty leaves the webhook open "
|
"/hooks/<flow>/<url>/<secret>. Empty leaves the webhook open "
|
||||||
'to anyone. Use {"$secret": "name"} to read it from the store.'
|
"to anyone."
|
||||||
),
|
),
|
||||||
|
json_schema_extra={"x-secret": True},
|
||||||
)
|
)
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@@ -568,6 +591,15 @@ class HttpNode(Node):
|
|||||||
logger.error("Request error in node '%s': %s", self.name, e)
|
logger.error("Request error in node '%s': %s", self.name, e)
|
||||||
raise
|
raise
|
||||||
|
|
||||||
|
async def start(self, app: FastAPI | None = None) -> None:
|
||||||
|
"""A webhook needs a route; a sender reaches out on its own."""
|
||||||
|
if self.mode is HttpNode.Mode.TRIGGER and app is not None:
|
||||||
|
self.register_route(app)
|
||||||
|
|
||||||
|
async def stop(self, app: FastAPI | None = None) -> None:
|
||||||
|
if self.mode is HttpNode.Mode.TRIGGER and app is not None:
|
||||||
|
self.unregister_route(app)
|
||||||
|
|
||||||
def register_route(self, app: FastAPI) -> None:
|
def register_route(self, app: FastAPI) -> None:
|
||||||
"""
|
"""
|
||||||
Register this node's HTTP endpoint with a FastAPI application.
|
Register this node's HTTP endpoint with a FastAPI application.
|
||||||
@@ -822,7 +854,9 @@ class MqttNode(Node):
|
|||||||
broker_host: str = "localhost"
|
broker_host: str = "localhost"
|
||||||
broker_port: int = 1883
|
broker_port: int = 1883
|
||||||
username: str | None = None
|
username: str | None = None
|
||||||
password: str | None = None
|
password: str | None = Field(
|
||||||
|
default=None, json_schema_extra={"x-secret": True}
|
||||||
|
)
|
||||||
client_id: str | None = None
|
client_id: str | None = None
|
||||||
qos: int = 0
|
qos: int = 0
|
||||||
retain: bool = False
|
retain: bool = False
|
||||||
@@ -1005,6 +1039,14 @@ class MqttNode(Node):
|
|||||||
logger.error("MQTT publish error in node '%s': %s", self.name, e)
|
logger.error("MQTT publish error in node '%s': %s", self.name, e)
|
||||||
raise
|
raise
|
||||||
|
|
||||||
|
async def start(self, app: FastAPI | None = None) -> None:
|
||||||
|
"""A subscriber listens; a publisher has nothing to start."""
|
||||||
|
if self.mode is MqttNode.Mode.SUBSCRIBER:
|
||||||
|
await self.start_subscription()
|
||||||
|
|
||||||
|
async def stop(self, app: FastAPI | None = None) -> None:
|
||||||
|
await self.stop_subscription()
|
||||||
|
|
||||||
async def start_subscription(self) -> None:
|
async def start_subscription(self) -> None:
|
||||||
"""
|
"""
|
||||||
Start the MQTT subscription for trigger mode nodes.
|
Start the MQTT subscription for trigger mode nodes.
|
||||||
@@ -1087,6 +1129,7 @@ class MqttNode(Node):
|
|||||||
for topic in self._topic_to_ports:
|
for topic in self._topic_to_ports:
|
||||||
await client.subscribe(topic, qos=self.qos)
|
await client.subscribe(topic, qos=self.qos)
|
||||||
logger.info("[%s] Subscribed to %s", self.name, topic)
|
logger.info("[%s] Subscribed to %s", self.name, topic)
|
||||||
|
self.report_health("ok")
|
||||||
|
|
||||||
async for message in client.messages:
|
async for message in client.messages:
|
||||||
if self._stop_event and self._stop_event.is_set():
|
if self._stop_event and self._stop_event.is_set():
|
||||||
@@ -1152,6 +1195,7 @@ class MqttNode(Node):
|
|||||||
self.name,
|
self.name,
|
||||||
e,
|
e,
|
||||||
)
|
)
|
||||||
|
self.report_health("down", str(e))
|
||||||
if not (self._stop_event and self._stop_event.is_set()):
|
if not (self._stop_event and self._stop_event.is_set()):
|
||||||
# Reconnect after a delay
|
# Reconnect after a delay
|
||||||
await asyncio.sleep(5)
|
await asyncio.sleep(5)
|
||||||
@@ -1311,7 +1355,7 @@ class InfluxDbNode(Node):
|
|||||||
model_config = ConfigDict(extra="allow")
|
model_config = ConfigDict(extra="allow")
|
||||||
|
|
||||||
url: str
|
url: str
|
||||||
token: str
|
token: str = Field(json_schema_extra={"x-secret": True})
|
||||||
org: str
|
org: str
|
||||||
bucket: str
|
bucket: str
|
||||||
write_precision: str = "ms"
|
write_precision: str = "ms"
|
||||||
@@ -1809,6 +1853,15 @@ class DelayNode(Node):
|
|||||||
# Cron scheduler
|
# Cron scheduler
|
||||||
# -----------------------------------------------------------------
|
# -----------------------------------------------------------------
|
||||||
|
|
||||||
|
async def start(self, app: FastAPI | None = None) -> None:
|
||||||
|
"""Only a node with a schedule has anything to run on its own."""
|
||||||
|
if self.cron_expr:
|
||||||
|
await self.start_cron()
|
||||||
|
|
||||||
|
async def stop(self, app: FastAPI | None = None) -> None:
|
||||||
|
if self.cron_expr:
|
||||||
|
await self.stop_cron()
|
||||||
|
|
||||||
async def start_cron(self) -> None:
|
async def start_cron(self) -> None:
|
||||||
"""
|
"""
|
||||||
Start the cron scheduler.
|
Start the cron scheduler.
|
||||||
|
|||||||
@@ -258,6 +258,33 @@ class Pipeline:
|
|||||||
def _timestamp_key(self, msg_name: str) -> str:
|
def _timestamp_key(self, msg_name: str) -> str:
|
||||||
return f"__ts__:{msg_name}"
|
return f"__ts__:{msg_name}"
|
||||||
|
|
||||||
|
def _delivered_key(self, node_name: str, msg_name: str) -> str:
|
||||||
|
"""When a rate-limited input last woke this node."""
|
||||||
|
return f"__in_ts__:{node_name}:{msg_name}"
|
||||||
|
|
||||||
|
def _throttled(self, node: Node, result: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Drop the outputs whose port is not due to publish yet.
|
||||||
|
|
||||||
|
The value is not lost — the port publishes the current one next time it
|
||||||
|
is due. Nothing declaring an interval means nothing to look up.
|
||||||
|
"""
|
||||||
|
limited = {
|
||||||
|
name: spec.interval
|
||||||
|
for name, spec in node.provides.items()
|
||||||
|
if spec.interval > 0 and name in result
|
||||||
|
}
|
||||||
|
if not limited:
|
||||||
|
return result
|
||||||
|
|
||||||
|
now = time.time()
|
||||||
|
stamps = self._state.get_multi([self._timestamp_key(name) for name in limited])
|
||||||
|
return {
|
||||||
|
name: value
|
||||||
|
for name, value in result.items()
|
||||||
|
if name not in limited
|
||||||
|
or now - (stamps.get(self._timestamp_key(name)) or 0) >= limited[name]
|
||||||
|
}
|
||||||
|
|
||||||
def _increment_message_versions(self, outputs: dict[str, Any]) -> None:
|
def _increment_message_versions(self, outputs: dict[str, Any]) -> None:
|
||||||
for msg_name in outputs:
|
for msg_name in outputs:
|
||||||
self._state.increment(self._version_key(msg_name))
|
self._state.increment(self._version_key(msg_name))
|
||||||
@@ -298,12 +325,47 @@ class Pipeline:
|
|||||||
|
|
||||||
return self._state.compare_and_swap_multi(expected, updates)
|
return self._state.compare_and_swap_multi(expected, updates)
|
||||||
|
|
||||||
|
def _input_is_due(self, node: Node) -> bool:
|
||||||
|
"""Has any rate-limited input waited out its interval?
|
||||||
|
|
||||||
|
A node runs when *any* of its inputs is due, so a slow port next to a
|
||||||
|
fast one throttles only itself. Nodes without a limited input never get
|
||||||
|
here.
|
||||||
|
"""
|
||||||
|
limited = {
|
||||||
|
name: spec.interval
|
||||||
|
for name, spec in node.requires.items()
|
||||||
|
if spec.interval > 0
|
||||||
|
}
|
||||||
|
if not limited:
|
||||||
|
return True
|
||||||
|
|
||||||
|
now = time.time()
|
||||||
|
keys = [self._delivered_key(node.id, name) for name in limited]
|
||||||
|
stamps = self._state.get_multi(keys)
|
||||||
|
due = [
|
||||||
|
name
|
||||||
|
for name, interval in limited.items()
|
||||||
|
if now - (stamps.get(self._delivered_key(node.id, name)) or 0) >= interval
|
||||||
|
]
|
||||||
|
# An unlimited input alongside a held-back one still wakes the node.
|
||||||
|
if not due and len(limited) == len(node.requires):
|
||||||
|
return False
|
||||||
|
|
||||||
|
with self._state.lock():
|
||||||
|
for name in due:
|
||||||
|
self._state[self._delivered_key(node.id, name)] = now
|
||||||
|
return True
|
||||||
|
|
||||||
def _is_node_ready(self, node: Node, state: StateBackend) -> bool:
|
def _is_node_ready(self, node: Node, state: StateBackend) -> bool:
|
||||||
with state.lock():
|
with state.lock():
|
||||||
for msg_name in node.requires:
|
for msg_name in node.requires:
|
||||||
if msg_name not in state:
|
if msg_name not in state:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
if not self._input_is_due(node):
|
||||||
|
return False
|
||||||
|
|
||||||
if not node.synchronous:
|
if not node.synchronous:
|
||||||
return True
|
return True
|
||||||
|
|
||||||
@@ -349,6 +411,9 @@ class Pipeline:
|
|||||||
result = node.execute(inputs)
|
result = node.execute(inputs)
|
||||||
self.publish_log(node, collected, "")
|
self.publish_log(node, collected, "")
|
||||||
|
|
||||||
|
if result:
|
||||||
|
result = self._throttled(node, result)
|
||||||
|
|
||||||
if result:
|
if result:
|
||||||
ts = time.time()
|
ts = time.time()
|
||||||
with state.lock():
|
with state.lock():
|
||||||
@@ -510,13 +575,18 @@ class Pipeline:
|
|||||||
down anyway, and anything still arriving from another thread would be
|
down anyway, and anything still arriving from another thread would be
|
||||||
work the flow was explicitly told not to do. A *paused* flow still
|
work the flow was explicitly told not to do. A *paused* flow still
|
||||||
publishes, so the incoming value is visible on the canvas, and holds
|
publishes, so the incoming value is visible on the canvas, and holds
|
||||||
the nodes downstream of it for stepping.
|
the nodes downstream of it.
|
||||||
"""
|
"""
|
||||||
state = self._state
|
state = self._state
|
||||||
|
|
||||||
if node.flow in self._disabled:
|
if node.flow in self._disabled:
|
||||||
return state
|
return state
|
||||||
|
|
||||||
|
if outputs:
|
||||||
|
# This is where a chatty subscriber gets thinned out, so a port set
|
||||||
|
# to publish every 60s does so whatever the broker sends.
|
||||||
|
outputs = self._throttled(node, outputs)
|
||||||
|
|
||||||
if outputs:
|
if outputs:
|
||||||
ts = time.time()
|
ts = time.time()
|
||||||
with state.lock():
|
with state.lock():
|
||||||
|
|||||||
@@ -0,0 +1,77 @@
|
|||||||
|
"""Loading connector plugins from installed packages.
|
||||||
|
|
||||||
|
A connector is an ordinary Python package that advertises its node class under
|
||||||
|
the ``fluksio.node_types`` entry point group. The entry point's name is the
|
||||||
|
node type as it appears on the canvas, and the package's own metadata — name
|
||||||
|
and version — is the manifest; there is no second format to keep in step.
|
||||||
|
|
||||||
|
Discovery happens once, at startup: a connector's code is imported, and Python
|
||||||
|
does not re-import a changed module, so a rescan would promise more than it
|
||||||
|
delivers. Installing or upgrading one means restarting the engine.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from importlib.metadata import entry_points
|
||||||
|
|
||||||
|
from app.flow.connector import CONTRACT_VERSION, ConnectorNode
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
ENTRY_POINT_GROUP = "fluksio.node_types"
|
||||||
|
|
||||||
|
|
||||||
|
def load_plugins() -> list[str]:
|
||||||
|
"""Register every installed connector. Returns the type names it added."""
|
||||||
|
from app.flow.controller import NODE_TYPES, NodeType, _schema_of
|
||||||
|
|
||||||
|
added: list[str] = []
|
||||||
|
for entry in entry_points(group=ENTRY_POINT_GROUP):
|
||||||
|
if entry.name in NODE_TYPES:
|
||||||
|
logger.warning(
|
||||||
|
"Ignoring connector '%s': that node type already exists", entry.name
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
cls = entry.load()
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Could not load connector '%s'", entry.name)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not (isinstance(cls, type) and issubclass(cls, ConnectorNode)):
|
||||||
|
logger.warning(
|
||||||
|
"Ignoring connector '%s': %r is not a ConnectorNode", entry.name, cls
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
# Checked on the class itself: inheriting a version from a base class
|
||||||
|
# would let a connector written against an older contract look current.
|
||||||
|
if cls.__dict__.get("contract") != CONTRACT_VERSION:
|
||||||
|
logger.warning(
|
||||||
|
"Ignoring connector '%s': written for contract version %s, "
|
||||||
|
"this engine speaks %s",
|
||||||
|
entry.name,
|
||||||
|
cls.__dict__.get("contract"),
|
||||||
|
CONTRACT_VERSION,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
NODE_TYPES[entry.name] = NodeType(
|
||||||
|
title=cls.title or entry.name,
|
||||||
|
description=cls.description,
|
||||||
|
cls=cls,
|
||||||
|
params_schema=_schema_of(cls),
|
||||||
|
plugin=_provenance(entry),
|
||||||
|
)
|
||||||
|
added.append(entry.name)
|
||||||
|
logger.info("Loaded connector '%s' from %s", entry.name, _provenance(entry))
|
||||||
|
|
||||||
|
return added
|
||||||
|
|
||||||
|
|
||||||
|
def _provenance(entry: object) -> str:
|
||||||
|
"""Which package this node type came from, for the editor to show."""
|
||||||
|
dist = getattr(entry, "dist", None)
|
||||||
|
if dist is None:
|
||||||
|
return "unknown"
|
||||||
|
return f"{dist.name} {dist.version}"
|
||||||
@@ -7,7 +7,7 @@ A flow is structure plus code: this module is the structure. Node logic for
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import re
|
import re
|
||||||
from typing import Any
|
from typing import Any, Literal
|
||||||
|
|
||||||
from pydantic import BaseModel, Field, field_validator
|
from pydantic import BaseModel, Field, field_validator
|
||||||
|
|
||||||
@@ -41,6 +41,9 @@ class NodeDef(BaseModel):
|
|||||||
params: dict[str, Any] = Field(default_factory=dict)
|
params: dict[str, Any] = Field(default_factory=dict)
|
||||||
requires: list[MessageSpec] = Field(default_factory=list)
|
requires: list[MessageSpec] = Field(default_factory=list)
|
||||||
provides: list[MessageSpec] = Field(default_factory=list)
|
provides: list[MessageSpec] = Field(default_factory=list)
|
||||||
|
#: Name of a shared source in the library, instead of this node's own file.
|
||||||
|
#: Editing it edits the copy every flow using it runs.
|
||||||
|
source_ref: str | None = None
|
||||||
|
|
||||||
@field_validator("id")
|
@field_validator("id")
|
||||||
@classmethod
|
@classmethod
|
||||||
@@ -76,12 +79,17 @@ class NodeSource(BaseModel):
|
|||||||
code: str
|
code: str
|
||||||
|
|
||||||
|
|
||||||
|
Health = Literal["ok", "degraded", "down"]
|
||||||
|
|
||||||
|
|
||||||
class NodeStatusPublic(BaseModel):
|
class NodeStatusPublic(BaseModel):
|
||||||
"""Whether a node loaded, and why not."""
|
"""Whether a node loaded, and how its connection is doing."""
|
||||||
|
|
||||||
id: str
|
id: str
|
||||||
status: str = "active"
|
status: str = "active"
|
||||||
error: str | None = None
|
error: str | None = None
|
||||||
|
health: Health = "ok"
|
||||||
|
health_detail: str | None = None
|
||||||
|
|
||||||
|
|
||||||
class MessageValue(BaseModel):
|
class MessageValue(BaseModel):
|
||||||
@@ -125,6 +133,13 @@ class FlowsPublic(BaseModel):
|
|||||||
count: int
|
count: int
|
||||||
|
|
||||||
|
|
||||||
|
class LibraryNode(BaseModel):
|
||||||
|
"""A node source shared across flows, and who is using it."""
|
||||||
|
|
||||||
|
name: str
|
||||||
|
used_by: list[str] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
class FlowStatePublic(BaseModel):
|
class FlowStatePublic(BaseModel):
|
||||||
values: dict[str, MessageValue] = Field(default_factory=dict)
|
values: dict[str, MessageValue] = Field(default_factory=dict)
|
||||||
nodes: list[NodeStatusPublic] = Field(default_factory=list)
|
nodes: list[NodeStatusPublic] = Field(default_factory=list)
|
||||||
@@ -138,3 +153,5 @@ class NodeTypeInfo(BaseModel):
|
|||||||
description: str
|
description: str
|
||||||
params_schema: dict[str, Any] = Field(default_factory=dict)
|
params_schema: dict[str, Any] = Field(default_factory=dict)
|
||||||
has_source: bool = False
|
has_source: bool = False
|
||||||
|
#: The package a connector came from; empty for the built-in types.
|
||||||
|
plugin: str | None = None
|
||||||
|
|||||||
@@ -25,6 +25,9 @@ from app.flow.schemas import FlowDef
|
|||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
#: Where the sources of nodes shared between flows live.
|
||||||
|
LIB_DIR = "_lib"
|
||||||
|
|
||||||
DEFAULT_SOURCE = '''"""A new node. Return a dict keyed by your output ports."""
|
DEFAULT_SOURCE = '''"""A new node. Return a dict keyed by your output ports."""
|
||||||
|
|
||||||
|
|
||||||
@@ -51,6 +54,24 @@ class FlowExists(ValueError):
|
|||||||
return f"There is already a flow named '{self.name}'"
|
return f"There is already a flow named '{self.name}'"
|
||||||
|
|
||||||
|
|
||||||
|
class LibNotFound(KeyError):
|
||||||
|
def __init__(self, name: str) -> None:
|
||||||
|
super().__init__(name)
|
||||||
|
self.name = name
|
||||||
|
|
||||||
|
def __str__(self) -> str:
|
||||||
|
return f"No shared node named '{self.name}'"
|
||||||
|
|
||||||
|
|
||||||
|
class LibExists(ValueError):
|
||||||
|
def __init__(self, name: str) -> None:
|
||||||
|
super().__init__(name)
|
||||||
|
self.name = name
|
||||||
|
|
||||||
|
def __str__(self) -> str:
|
||||||
|
return f"There is already a shared node named '{self.name}'"
|
||||||
|
|
||||||
|
|
||||||
class StaleVersion(ValueError):
|
class StaleVersion(ValueError):
|
||||||
"""Someone else saved this flow since the version being written was read."""
|
"""Someone else saved this flow since the version being written was read."""
|
||||||
|
|
||||||
@@ -136,6 +157,109 @@ class FlowStore:
|
|||||||
def _runtime_file(self, name: str) -> Path:
|
def _runtime_file(self, name: str) -> Path:
|
||||||
return self._flow_dir(name) / "runtime.json"
|
return self._flow_dir(name) / "runtime.json"
|
||||||
|
|
||||||
|
def _lib_file(self, name: str) -> Path:
|
||||||
|
return self.root / LIB_DIR / f"{name}.py"
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Shared node sources
|
||||||
|
#
|
||||||
|
# A node marked reusable moves its code to ``_lib/``; the flows using it
|
||||||
|
# keep their own ports and parameters and point at it by name, so one fix
|
||||||
|
# reaches all of them. ``_lib`` cannot collide with a flow — a flow name has
|
||||||
|
# to start with a letter — and holds no ``flow.json``, so it is invisible to
|
||||||
|
# everything that lists flows.
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def list_lib(self) -> list[str]:
|
||||||
|
return sorted(path.stem for path in (self.root / LIB_DIR).glob("*.py"))
|
||||||
|
|
||||||
|
def lib_exists(self, name: str) -> bool:
|
||||||
|
return self._lib_file(name).exists()
|
||||||
|
|
||||||
|
def read_lib_source(self, name: str) -> str:
|
||||||
|
path = self._lib_file(name)
|
||||||
|
if not path.exists():
|
||||||
|
raise LibNotFound(name)
|
||||||
|
return path.read_text()
|
||||||
|
|
||||||
|
def write_lib_source(self, name: str, code: str) -> bool:
|
||||||
|
path = self._lib_file(name)
|
||||||
|
if path.exists() and path.read_text() == code:
|
||||||
|
return False
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(code)
|
||||||
|
self._commit(f"Update shared node '{name}'")
|
||||||
|
return True
|
||||||
|
|
||||||
|
def delete_lib_source(self, name: str) -> None:
|
||||||
|
path = self._lib_file(name)
|
||||||
|
if not path.exists():
|
||||||
|
raise LibNotFound(name)
|
||||||
|
path.unlink()
|
||||||
|
self._commit(f"Delete shared node '{name}'")
|
||||||
|
|
||||||
|
def share_node(self, flow: str, node_id: str, lib_name: str) -> None:
|
||||||
|
"""Move a node's source into the library and point the node at it."""
|
||||||
|
with self._write_lock:
|
||||||
|
if self.lib_exists(lib_name):
|
||||||
|
raise LibExists(lib_name)
|
||||||
|
|
||||||
|
code = self.read_node_source(flow, node_id, draft=True)
|
||||||
|
self._lib_file(lib_name).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
self._lib_file(lib_name).write_text(code)
|
||||||
|
|
||||||
|
for path in (self._node_file(flow, node_id),
|
||||||
|
self._draft_node_file(flow, node_id)):
|
||||||
|
if path.exists():
|
||||||
|
path.unlink()
|
||||||
|
|
||||||
|
self._set_source_ref(flow, node_id, lib_name)
|
||||||
|
self._commit(f"Share node '{flow}.{node_id}' as '{lib_name}'")
|
||||||
|
|
||||||
|
def unshare_node(self, flow: str, node_id: str) -> None:
|
||||||
|
"""Take a private copy of the shared source back into the flow."""
|
||||||
|
with self._write_lock:
|
||||||
|
definition = self.read_flow(flow, draft=True)
|
||||||
|
node = next((n for n in definition.nodes if n.id == node_id), None)
|
||||||
|
if node is None or not node.source_ref:
|
||||||
|
return
|
||||||
|
code = self.read_lib_source(node.source_ref)
|
||||||
|
# Written as a draft: the running flow keeps the shared code until
|
||||||
|
# this is published, like every other edit.
|
||||||
|
self._draft_node_file(flow, node_id).parent.mkdir(
|
||||||
|
parents=True, exist_ok=True
|
||||||
|
)
|
||||||
|
self._draft_node_file(flow, node_id).write_text(code)
|
||||||
|
self._set_source_ref(flow, node_id, None)
|
||||||
|
self._commit(f"Unshare node '{flow}.{node_id}'")
|
||||||
|
|
||||||
|
def usages(self, lib_name: str) -> list[str]:
|
||||||
|
"""Every ``flow.node`` pointing at this shared source."""
|
||||||
|
used = []
|
||||||
|
for name in self.list_flows():
|
||||||
|
try:
|
||||||
|
definition = self.read_flow(name, draft=True)
|
||||||
|
except FlowNotFound:
|
||||||
|
continue
|
||||||
|
used += [
|
||||||
|
f"{name}.{node.id}"
|
||||||
|
for node in definition.nodes
|
||||||
|
if node.source_ref == lib_name
|
||||||
|
]
|
||||||
|
return sorted(used)
|
||||||
|
|
||||||
|
def _set_source_ref(self, flow: str, node_id: str, ref: str | None) -> None:
|
||||||
|
"""Point a node at a shared source, in the draft the editor is on."""
|
||||||
|
definition = self.read_flow(flow, draft=True)
|
||||||
|
definition.nodes = [
|
||||||
|
node.model_copy(update={"source_ref": ref}) if node.id == node_id else node
|
||||||
|
for node in definition.nodes
|
||||||
|
]
|
||||||
|
stored = definition.model_copy(update={"version": definition.version + 1})
|
||||||
|
path = self._draft_file(flow)
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(stored.model_dump_json(indent=2) + "\n")
|
||||||
|
|
||||||
# -------------------------------------------------------------------------
|
# -------------------------------------------------------------------------
|
||||||
# Runtime state
|
# Runtime state
|
||||||
# -------------------------------------------------------------------------
|
# -------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ from app.core.config import settings
|
|||||||
from app.flow import logs
|
from app.flow import logs
|
||||||
from app.flow.controller import FlowController
|
from app.flow.controller import FlowController
|
||||||
from app.flow.events import event_bus
|
from app.flow.events import event_bus
|
||||||
|
from app.flow.plugins import load_plugins
|
||||||
from app.flow.secrets import init_secrets
|
from app.flow.secrets import init_secrets
|
||||||
from app.flow.state import MemoryState, RedisState, StateBackend
|
from app.flow.state import MemoryState, RedisState, StateBackend
|
||||||
from app.flow.store import FlowStore
|
from app.flow.store import FlowStore
|
||||||
@@ -38,6 +39,8 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
|||||||
# Node code is user code, and `print` is how it says things.
|
# Node code is user code, and `print` is how it says things.
|
||||||
logs.install()
|
logs.install()
|
||||||
init_secrets(settings.SECRETS_FILE, settings.SECRET_KEY)
|
init_secrets(settings.SECRETS_FILE, settings.SECRET_KEY)
|
||||||
|
# Connectors register their node types before any flow is built with them.
|
||||||
|
load_plugins()
|
||||||
|
|
||||||
controller = FlowController(
|
controller = FlowController(
|
||||||
store=FlowStore(settings.FLOWS_DIR),
|
store=FlowStore(settings.FLOWS_DIR),
|
||||||
|
|||||||
@@ -0,0 +1,156 @@
|
|||||||
|
"""The connector contract: polling, deduplication, health and discovery."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from app.flow.connector import CONTRACT_VERSION, ConnectorNode
|
||||||
|
from app.flow.controller import NODE_TYPES
|
||||||
|
from app.flow.messages import DType, MessageSpec
|
||||||
|
from app.flow.nodes import Node
|
||||||
|
from app.flow.pipeline import Pipeline
|
||||||
|
from app.flow.plugins import load_plugins
|
||||||
|
|
||||||
|
|
||||||
|
class Sensor(ConnectorNode):
|
||||||
|
contract = CONTRACT_VERSION
|
||||||
|
title = "Test sensor"
|
||||||
|
description = "Reads whatever it is told to."
|
||||||
|
|
||||||
|
class Params(ConnectorNode.Params):
|
||||||
|
secret_token: str | None = None
|
||||||
|
|
||||||
|
def __init__(self, readings: list[Any] | None = None, **kwargs: Any) -> None:
|
||||||
|
super().__init__(**kwargs)
|
||||||
|
self._readings = list(readings or [])
|
||||||
|
self.polls = 0
|
||||||
|
|
||||||
|
async def poll(self) -> dict[str, Any] | None:
|
||||||
|
self.polls += 1
|
||||||
|
if not self._readings:
|
||||||
|
return None
|
||||||
|
value = self._readings.pop(0)
|
||||||
|
if isinstance(value, Exception):
|
||||||
|
raise value
|
||||||
|
return {"reading": value}
|
||||||
|
|
||||||
|
|
||||||
|
def a_sensor(readings: list[Any], **params: Any) -> Sensor:
|
||||||
|
node = Sensor(
|
||||||
|
readings=readings,
|
||||||
|
provides=[MessageSpec(name="reading", dtype=DType.FLOAT)],
|
||||||
|
params={"poll_interval": 0.01, **params},
|
||||||
|
)
|
||||||
|
node.assign_flow("demo", "sensor")
|
||||||
|
return node
|
||||||
|
|
||||||
|
|
||||||
|
def run_briefly(node: ConnectorNode, seconds: float = 0.12) -> None:
|
||||||
|
"""Start the poll loop, let it tick a few times, stop it."""
|
||||||
|
|
||||||
|
async def cycle() -> None:
|
||||||
|
await node.start()
|
||||||
|
await asyncio.sleep(seconds)
|
||||||
|
await node.stop()
|
||||||
|
|
||||||
|
asyncio.run(cycle())
|
||||||
|
|
||||||
|
|
||||||
|
def test_polling_publishes_what_it_reads():
|
||||||
|
node = a_sensor([21.5])
|
||||||
|
pipeline = Pipeline(nodes=[node])
|
||||||
|
|
||||||
|
run_briefly(node)
|
||||||
|
|
||||||
|
assert pipeline.state["demo.reading"] == 21.5
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unchanged_reading_is_not_republished():
|
||||||
|
node = a_sensor([21.5, 21.5, 21.5])
|
||||||
|
consumer_ran: list[float] = []
|
||||||
|
|
||||||
|
def consume(reading, params):
|
||||||
|
consumer_ran.append(reading)
|
||||||
|
return None
|
||||||
|
|
||||||
|
consumer = Node(
|
||||||
|
f=consume,
|
||||||
|
requires=[MessageSpec(name="reading", dtype=DType.FLOAT)],
|
||||||
|
name="consumer",
|
||||||
|
)
|
||||||
|
consumer.assign_flow("demo", "consumer")
|
||||||
|
Pipeline(nodes=[node, consumer])
|
||||||
|
|
||||||
|
run_briefly(node)
|
||||||
|
|
||||||
|
# Polled repeatedly, but the value never changed, so downstream ran once.
|
||||||
|
assert node.polls > 1
|
||||||
|
assert consumer_ran == [21.5]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_failing_poll_reports_down_and_keeps_going():
|
||||||
|
health: list[tuple[str, str | None]] = []
|
||||||
|
node = a_sensor([RuntimeError("device unplugged"), 21.5])
|
||||||
|
node._on_health = lambda _node, status, detail: health.append((status, detail))
|
||||||
|
Pipeline(nodes=[node])
|
||||||
|
|
||||||
|
run_briefly(node)
|
||||||
|
|
||||||
|
assert ("down", "RuntimeError: device unplugged") in health
|
||||||
|
# It recovered rather than giving up.
|
||||||
|
assert health[-1][0] == "ok"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_credential_param_is_marked_for_the_editor():
|
||||||
|
schema = Sensor.Params.model_json_schema()
|
||||||
|
assert schema["properties"]["poll_interval"]["default"] == 0
|
||||||
|
assert "secret_token" in schema["properties"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_connector_is_discovered_from_its_entry_point(monkeypatch):
|
||||||
|
class FakeDist:
|
||||||
|
name = "fluksio-connector-test"
|
||||||
|
version = "0.1.0"
|
||||||
|
|
||||||
|
class FakeEntry:
|
||||||
|
name = "test_sensor"
|
||||||
|
dist = FakeDist()
|
||||||
|
|
||||||
|
def load(self):
|
||||||
|
return Sensor
|
||||||
|
|
||||||
|
monkeypatch.setattr("app.flow.plugins.entry_points", lambda group: [FakeEntry()])
|
||||||
|
try:
|
||||||
|
assert load_plugins() == ["test_sensor"]
|
||||||
|
assert NODE_TYPES["test_sensor"].plugin == "fluksio-connector-test 0.1.0"
|
||||||
|
assert NODE_TYPES["test_sensor"].title == "Test sensor"
|
||||||
|
finally:
|
||||||
|
NODE_TYPES.pop("test_sensor", None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_connector_written_for_another_contract_is_refused(monkeypatch):
|
||||||
|
class Outdated(ConnectorNode):
|
||||||
|
contract = CONTRACT_VERSION + 1
|
||||||
|
|
||||||
|
class FakeEntry:
|
||||||
|
name = "outdated"
|
||||||
|
dist = None
|
||||||
|
|
||||||
|
def load(self):
|
||||||
|
return Outdated
|
||||||
|
|
||||||
|
monkeypatch.setattr("app.flow.plugins.entry_points", lambda group: [FakeEntry()])
|
||||||
|
assert load_plugins() == []
|
||||||
|
assert "outdated" not in NODE_TYPES
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_connector_may_not_take_over_a_built_in_type(monkeypatch):
|
||||||
|
class FakeEntry:
|
||||||
|
name = "mqtt"
|
||||||
|
dist = None
|
||||||
|
|
||||||
|
def load(self): # pragma: no cover - never reached
|
||||||
|
raise AssertionError("should not be loaded")
|
||||||
|
|
||||||
|
monkeypatch.setattr("app.flow.plugins.entry_points", lambda group: [FakeEntry()])
|
||||||
|
assert load_plugins() == []
|
||||||
|
assert NODE_TYPES["mqtt"].plugin is None
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
"""Per-port intervals: deliver at most every x seconds."""
|
||||||
|
|
||||||
|
from app.flow.messages import DType, MessageSpec
|
||||||
|
from app.flow.nodes import Node
|
||||||
|
from app.flow.pipeline import Pipeline
|
||||||
|
|
||||||
|
|
||||||
|
def spec(name: str, interval: float = 0) -> MessageSpec:
|
||||||
|
return MessageSpec(name=name, dtype=DType.FLOAT, interval=interval)
|
||||||
|
|
||||||
|
|
||||||
|
def make_node(node_id: str, f, requires=(), provides=()) -> Node:
|
||||||
|
node = Node(f=f, requires=list(requires), provides=list(provides), name=node_id)
|
||||||
|
node.assign_flow("demo", node_id)
|
||||||
|
return node
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_limited_output_publishes_once_inside_its_window():
|
||||||
|
readings = iter([1.0, 2.0, 3.0])
|
||||||
|
source = make_node(
|
||||||
|
"source",
|
||||||
|
lambda params: {"temp": next(readings)},
|
||||||
|
provides=[spec("temp", interval=60)],
|
||||||
|
)
|
||||||
|
pipeline = Pipeline(nodes=[source])
|
||||||
|
|
||||||
|
pipeline.run({})
|
||||||
|
assert pipeline.state["demo.temp"] == 1.0
|
||||||
|
|
||||||
|
# Same window: the reading is taken but not published.
|
||||||
|
pipeline.run({})
|
||||||
|
assert pipeline.state["demo.temp"] == 1.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unlimited_output_publishes_every_time():
|
||||||
|
readings = iter([1.0, 2.0])
|
||||||
|
source = make_node(
|
||||||
|
"source",
|
||||||
|
lambda params: {"temp": next(readings)},
|
||||||
|
provides=[spec("temp")],
|
||||||
|
)
|
||||||
|
pipeline = Pipeline(nodes=[source])
|
||||||
|
|
||||||
|
pipeline.run({})
|
||||||
|
pipeline.run({})
|
||||||
|
assert pipeline.state["demo.temp"] == 2.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_limited_input_wakes_its_node_once_inside_the_window():
|
||||||
|
seen: list[float] = []
|
||||||
|
source = make_node("source", lambda params: {"temp": 20.0}, provides=[spec("temp")])
|
||||||
|
consumer = make_node(
|
||||||
|
"consumer",
|
||||||
|
lambda temp, params: seen.append(temp),
|
||||||
|
requires=[spec("temp", interval=60)],
|
||||||
|
)
|
||||||
|
# Binding the nodes is what the pipeline is for here.
|
||||||
|
Pipeline(nodes=[source, consumer])
|
||||||
|
|
||||||
|
source.inject({"temp": 20.0})
|
||||||
|
source.inject({"temp": 21.0})
|
||||||
|
|
||||||
|
assert seen == [20.0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unthrottled_input_still_wakes_a_node_beside_a_throttled_one():
|
||||||
|
seen: list[tuple[float, float]] = []
|
||||||
|
fast = make_node("fast", lambda params: None, provides=[spec("quick")])
|
||||||
|
slow = make_node("slow", lambda params: None, provides=[spec("rare")])
|
||||||
|
consumer = make_node(
|
||||||
|
"consumer",
|
||||||
|
lambda quick, rare, params: seen.append((quick, rare)),
|
||||||
|
requires=[spec("quick"), spec("rare", interval=60)],
|
||||||
|
)
|
||||||
|
pipeline = Pipeline(nodes=[fast, slow, consumer])
|
||||||
|
pipeline.state["demo.rare"] = 1.0
|
||||||
|
|
||||||
|
fast.inject({"quick": 1.0})
|
||||||
|
fast.inject({"quick": 2.0})
|
||||||
|
|
||||||
|
# The throttled port holds back only itself.
|
||||||
|
assert [quick for quick, _ in seen] == [1.0, 2.0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_manual_run_is_never_throttled_on_its_inputs():
|
||||||
|
seen: list[float] = []
|
||||||
|
source = make_node("source", lambda params: {"temp": 20.0}, provides=[spec("temp")])
|
||||||
|
consumer = make_node(
|
||||||
|
"consumer",
|
||||||
|
lambda temp, params: seen.append(temp),
|
||||||
|
requires=[spec("temp", interval=3600)],
|
||||||
|
)
|
||||||
|
pipeline = Pipeline(nodes=[source, consumer])
|
||||||
|
|
||||||
|
# Pressing Run is an explicit ask; the interval governs the flow's own
|
||||||
|
# traffic, not what the person in front of it asked for.
|
||||||
|
pipeline.run({})
|
||||||
|
pipeline.run({})
|
||||||
|
|
||||||
|
assert len(seen) == 2
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
"""Nodes shared across flows: one source, many instances."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from app.flow.messages import MessageSpec
|
||||||
|
from app.flow.schemas import FlowDef, NodeDef
|
||||||
|
from app.flow.store import FlowStore, LibExists, LibNotFound
|
||||||
|
|
||||||
|
SOURCE = "def process(params):\n return {'temp': 1}\n"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def store(tmp_path: Path) -> FlowStore:
|
||||||
|
return FlowStore(tmp_path / "flows")
|
||||||
|
|
||||||
|
|
||||||
|
def a_flow(name: str = "heating") -> FlowDef:
|
||||||
|
return FlowDef(
|
||||||
|
name=name,
|
||||||
|
nodes=[NodeDef(id="sensor", provides=[MessageSpec(name="temp")])],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_sharing_moves_the_source_and_points_the_node_at_it(store: FlowStore):
|
||||||
|
store.write_flow(a_flow())
|
||||||
|
store.write_node_source("heating", "sensor", SOURCE)
|
||||||
|
|
||||||
|
store.share_node("heating", "sensor", "read_temp")
|
||||||
|
|
||||||
|
assert store.list_lib() == ["read_temp"]
|
||||||
|
assert store.read_lib_source("read_temp") == SOURCE
|
||||||
|
node = store.read_flow("heating", draft=True).nodes[0]
|
||||||
|
assert node.source_ref == "read_temp"
|
||||||
|
# The private copy is gone; the library one is what it runs.
|
||||||
|
assert not (store.root / "heating" / "nodes" / "sensor.py").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_library_is_not_mistaken_for_a_flow(store: FlowStore):
|
||||||
|
store.write_flow(a_flow())
|
||||||
|
store.write_node_source("heating", "sensor", SOURCE)
|
||||||
|
store.share_node("heating", "sensor", "read_temp")
|
||||||
|
|
||||||
|
assert store.list_flows() == ["heating"]
|
||||||
|
assert [flow.name for flow in store.read_all()] == ["heating"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_second_flow_can_use_the_same_source(store: FlowStore):
|
||||||
|
store.write_flow(a_flow())
|
||||||
|
store.write_node_source("heating", "sensor", SOURCE)
|
||||||
|
store.share_node("heating", "sensor", "read_temp")
|
||||||
|
|
||||||
|
store.write_flow(
|
||||||
|
FlowDef(
|
||||||
|
name="cooling",
|
||||||
|
nodes=[
|
||||||
|
NodeDef(
|
||||||
|
id="sensor",
|
||||||
|
source_ref="read_temp",
|
||||||
|
provides=[MessageSpec(name="temp")],
|
||||||
|
)
|
||||||
|
],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert store.usages("read_temp") == ["cooling.sensor", "heating.sensor"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_shared_name_is_not_taken_twice(store: FlowStore):
|
||||||
|
store.write_flow(a_flow())
|
||||||
|
store.write_node_source("heating", "sensor", SOURCE)
|
||||||
|
store.share_node("heating", "sensor", "read_temp")
|
||||||
|
|
||||||
|
store.write_flow(FlowDef(name="cooling", nodes=[NodeDef(id="sensor")]))
|
||||||
|
with pytest.raises(LibExists):
|
||||||
|
store.share_node("cooling", "sensor", "read_temp")
|
||||||
|
|
||||||
|
|
||||||
|
def test_unsharing_takes_a_private_copy_back(store: FlowStore):
|
||||||
|
store.write_flow(a_flow())
|
||||||
|
store.write_node_source("heating", "sensor", SOURCE)
|
||||||
|
store.share_node("heating", "sensor", "read_temp")
|
||||||
|
|
||||||
|
store.unshare_node("heating", "sensor")
|
||||||
|
|
||||||
|
node = store.read_flow("heating", draft=True).nodes[0]
|
||||||
|
assert node.source_ref is None
|
||||||
|
assert store.read_node_source("heating", "sensor", draft=True) == SOURCE
|
||||||
|
# The library keeps its copy for whoever else is using it.
|
||||||
|
assert store.list_lib() == ["read_temp"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_missing_shared_source_is_reported(store: FlowStore):
|
||||||
|
with pytest.raises(LibNotFound):
|
||||||
|
store.read_lib_source("nothing_here")
|
||||||
@@ -343,6 +343,26 @@ export const HistoryPointSchema = {
|
|||||||
description: 'One numeric value a message carried, and when.'
|
description: 'One numeric value a message carried, and when.'
|
||||||
} as const;
|
} as const;
|
||||||
|
|
||||||
|
export const LibraryNodeSchema = {
|
||||||
|
properties: {
|
||||||
|
name: {
|
||||||
|
type: 'string',
|
||||||
|
title: 'Name'
|
||||||
|
},
|
||||||
|
used_by: {
|
||||||
|
items: {
|
||||||
|
type: 'string'
|
||||||
|
},
|
||||||
|
type: 'array',
|
||||||
|
title: 'Used By'
|
||||||
|
}
|
||||||
|
},
|
||||||
|
type: 'object',
|
||||||
|
required: ['name'],
|
||||||
|
title: 'LibraryNode',
|
||||||
|
description: 'A node source shared across flows, and who is using it.'
|
||||||
|
} as const;
|
||||||
|
|
||||||
export const MessageSchema = {
|
export const MessageSchema = {
|
||||||
properties: {
|
properties: {
|
||||||
message: {
|
message: {
|
||||||
@@ -398,6 +418,12 @@ export const MessageSpecSchema = {
|
|||||||
dtype: {
|
dtype: {
|
||||||
'$ref': '#/components/schemas/DType',
|
'$ref': '#/components/schemas/DType',
|
||||||
default: 'float'
|
default: 'float'
|
||||||
|
},
|
||||||
|
interval: {
|
||||||
|
type: 'number',
|
||||||
|
minimum: 0,
|
||||||
|
title: 'Interval',
|
||||||
|
default: 0
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
type: 'object',
|
type: 'object',
|
||||||
@@ -408,7 +434,11 @@ export const MessageSpecSchema = {
|
|||||||
the flow name at load time; empty means the port is unbound.
|
the flow name at load time; empty means the port is unbound.
|
||||||
:param port: The identifier the node function sees. Defaults to the last
|
:param port: The identifier the node function sees. Defaults to the last
|
||||||
segment of \`\`name\`\`, so unqualified flows read naturally.
|
segment of \`\`name\`\`, so unqualified flows read naturally.
|
||||||
:param dtype: Payload type, validated on every message that passes through.`
|
:param dtype: Payload type, validated on every message that passes through.
|
||||||
|
:param interval: Deliver at most every this many seconds; 0 is every time.
|
||||||
|
On an output it holds back publishing, on an input it holds back waking
|
||||||
|
the node. The value is never lost — state keeps the latest — only the
|
||||||
|
delivery is skipped.`
|
||||||
} as const;
|
} as const;
|
||||||
|
|
||||||
export const MessageValueSchema = {
|
export const MessageValueSchema = {
|
||||||
@@ -492,6 +522,17 @@ export const NodeDef_InputSchema = {
|
|||||||
},
|
},
|
||||||
type: 'array',
|
type: 'array',
|
||||||
title: 'Provides'
|
title: 'Provides'
|
||||||
|
},
|
||||||
|
source_ref: {
|
||||||
|
anyOf: [
|
||||||
|
{
|
||||||
|
type: 'string'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'null'
|
||||||
|
}
|
||||||
|
],
|
||||||
|
title: 'Source Ref'
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
type: 'object',
|
type: 'object',
|
||||||
@@ -541,6 +582,17 @@ export const NodeDef_OutputSchema = {
|
|||||||
},
|
},
|
||||||
type: 'array',
|
type: 'array',
|
||||||
title: 'Provides'
|
title: 'Provides'
|
||||||
|
},
|
||||||
|
source_ref: {
|
||||||
|
anyOf: [
|
||||||
|
{
|
||||||
|
type: 'string'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'null'
|
||||||
|
}
|
||||||
|
],
|
||||||
|
title: 'Source Ref'
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
type: 'object',
|
type: 'object',
|
||||||
@@ -583,12 +635,29 @@ export const NodeStatusPublicSchema = {
|
|||||||
}
|
}
|
||||||
],
|
],
|
||||||
title: 'Error'
|
title: 'Error'
|
||||||
|
},
|
||||||
|
health: {
|
||||||
|
type: 'string',
|
||||||
|
enum: ['ok', 'degraded', 'down'],
|
||||||
|
title: 'Health',
|
||||||
|
default: 'ok'
|
||||||
|
},
|
||||||
|
health_detail: {
|
||||||
|
anyOf: [
|
||||||
|
{
|
||||||
|
type: 'string'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'null'
|
||||||
|
}
|
||||||
|
],
|
||||||
|
title: 'Health Detail'
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
type: 'object',
|
type: 'object',
|
||||||
required: ['id'],
|
required: ['id'],
|
||||||
title: 'NodeStatusPublic',
|
title: 'NodeStatusPublic',
|
||||||
description: 'Whether a node loaded, and why not.'
|
description: 'Whether a node loaded, and how its connection is doing.'
|
||||||
} as const;
|
} as const;
|
||||||
|
|
||||||
export const NodeTypeInfoSchema = {
|
export const NodeTypeInfoSchema = {
|
||||||
@@ -614,6 +683,17 @@ export const NodeTypeInfoSchema = {
|
|||||||
type: 'boolean',
|
type: 'boolean',
|
||||||
title: 'Has Source',
|
title: 'Has Source',
|
||||||
default: false
|
default: false
|
||||||
|
},
|
||||||
|
plugin: {
|
||||||
|
anyOf: [
|
||||||
|
{
|
||||||
|
type: 'string'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'null'
|
||||||
|
}
|
||||||
|
],
|
||||||
|
title: 'Plugin'
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
type: 'object',
|
type: 'object',
|
||||||
@@ -733,6 +813,18 @@ export const SecretValueSchema = {
|
|||||||
title: 'SecretValue'
|
title: 'SecretValue'
|
||||||
} as const;
|
} as const;
|
||||||
|
|
||||||
|
export const ShareRequestSchema = {
|
||||||
|
properties: {
|
||||||
|
lib_name: {
|
||||||
|
type: 'string',
|
||||||
|
title: 'Lib Name'
|
||||||
|
}
|
||||||
|
},
|
||||||
|
type: 'object',
|
||||||
|
required: ['lib_name'],
|
||||||
|
title: 'ShareRequest'
|
||||||
|
} as const;
|
||||||
|
|
||||||
export const TokenSchema = {
|
export const TokenSchema = {
|
||||||
properties: {
|
properties: {
|
||||||
access_token: {
|
access_token: {
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
import type { CancelablePromise } from './core/CancelablePromise';
|
import type { CancelablePromise } from './core/CancelablePromise';
|
||||||
import { OpenAPI } from './core/OpenAPI';
|
import { OpenAPI } from './core/OpenAPI';
|
||||||
import { request as __request } from './core/request';
|
import { request as __request } from './core/request';
|
||||||
import type { FlowsReadFlowsResponse, FlowsReadNodeTypesResponse, FlowsReadFlowData, FlowsReadFlowResponse, FlowsSaveFlowData, FlowsSaveFlowResponse, FlowsDeleteFlowData, FlowsDeleteFlowResponse, FlowsPublishFlowData, FlowsPublishFlowResponse, FlowsDiscardDraftData, FlowsDiscardDraftResponse, FlowsRenameFlowData, FlowsRenameFlowResponse, FlowsReadNodeSourceData, FlowsReadNodeSourceResponse, FlowsSaveNodeSourceData, FlowsSaveNodeSourceResponse, FlowsStartFlowData, FlowsStartFlowResponse, FlowsStopFlowData, FlowsStopFlowResponse, FlowsPauseFlowData, FlowsPauseFlowResponse, FlowsResumeFlowData, FlowsResumeFlowResponse, FlowsValidateFlowData, FlowsValidateFlowResponse, FlowsRunFlowData, FlowsRunFlowResponse, FlowsTriggerNodeData, FlowsTriggerNodeResponse, FlowsReadFlowStateData, FlowsReadFlowStateResponse, FlowsReadMessageHistoryData, FlowsReadMessageHistoryResponse, LoginLoginAccessTokenData, LoginLoginAccessTokenResponse, LoginTestTokenResponse, LoginRecoverPasswordData, LoginRecoverPasswordResponse, LoginResetPasswordData, LoginResetPasswordResponse, LoginRecoverPasswordHtmlContentData, LoginRecoverPasswordHtmlContentResponse, PrivateCreateUserData, PrivateCreateUserResponse, SecretsReadSecretsResponse, SecretsSaveSecretData, SecretsSaveSecretResponse, SecretsDeleteSecretData, SecretsDeleteSecretResponse, UsersReadUsersData, UsersReadUsersResponse, UsersCreateUserData, UsersCreateUserResponse, UsersReadUserMeResponse, UsersDeleteUserMeResponse, UsersUpdateUserMeData, UsersUpdateUserMeResponse, UsersUpdatePasswordMeData, UsersUpdatePasswordMeResponse, UsersRegisterUserData, UsersRegisterUserResponse, UsersReadUserByIdData, UsersReadUserByIdResponse, UsersUpdateUserData, UsersUpdateUserResponse, UsersDeleteUserData, UsersDeleteUserResponse, UtilsTestEmailData, UtilsTestEmailResponse, UtilsHealthCheckResponse } from './types.gen';
|
import type { FlowsReadFlowsResponse, FlowsReadNodeTypesResponse, FlowsReadLibraryResponse, FlowsDeleteSharedNodeData, FlowsDeleteSharedNodeResponse, FlowsReadFlowData, FlowsReadFlowResponse, FlowsSaveFlowData, FlowsSaveFlowResponse, FlowsDeleteFlowData, FlowsDeleteFlowResponse, FlowsPublishFlowData, FlowsPublishFlowResponse, FlowsDiscardDraftData, FlowsDiscardDraftResponse, FlowsRenameFlowData, FlowsRenameFlowResponse, FlowsReadNodeSourceData, FlowsReadNodeSourceResponse, FlowsSaveNodeSourceData, FlowsSaveNodeSourceResponse, FlowsShareNodeData, FlowsShareNodeResponse, FlowsUnshareNodeData, FlowsUnshareNodeResponse, FlowsStartFlowData, FlowsStartFlowResponse, FlowsStopFlowData, FlowsStopFlowResponse, FlowsPauseFlowData, FlowsPauseFlowResponse, FlowsResumeFlowData, FlowsResumeFlowResponse, FlowsValidateFlowData, FlowsValidateFlowResponse, FlowsRunFlowData, FlowsRunFlowResponse, FlowsTriggerNodeData, FlowsTriggerNodeResponse, FlowsReadFlowStateData, FlowsReadFlowStateResponse, FlowsReadMessageHistoryData, FlowsReadMessageHistoryResponse, LoginLoginAccessTokenData, LoginLoginAccessTokenResponse, LoginTestTokenResponse, LoginRecoverPasswordData, LoginRecoverPasswordResponse, LoginResetPasswordData, LoginResetPasswordResponse, LoginRecoverPasswordHtmlContentData, LoginRecoverPasswordHtmlContentResponse, PrivateCreateUserData, PrivateCreateUserResponse, SecretsReadSecretsResponse, SecretsSaveSecretData, SecretsSaveSecretResponse, SecretsDeleteSecretData, SecretsDeleteSecretResponse, UsersReadUsersData, UsersReadUsersResponse, UsersCreateUserData, UsersCreateUserResponse, UsersReadUserMeResponse, UsersDeleteUserMeResponse, UsersUpdateUserMeData, UsersUpdateUserMeResponse, UsersUpdatePasswordMeData, UsersUpdatePasswordMeResponse, UsersRegisterUserData, UsersRegisterUserResponse, UsersReadUserByIdData, UsersReadUserByIdResponse, UsersUpdateUserData, UsersUpdateUserResponse, UsersDeleteUserData, UsersDeleteUserResponse, UtilsTestEmailData, UtilsTestEmailResponse, UtilsHealthCheckResponse } from './types.gen';
|
||||||
|
|
||||||
export class FlowsService {
|
export class FlowsService {
|
||||||
/**
|
/**
|
||||||
@@ -32,6 +32,40 @@ export class FlowsService {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Read Library
|
||||||
|
* The node sources shared across flows, and which nodes use each.
|
||||||
|
* @returns LibraryNode Successful Response
|
||||||
|
* @throws ApiError
|
||||||
|
*/
|
||||||
|
public static readLibrary(): CancelablePromise<FlowsReadLibraryResponse> {
|
||||||
|
return __request(OpenAPI, {
|
||||||
|
method: 'GET',
|
||||||
|
url: '/api/v1/flows/library'
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Delete Shared Node
|
||||||
|
* Remove a shared source, as long as no flow still runs it.
|
||||||
|
* @param data The data for the request.
|
||||||
|
* @param data.libName
|
||||||
|
* @returns Message Successful Response
|
||||||
|
* @throws ApiError
|
||||||
|
*/
|
||||||
|
public static deleteSharedNode(data: FlowsDeleteSharedNodeData): CancelablePromise<FlowsDeleteSharedNodeResponse> {
|
||||||
|
return __request(OpenAPI, {
|
||||||
|
method: 'DELETE',
|
||||||
|
url: '/api/v1/flows/library/{lib_name}',
|
||||||
|
path: {
|
||||||
|
lib_name: data.libName
|
||||||
|
},
|
||||||
|
errors: {
|
||||||
|
422: 'Validation Error'
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Read Flow
|
* Read Flow
|
||||||
* Read one flow, with the state of its nodes.
|
* Read one flow, with the state of its nodes.
|
||||||
@@ -202,6 +236,10 @@ export class FlowsService {
|
|||||||
* The answer comes from compiling the code rather than from the running
|
* The answer comes from compiling the code rather than from the running
|
||||||
* pipeline: a draft is not deployed, and compiling is both faster and more
|
* pipeline: a draft is not deployed, and compiling is both faster and more
|
||||||
* precise about what the author just typed.
|
* precise about what the author just typed.
|
||||||
|
*
|
||||||
|
* A shared node writes to the library, so the fix reaches every flow using
|
||||||
|
* it — and that one is live immediately rather than waiting for a publish,
|
||||||
|
* because the copy is not any single flow's to hold back.
|
||||||
* @param data The data for the request.
|
* @param data The data for the request.
|
||||||
* @param data.name
|
* @param data.name
|
||||||
* @param data.nodeId
|
* @param data.nodeId
|
||||||
@@ -225,6 +263,55 @@ export class FlowsService {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Share Node
|
||||||
|
* Move this node's code into the library so other flows can run it too.
|
||||||
|
* @param data The data for the request.
|
||||||
|
* @param data.name
|
||||||
|
* @param data.nodeId
|
||||||
|
* @param data.requestBody
|
||||||
|
* @returns FlowDetail Successful Response
|
||||||
|
* @throws ApiError
|
||||||
|
*/
|
||||||
|
public static shareNode(data: FlowsShareNodeData): CancelablePromise<FlowsShareNodeResponse> {
|
||||||
|
return __request(OpenAPI, {
|
||||||
|
method: 'POST',
|
||||||
|
url: '/api/v1/flows/{name}/nodes/{node_id}/share',
|
||||||
|
path: {
|
||||||
|
name: data.name,
|
||||||
|
node_id: data.nodeId
|
||||||
|
},
|
||||||
|
body: data.requestBody,
|
||||||
|
mediaType: 'application/json',
|
||||||
|
errors: {
|
||||||
|
422: 'Validation Error'
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Unshare Node
|
||||||
|
* Take a private copy of the shared code back into this flow.
|
||||||
|
* @param data The data for the request.
|
||||||
|
* @param data.name
|
||||||
|
* @param data.nodeId
|
||||||
|
* @returns FlowDetail Successful Response
|
||||||
|
* @throws ApiError
|
||||||
|
*/
|
||||||
|
public static unshareNode(data: FlowsUnshareNodeData): CancelablePromise<FlowsUnshareNodeResponse> {
|
||||||
|
return __request(OpenAPI, {
|
||||||
|
method: 'POST',
|
||||||
|
url: '/api/v1/flows/{name}/nodes/{node_id}/unshare',
|
||||||
|
path: {
|
||||||
|
name: data.name,
|
||||||
|
node_id: data.nodeId
|
||||||
|
},
|
||||||
|
errors: {
|
||||||
|
422: 'Validation Error'
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Start Flow
|
* Start Flow
|
||||||
* Let the engine run this flow again.
|
* Let the engine run this flow again.
|
||||||
|
|||||||
@@ -105,6 +105,14 @@ export type HTTPValidationError = {
|
|||||||
detail?: Array<ValidationError>;
|
detail?: Array<ValidationError>;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A node source shared across flows, and who is using it.
|
||||||
|
*/
|
||||||
|
export type LibraryNode = {
|
||||||
|
name: string;
|
||||||
|
used_by?: Array<(string)>;
|
||||||
|
};
|
||||||
|
|
||||||
export type Message = {
|
export type Message = {
|
||||||
message: string;
|
message: string;
|
||||||
};
|
};
|
||||||
@@ -129,11 +137,16 @@ export type MessageHistory = {
|
|||||||
* :param port: The identifier the node function sees. Defaults to the last
|
* :param port: The identifier the node function sees. Defaults to the last
|
||||||
* segment of ``name``, so unqualified flows read naturally.
|
* segment of ``name``, so unqualified flows read naturally.
|
||||||
* :param dtype: Payload type, validated on every message that passes through.
|
* :param dtype: Payload type, validated on every message that passes through.
|
||||||
|
* :param interval: Deliver at most every this many seconds; 0 is every time.
|
||||||
|
* On an output it holds back publishing, on an input it holds back waking
|
||||||
|
* the node. The value is never lost — state keeps the latest — only the
|
||||||
|
* delivery is skipped.
|
||||||
*/
|
*/
|
||||||
export type MessageSpec = {
|
export type MessageSpec = {
|
||||||
name?: string;
|
name?: string;
|
||||||
port?: string;
|
port?: string;
|
||||||
dtype?: DType;
|
dtype?: DType;
|
||||||
|
interval?: number;
|
||||||
};
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -162,6 +175,7 @@ export type NodeDef_Input = {
|
|||||||
};
|
};
|
||||||
requires?: Array<MessageSpec>;
|
requires?: Array<MessageSpec>;
|
||||||
provides?: Array<MessageSpec>;
|
provides?: Array<MessageSpec>;
|
||||||
|
source_ref?: (string | null);
|
||||||
};
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -177,6 +191,7 @@ export type NodeDef_Output = {
|
|||||||
};
|
};
|
||||||
requires?: Array<MessageSpec>;
|
requires?: Array<MessageSpec>;
|
||||||
provides?: Array<MessageSpec>;
|
provides?: Array<MessageSpec>;
|
||||||
|
source_ref?: (string | null);
|
||||||
};
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -187,14 +202,18 @@ export type NodeSource = {
|
|||||||
};
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Whether a node loaded, and why not.
|
* Whether a node loaded, and how its connection is doing.
|
||||||
*/
|
*/
|
||||||
export type NodeStatusPublic = {
|
export type NodeStatusPublic = {
|
||||||
id: string;
|
id: string;
|
||||||
status?: string;
|
status?: string;
|
||||||
error?: (string | null);
|
error?: (string | null);
|
||||||
|
health?: 'ok' | 'degraded' | 'down';
|
||||||
|
health_detail?: (string | null);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export type health = 'ok' | 'degraded' | 'down';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A node type the editor can offer, with its parameter schema.
|
* A node type the editor can offer, with its parameter schema.
|
||||||
*/
|
*/
|
||||||
@@ -206,6 +225,7 @@ export type NodeTypeInfo = {
|
|||||||
[key: string]: unknown;
|
[key: string]: unknown;
|
||||||
};
|
};
|
||||||
has_source?: boolean;
|
has_source?: boolean;
|
||||||
|
plugin?: (string | null);
|
||||||
};
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -246,6 +266,10 @@ export type SecretValue = {
|
|||||||
value: string;
|
value: string;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export type ShareRequest = {
|
||||||
|
lib_name: string;
|
||||||
|
};
|
||||||
|
|
||||||
export type Token = {
|
export type Token = {
|
||||||
access_token: string;
|
access_token: string;
|
||||||
token_type?: string;
|
token_type?: string;
|
||||||
@@ -332,6 +356,14 @@ export type FlowsReadFlowsResponse = (FlowsPublic);
|
|||||||
|
|
||||||
export type FlowsReadNodeTypesResponse = (Array<NodeTypeInfo>);
|
export type FlowsReadNodeTypesResponse = (Array<NodeTypeInfo>);
|
||||||
|
|
||||||
|
export type FlowsReadLibraryResponse = (Array<LibraryNode>);
|
||||||
|
|
||||||
|
export type FlowsDeleteSharedNodeData = {
|
||||||
|
libName: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type FlowsDeleteSharedNodeResponse = (Message);
|
||||||
|
|
||||||
export type FlowsReadFlowData = {
|
export type FlowsReadFlowData = {
|
||||||
name: string;
|
name: string;
|
||||||
};
|
};
|
||||||
@@ -386,6 +418,21 @@ export type FlowsSaveNodeSourceData = {
|
|||||||
|
|
||||||
export type FlowsSaveNodeSourceResponse = (NodeStatusPublic);
|
export type FlowsSaveNodeSourceResponse = (NodeStatusPublic);
|
||||||
|
|
||||||
|
export type FlowsShareNodeData = {
|
||||||
|
name: string;
|
||||||
|
nodeId: string;
|
||||||
|
requestBody: ShareRequest;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type FlowsShareNodeResponse = (FlowDetail);
|
||||||
|
|
||||||
|
export type FlowsUnshareNodeData = {
|
||||||
|
name: string;
|
||||||
|
nodeId: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type FlowsUnshareNodeResponse = (FlowDetail);
|
||||||
|
|
||||||
export type FlowsStartFlowData = {
|
export type FlowsStartFlowData = {
|
||||||
name: string;
|
name: string;
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import { useQuery } from "@tanstack/react-query"
|
||||||
import { useNavigate } from "@tanstack/react-router"
|
import { useNavigate } from "@tanstack/react-router"
|
||||||
import { useEffect } from "react"
|
import { useEffect } from "react"
|
||||||
|
|
||||||
@@ -10,6 +11,7 @@ import {
|
|||||||
CommandItem,
|
CommandItem,
|
||||||
CommandList,
|
CommandList,
|
||||||
} from "@/components/ui/command"
|
} from "@/components/ui/command"
|
||||||
|
import { libraryQueryOptions } from "./queries"
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* ⌘K: add a node, jump to another flow, or run the current one, without
|
* ⌘K: add a node, jump to another flow, or run the current one, without
|
||||||
@@ -21,6 +23,7 @@ export function CommandPalette({
|
|||||||
nodeTypes,
|
nodeTypes,
|
||||||
flows,
|
flows,
|
||||||
onAddNode,
|
onAddNode,
|
||||||
|
onAddSharedNode,
|
||||||
onRun,
|
onRun,
|
||||||
}: {
|
}: {
|
||||||
open: boolean
|
open: boolean
|
||||||
@@ -28,9 +31,14 @@ export function CommandPalette({
|
|||||||
nodeTypes: NodeTypeInfo[]
|
nodeTypes: NodeTypeInfo[]
|
||||||
flows: FlowSummary[]
|
flows: FlowSummary[]
|
||||||
onAddNode: (type: string) => void
|
onAddNode: (type: string) => void
|
||||||
|
onAddSharedNode: (libName: string) => void
|
||||||
onRun: () => void
|
onRun: () => void
|
||||||
}) {
|
}) {
|
||||||
const navigate = useNavigate()
|
const navigate = useNavigate()
|
||||||
|
const { data: libraryData } = useQuery({
|
||||||
|
...libraryQueryOptions(),
|
||||||
|
enabled: open,
|
||||||
|
})
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
const onKey = (event: KeyboardEvent) => {
|
const onKey = (event: KeyboardEvent) => {
|
||||||
@@ -48,6 +56,8 @@ export function CommandPalette({
|
|||||||
// dialog outright is deterministic; the palette does not need to fade out.
|
// dialog outright is deterministic; the palette does not need to fade out.
|
||||||
if (!open) return null
|
if (!open) return null
|
||||||
|
|
||||||
|
const library = libraryData ?? []
|
||||||
|
|
||||||
const close = (action: () => void) => {
|
const close = (action: () => void) => {
|
||||||
onOpenChange(false)
|
onOpenChange(false)
|
||||||
action()
|
action()
|
||||||
@@ -81,6 +91,28 @@ export function CommandPalette({
|
|||||||
))}
|
))}
|
||||||
</CommandGroup>
|
</CommandGroup>
|
||||||
|
|
||||||
|
{library.length > 0 ? (
|
||||||
|
<CommandGroup heading="Reusable nodes">
|
||||||
|
{library.map((entry) => (
|
||||||
|
<CommandItem
|
||||||
|
key={entry.name}
|
||||||
|
value={`shared ${entry.name}`}
|
||||||
|
onSelect={() => close(() => onAddSharedNode(entry.name))}
|
||||||
|
>
|
||||||
|
<span className="flex flex-col">
|
||||||
|
<span className="font-mono">{entry.name}</span>
|
||||||
|
<span className="text-xs text-muted-foreground">
|
||||||
|
{(entry.used_by ?? []).length === 1
|
||||||
|
? "Used once"
|
||||||
|
: `Used ${(entry.used_by ?? []).length} times`}
|
||||||
|
, code shared with every flow using it
|
||||||
|
</span>
|
||||||
|
</span>
|
||||||
|
</CommandItem>
|
||||||
|
))}
|
||||||
|
</CommandGroup>
|
||||||
|
) : null}
|
||||||
|
|
||||||
{flows.length > 0 ? (
|
{flows.length > 0 ? (
|
||||||
<CommandGroup heading="Flows">
|
<CommandGroup heading="Flows">
|
||||||
{flows.map((flow) => (
|
{flows.map((flow) => (
|
||||||
|
|||||||
@@ -453,8 +453,8 @@ function FlowEditorInner({
|
|||||||
})
|
})
|
||||||
|
|
||||||
const addNode = useCallback(
|
const addNode = useCallback(
|
||||||
(type: string) => {
|
(type: string, sourceRef?: string) => {
|
||||||
const id = uniqueNodeId(definitions, type)
|
const id = uniqueNodeId(definitions, sourceRef ?? type)
|
||||||
// Drop it where the user is looking, but never on top of another node.
|
// Drop it where the user is looking, but never on top of another node.
|
||||||
const position = freePosition(
|
const position = freePosition(
|
||||||
definitions,
|
definitions,
|
||||||
@@ -470,6 +470,9 @@ function FlowEditorInner({
|
|||||||
params: {},
|
params: {},
|
||||||
requires: [],
|
requires: [],
|
||||||
provides: [],
|
provides: [],
|
||||||
|
// A shared node brings the code; the ports and settings are this
|
||||||
|
// flow's own.
|
||||||
|
...(sourceRef ? { source_ref: sourceRef } : {}),
|
||||||
}
|
}
|
||||||
const nextDefinitions = [...definitions, node]
|
const nextDefinitions = [...definitions, node]
|
||||||
const nextCanvas = [
|
const nextCanvas = [
|
||||||
@@ -809,6 +812,9 @@ function FlowEditorInner({
|
|||||||
onSaveSource={(code) => {
|
onSaveSource={(code) => {
|
||||||
if (selected) sourceMutation.mutate({ nodeId: selected.id, code })
|
if (selected) sourceMutation.mutate({ nodeId: selected.id, code })
|
||||||
}}
|
}}
|
||||||
|
// Sharing rewrites the stored document, so the canvas takes the
|
||||||
|
// server's copy rather than keeping its own.
|
||||||
|
onShared={onReload}
|
||||||
onClose={() => {
|
onClose={() => {
|
||||||
flush()
|
flush()
|
||||||
setEditorExpanded(false)
|
setEditorExpanded(false)
|
||||||
@@ -830,6 +836,7 @@ function FlowEditorInner({
|
|||||||
nodeTypes={nodeTypeInfo ?? []}
|
nodeTypes={nodeTypeInfo ?? []}
|
||||||
flows={flows.data}
|
flows={flows.data}
|
||||||
onAddNode={addNode}
|
onAddNode={addNode}
|
||||||
|
onAddSharedNode={(libName) => addNode("python", libName)}
|
||||||
onRun={() => runMutation.mutate()}
|
onRun={() => runMutation.mutate()}
|
||||||
/>
|
/>
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,14 @@
|
|||||||
import { useQuery } from "@tanstack/react-query"
|
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"
|
||||||
import { Maximize2, Minimize2, X } from "lucide-react"
|
import { Maximize2, Minimize2, X } from "lucide-react"
|
||||||
import { lazy, Suspense, useEffect, useRef, useState } from "react"
|
import { lazy, Suspense, useEffect, useRef, useState } from "react"
|
||||||
|
|
||||||
import type { DType, MessageSpec, NodeDef_Input, NodeTypeInfo } from "@/client"
|
import {
|
||||||
|
type DType,
|
||||||
|
FlowsService,
|
||||||
|
type MessageSpec,
|
||||||
|
type NodeDef_Input,
|
||||||
|
type NodeTypeInfo,
|
||||||
|
} from "@/client"
|
||||||
import { Button } from "@/components/ui/button"
|
import { Button } from "@/components/ui/button"
|
||||||
import {
|
import {
|
||||||
Command,
|
Command,
|
||||||
@@ -22,15 +28,24 @@ import {
|
|||||||
SelectValue,
|
SelectValue,
|
||||||
} from "@/components/ui/select"
|
} from "@/components/ui/select"
|
||||||
import { Switch } from "@/components/ui/switch"
|
import { Switch } from "@/components/ui/switch"
|
||||||
|
import useCustomToast from "@/hooks/useCustomToast"
|
||||||
import { cn } from "@/lib/utils"
|
import { cn } from "@/lib/utils"
|
||||||
import { MessageSparkline } from "./MessageSparkline"
|
import { MessageSparkline } from "./MessageSparkline"
|
||||||
import { nodeSourceQueryOptions } from "./queries"
|
import {
|
||||||
|
flowKeys,
|
||||||
|
libraryQueryOptions,
|
||||||
|
nodeSourceQueryOptions,
|
||||||
|
secretsQueryOptions,
|
||||||
|
} from "./queries"
|
||||||
import { PANEL_SECTION, SidePanel } from "./SidePanel"
|
import { PANEL_SECTION, SidePanel } from "./SidePanel"
|
||||||
|
|
||||||
const NodeEditor = lazy(() => import("./NodeEditor"))
|
const NodeEditor = lazy(() => import("./NodeEditor"))
|
||||||
|
|
||||||
const DTYPES: DType[] = ["float", "int", "str", "bool", "json"]
|
const DTYPES: DType[] = ["float", "int", "str", "bool", "json"]
|
||||||
|
|
||||||
|
/** Radix selects cannot hold an empty value, so "no secret" needs a name. */
|
||||||
|
const NO_SECRET = "__none__"
|
||||||
|
|
||||||
const SECTION = PANEL_SECTION
|
const SECTION = PANEL_SECTION
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -208,6 +223,19 @@ function PortList({
|
|||||||
))}
|
))}
|
||||||
</SelectContent>
|
</SelectContent>
|
||||||
</Select>
|
</Select>
|
||||||
|
<Input
|
||||||
|
type="number"
|
||||||
|
min={0}
|
||||||
|
step="any"
|
||||||
|
value={spec.interval ? String(spec.interval) : ""}
|
||||||
|
placeholder="∞"
|
||||||
|
aria-label="Deliver at most every n seconds"
|
||||||
|
title="Deliver at most every n seconds; empty is every time"
|
||||||
|
className="h-8 w-16 text-sm"
|
||||||
|
onChange={(event) =>
|
||||||
|
update(index, { interval: Number(event.target.value) || 0 })
|
||||||
|
}
|
||||||
|
/>
|
||||||
<Button
|
<Button
|
||||||
variant="ghost"
|
variant="ghost"
|
||||||
size="icon-sm"
|
size="icon-sm"
|
||||||
@@ -235,6 +263,8 @@ function ParamsForm({
|
|||||||
params: Record<string, unknown>
|
params: Record<string, unknown>
|
||||||
onChange: (next: Record<string, unknown>) => void
|
onChange: (next: Record<string, unknown>) => void
|
||||||
}) {
|
}) {
|
||||||
|
const { data: secretList } = useQuery(secretsQueryOptions())
|
||||||
|
const secrets = secretList?.data ?? []
|
||||||
const properties = (schema?.properties ?? {}) as Record<
|
const properties = (schema?.properties ?? {}) as Record<
|
||||||
string,
|
string,
|
||||||
{
|
{
|
||||||
@@ -242,6 +272,7 @@ function ParamsForm({
|
|||||||
title?: string
|
title?: string
|
||||||
description?: string
|
description?: string
|
||||||
default?: unknown
|
default?: unknown
|
||||||
|
"x-secret"?: boolean
|
||||||
}
|
}
|
||||||
>
|
>
|
||||||
const entries = Object.entries(properties)
|
const entries = Object.entries(properties)
|
||||||
@@ -272,6 +303,41 @@ function ParamsForm({
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A credential is stored once and referenced, so it never ends up in
|
||||||
|
// flow.json where the whole team can read it.
|
||||||
|
if (property["x-secret"]) {
|
||||||
|
const reference = (params[key] ?? null) as { $secret?: string } | null
|
||||||
|
return (
|
||||||
|
<div key={key} className="grid gap-1.5">
|
||||||
|
<Label className="text-sm font-normal">{label}</Label>
|
||||||
|
<Select
|
||||||
|
value={reference?.$secret ?? ""}
|
||||||
|
onValueChange={(name) =>
|
||||||
|
set(key, name === NO_SECRET ? null : { $secret: name })
|
||||||
|
}
|
||||||
|
>
|
||||||
|
<SelectTrigger className="!h-8 text-sm" aria-label={label}>
|
||||||
|
<SelectValue placeholder="Pick a stored secret" />
|
||||||
|
</SelectTrigger>
|
||||||
|
<SelectContent>
|
||||||
|
<SelectItem value={NO_SECRET}>None</SelectItem>
|
||||||
|
{secrets.map((secret) => (
|
||||||
|
<SelectItem key={secret} value={secret}>
|
||||||
|
{secret}
|
||||||
|
</SelectItem>
|
||||||
|
))}
|
||||||
|
</SelectContent>
|
||||||
|
</Select>
|
||||||
|
<p className="text-xs text-muted-foreground">
|
||||||
|
{secrets.length
|
||||||
|
? property.description ||
|
||||||
|
"Stored in the secrets store, never in the flow file."
|
||||||
|
: "No secrets stored yet."}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
if (property.type === "object" || property.type === "array") {
|
if (property.type === "object" || property.type === "array") {
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
@@ -307,6 +373,108 @@ function ParamsForm({
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Sharing a node moves its code to the library, where other flows can point at
|
||||||
|
* it. Each flow keeps its own ports and settings; only the code is common, so
|
||||||
|
* one fix reaches all of them.
|
||||||
|
*/
|
||||||
|
function SharingSection({
|
||||||
|
flow,
|
||||||
|
node,
|
||||||
|
onShared,
|
||||||
|
}: {
|
||||||
|
flow: string
|
||||||
|
node: NodeDef_Input
|
||||||
|
onShared: () => void
|
||||||
|
}) {
|
||||||
|
const queryClient = useQueryClient()
|
||||||
|
const { showErrorToast } = useCustomToast()
|
||||||
|
const { data: library } = useQuery(libraryQueryOptions())
|
||||||
|
const [name, setName] = useState("")
|
||||||
|
|
||||||
|
const shared = node.source_ref
|
||||||
|
const usages = library?.find((entry) => entry.name === shared)?.used_by ?? []
|
||||||
|
|
||||||
|
const done = () => {
|
||||||
|
queryClient.invalidateQueries({ queryKey: flowKeys.library })
|
||||||
|
queryClient.invalidateQueries({ queryKey: flowKeys.detail(flow) })
|
||||||
|
onShared()
|
||||||
|
}
|
||||||
|
|
||||||
|
const share = useMutation({
|
||||||
|
mutationFn: (libName: string) =>
|
||||||
|
FlowsService.shareNode({
|
||||||
|
name: flow,
|
||||||
|
nodeId: node.id,
|
||||||
|
requestBody: { lib_name: libName },
|
||||||
|
}),
|
||||||
|
onSuccess: done,
|
||||||
|
onError: () =>
|
||||||
|
showErrorToast("That name is taken, or is not a valid name."),
|
||||||
|
})
|
||||||
|
|
||||||
|
const unshare = useMutation({
|
||||||
|
mutationFn: () => FlowsService.unshareNode({ name: flow, nodeId: node.id }),
|
||||||
|
onSuccess: done,
|
||||||
|
onError: () => showErrorToast("The node could not be unshared."),
|
||||||
|
})
|
||||||
|
|
||||||
|
if (shared) {
|
||||||
|
return (
|
||||||
|
<div className="grid gap-2">
|
||||||
|
<span className={SECTION}>Shared</span>
|
||||||
|
<p className="text-sm text-muted-foreground">
|
||||||
|
Runs <span className="font-mono">{shared}</span> from the library
|
||||||
|
{usages.length > 1
|
||||||
|
? `, along with ${usages.length - 1} other node${
|
||||||
|
usages.length === 2 ? "" : "s"
|
||||||
|
}`
|
||||||
|
: ""}
|
||||||
|
. Editing the code here changes it everywhere.
|
||||||
|
</p>
|
||||||
|
<Button
|
||||||
|
variant="outline"
|
||||||
|
size="sm"
|
||||||
|
className="h-8 justify-self-start"
|
||||||
|
disabled={unshare.isPending}
|
||||||
|
onClick={() => unshare.mutate()}
|
||||||
|
data-testid="unshare-node"
|
||||||
|
>
|
||||||
|
Keep a private copy
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="grid gap-2">
|
||||||
|
<span className={SECTION}>Reuse</span>
|
||||||
|
<p className="text-sm text-muted-foreground">
|
||||||
|
Move this node's code to the library so other flows can run it too.
|
||||||
|
</p>
|
||||||
|
<div className="flex items-center gap-1.5">
|
||||||
|
<Input
|
||||||
|
value={name}
|
||||||
|
placeholder="read_temperature"
|
||||||
|
aria-label="Shared name"
|
||||||
|
autoComplete="off"
|
||||||
|
className="h-8 flex-1 font-mono text-sm"
|
||||||
|
onChange={(event) => setName(event.target.value)}
|
||||||
|
/>
|
||||||
|
<Button
|
||||||
|
size="sm"
|
||||||
|
className="h-8"
|
||||||
|
disabled={!name || share.isPending}
|
||||||
|
onClick={() => share.mutate(name)}
|
||||||
|
data-testid="share-node"
|
||||||
|
>
|
||||||
|
Share
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
function PanelBody({
|
function PanelBody({
|
||||||
node,
|
node,
|
||||||
flow,
|
flow,
|
||||||
@@ -316,6 +484,7 @@ function PanelBody({
|
|||||||
onChange,
|
onChange,
|
||||||
onRenameMessage,
|
onRenameMessage,
|
||||||
onSaveSource,
|
onSaveSource,
|
||||||
|
onShared,
|
||||||
onToggleExpand,
|
onToggleExpand,
|
||||||
}: {
|
}: {
|
||||||
node: NodeDef_Input
|
node: NodeDef_Input
|
||||||
@@ -326,6 +495,7 @@ function PanelBody({
|
|||||||
onChange: (next: NodeDef_Input) => void
|
onChange: (next: NodeDef_Input) => void
|
||||||
onRenameMessage: (previous: string, next: string) => void
|
onRenameMessage: (previous: string, next: string) => void
|
||||||
onSaveSource: (code: string) => void
|
onSaveSource: (code: string) => void
|
||||||
|
onShared: () => void
|
||||||
onToggleExpand: () => void
|
onToggleExpand: () => void
|
||||||
}) {
|
}) {
|
||||||
const hasSource = nodeType?.has_source ?? node.type === "python"
|
const hasSource = nodeType?.has_source ?? node.type === "python"
|
||||||
@@ -387,12 +557,17 @@ function PanelBody({
|
|||||||
params={node.params ?? {}}
|
params={node.params ?? {}}
|
||||||
onChange={(params) => onChange({ ...node, params })}
|
onChange={(params) => onChange({ ...node, params })}
|
||||||
/>
|
/>
|
||||||
|
{hasSource ? (
|
||||||
|
<SharingSection flow={flow} node={node} onShared={onShared} />
|
||||||
|
) : null}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{hasSource ? (
|
{hasSource ? (
|
||||||
<div className="flex min-h-[280px] flex-1 flex-col gap-2 px-4 pb-4">
|
<div className="flex min-h-[280px] flex-1 flex-col gap-2 px-4 pb-4">
|
||||||
<div className="flex items-center justify-between">
|
<div className="flex items-center justify-between">
|
||||||
<span className={SECTION}>Code</span>
|
<span className={SECTION}>
|
||||||
|
{node.source_ref ? `Shared code · ${node.source_ref}` : "Code"}
|
||||||
|
</span>
|
||||||
<Button
|
<Button
|
||||||
variant="ghost"
|
variant="ghost"
|
||||||
size="icon-sm"
|
size="icon-sm"
|
||||||
@@ -440,6 +615,7 @@ export function NodePanel({
|
|||||||
onChange,
|
onChange,
|
||||||
onRenameMessage,
|
onRenameMessage,
|
||||||
onSaveSource,
|
onSaveSource,
|
||||||
|
onShared,
|
||||||
onToggleExpand,
|
onToggleExpand,
|
||||||
onClose,
|
onClose,
|
||||||
onDelete,
|
onDelete,
|
||||||
@@ -452,6 +628,7 @@ export function NodePanel({
|
|||||||
onChange: (next: NodeDef_Input) => void
|
onChange: (next: NodeDef_Input) => void
|
||||||
onRenameMessage: (previous: string, next: string) => void
|
onRenameMessage: (previous: string, next: string) => void
|
||||||
onSaveSource: (code: string) => void
|
onSaveSource: (code: string) => void
|
||||||
|
onShared: () => void
|
||||||
onToggleExpand: () => void
|
onToggleExpand: () => void
|
||||||
onClose: () => void
|
onClose: () => void
|
||||||
onDelete: () => void
|
onDelete: () => void
|
||||||
@@ -506,6 +683,7 @@ export function NodePanel({
|
|||||||
onChange={onChange}
|
onChange={onChange}
|
||||||
onRenameMessage={onRenameMessage}
|
onRenameMessage={onRenameMessage}
|
||||||
onSaveSource={onSaveSource}
|
onSaveSource={onSaveSource}
|
||||||
|
onShared={onShared}
|
||||||
onToggleExpand={onToggleExpand}
|
onToggleExpand={onToggleExpand}
|
||||||
/>
|
/>
|
||||||
) : null}
|
) : null}
|
||||||
|
|||||||
@@ -5,7 +5,12 @@ import {
|
|||||||
} from "@tanstack/react-query"
|
} from "@tanstack/react-query"
|
||||||
import { useCallback, useEffect, useRef, useState } from "react"
|
import { useCallback, useEffect, useRef, useState } from "react"
|
||||||
|
|
||||||
import { ApiError, type FlowDef_Input, FlowsService } from "@/client"
|
import {
|
||||||
|
ApiError,
|
||||||
|
type FlowDef_Input,
|
||||||
|
FlowsService,
|
||||||
|
SecretsService,
|
||||||
|
} from "@/client"
|
||||||
|
|
||||||
export const flowKeys = {
|
export const flowKeys = {
|
||||||
all: ["flows"] as const,
|
all: ["flows"] as const,
|
||||||
@@ -15,6 +20,7 @@ export const flowKeys = {
|
|||||||
history: (name: string, message: string) =>
|
history: (name: string, message: string) =>
|
||||||
["flows", name, "history", message] as const,
|
["flows", name, "history", message] as const,
|
||||||
nodeTypes: ["flows", "node-types"] as const,
|
nodeTypes: ["flows", "node-types"] as const,
|
||||||
|
library: ["flows", "library"] as const,
|
||||||
}
|
}
|
||||||
|
|
||||||
export const flowsQueryOptions = () => ({
|
export const flowsQueryOptions = () => ({
|
||||||
@@ -27,6 +33,18 @@ export const flowQueryOptions = (name: string) => ({
|
|||||||
queryFn: () => FlowsService.readFlow({ name }),
|
queryFn: () => FlowsService.readFlow({ name }),
|
||||||
})
|
})
|
||||||
|
|
||||||
|
/** Names of the stored secrets, for pointing a credential parameter at one. */
|
||||||
|
export const secretsQueryOptions = () => ({
|
||||||
|
queryKey: ["secrets"] as const,
|
||||||
|
queryFn: () => SecretsService.readSecrets(),
|
||||||
|
})
|
||||||
|
|
||||||
|
/** Node sources shared across flows, with the nodes using each. */
|
||||||
|
export const libraryQueryOptions = () => ({
|
||||||
|
queryKey: flowKeys.library,
|
||||||
|
queryFn: () => FlowsService.readLibrary(),
|
||||||
|
})
|
||||||
|
|
||||||
export const nodeTypesQueryOptions = () => ({
|
export const nodeTypesQueryOptions = () => ({
|
||||||
queryKey: flowKeys.nodeTypes,
|
queryKey: flowKeys.nodeTypes,
|
||||||
queryFn: () => FlowsService.readNodeTypes(),
|
queryFn: () => FlowsService.readNodeTypes(),
|
||||||
@@ -107,7 +125,14 @@ export function useAutosave(name: string): {
|
|||||||
mutationFn: (definition: FlowDef_Input) =>
|
mutationFn: (definition: FlowDef_Input) =>
|
||||||
FlowsService.saveFlow({
|
FlowsService.saveFlow({
|
||||||
name,
|
name,
|
||||||
requestBody: { ...definition, version: version.current ?? undefined },
|
// The ref holds the freshest version this client has seen; before the
|
||||||
|
// first save that is the one the document was loaded with. Sending
|
||||||
|
// `undefined` would drop the field and let the server read the default,
|
||||||
|
// which conflicts with every flow saved more than once.
|
||||||
|
requestBody: {
|
||||||
|
...definition,
|
||||||
|
version: version.current ?? definition.version ?? 1,
|
||||||
|
},
|
||||||
}),
|
}),
|
||||||
onMutate: (definition) => {
|
onMutate: (definition) => {
|
||||||
inFlight.current = true
|
inFlight.current = true
|
||||||
|
|||||||
Reference in New Issue
Block a user