This commit is contained in:
2026-10-13 10:02:20 +02:00
parent a93f865e91
commit 80e595c02e
24 changed files with 2210 additions and 4310 deletions
+133 -208
View File
@@ -1,180 +1,130 @@
# The connector contract
A connector is the device-facing node class. It talks to something outside the
engine (a device, a service, a protocol) and publishes what it finds as
ordinary typed messages, so the rest of a flow cannot tell the difference
between a reading from a heat pump and one from a `print` statement.
This document is normative: everything here is what the engine relies on and
what a connector may rely on in return. The authoring walkthrough is in
[Writing a connector](../code/connectors.md).
Connectors integrate devices and services through typed messages.
This reference covers the contract the engine supports; start with
[Writing a connector](../code/connectors.md) for a working package.
## Contract version
```python
from fluksio.flow.connector import CONTRACT_VERSION, ConnectorNode
class MyConnector(ConnectorNode):
contract = CONTRACT_VERSION
title = "My connector"
description = "Reads a device."
```
`contract` must be declared **on the connector class itself**. The loader reads
it from the class `__dict__` rather than through inheritance, so a connector
written against an older base cannot pass itself off as current. A mismatch is
logged and the connector is skipped; nothing else in the engine changes.
Declare `contract` on the class itself: inheritance does not count.
The current contract version is `1`. A mismatch is logged and the connector
is skipped. Breaking changes bump the version; compatible additions do not.
The version is bumped only when a change would break connectors written against
the old surface. Additions that leave existing connectors working do not bump
it.
`title` defaults to the entry point name; `description` may be empty.
## Packaging and discovery
A connector is an ordinary Python package. It advertises its node class through
an entry point in the `fluksio.node_types` group:
Register the class in your package's `pyproject.toml`:
```toml
[project]
name = "fluksio-connector-skeleton"
version = "0.1.0"
[project.entry-points."fluksio.node_types"]
random_sensor = "fluksio_connector_skeleton:RandomSensor"
my_device = "fluksio_connector_mydevice:MyConnector"
```
- The **entry point name** is the node type as it appears on the canvas and in
a stored flow. It must not collide with a built-in type (`python`, `mqtt`,
`http`, `influxdb`, `delay`, `mlp`); a collision is logged and skipped.
- The **package name and version** are the manifest. There is no second
metadata file to keep in step, and the editor shows the provenance
(`fluksio-connector-skeleton 0.1.0`) on the node type.
The entry point name is the stored node type. Names already registered,
including built-in types, are skipped. The editor shows the package name
and version from its installed metadata.
Discovery happens once, when the engine starts. A connector's code is imported,
and Python does not re-import a changed module, so installing or upgrading one
means restarting the engine.
## Declaring what a connector is
```python
class RandomSensor(ConnectorNode):
contract = CONTRACT_VERSION
title = "Random sensor" # shown in the add-node palette
description = "Emits a random reading."
```
`title` falls back to the entry point name; `description` may be empty.
Discovery runs at engine startup. Install connectors in the engine's
environment and restart after installation or code changes.
## Parameters
Settings are a pydantic model, and the editor builds a form from its JSON
schema. Subclass the base so `poll_interval` stays available:
Subclass `ConnectorNode.Params` to keep `poll_interval` and add your settings:
```python
from pydantic import Field
class Params(ConnectorNode.Params):
host: str = "localhost"
api_key: str | None = Field(default=None, json_schema_extra={"x-secret": True})
api_key: str | None = Field(
default=None, json_schema_extra={"x-secret": True}
)
```
- Booleans render as switches, numbers and strings as inputs. Objects and
arrays are not rendered, so keep settings flat.
- A field carrying **`x-secret`** renders as a picker over the stored secrets
and writes a reference, `{"$secret": "name"}`, rather than the value. The
engine resolves it when the node is built, so a credential never lands in
`flow.json`. Always mark credentials this way.
- The parsed model is available as `self.config`.
The node panel builds a form from this model's JSON schema: switches for
booleans, inputs for numbers and strings, and JSON fields for objects and
arrays. Parsed settings are available as `self.config`.
Mark credentials with `x-secret`. The editor stores a reference such as
`{"$secret": "device-key"}`, and the engine resolves it while building the node.
## Ports and messages
Ports are declared by whoever places the node, not by the connector, and they
are `MessageSpec`s like any other node's: a message name, a `DType`, and an
optional `interval`.
The flow author declares ports as `MessageSpec` values. Each has a message
name, a `DType`, and optional delivery settings.
- `poll()` returns a dict **keyed by output port**, and the base class maps it
onto message names; `write()` receives a dict **keyed by input port** the
same way.
- Every value is checked against the port's declared `DType` on the way out. A
wrong type is an error, not a hint.
- `interval` on a port is enforced by the engine, not by the connector: an
output port publishes at most every *n* seconds, an input port wakes its node
at most that often. A connector should poll at the rate the device is
comfortable with and leave delivery rates to whoever wires it up.
- A key no port declares is an error, and it fails the whole reading rather
than the one value. A mistyped metric name is how a training curve goes
missing.
- `poll()` returns values keyed by **output port**.
- `write()` receives values keyed by **input port**.
- The base class maps local port names to message names and validates types.
- Unknown output keys fail the reading. Return only declared ports.
- Port `interval` limits are enforced by the engine; the connector controls
how often it reads the device.
### When the device decides what the ports are
A connector for a device whose readings vary by model (which components a
relay has, which entities were flashed onto a board) cannot know its ports in
advance, and returning everything the device reports would fail on the first
value nobody bound. Narrow the reading to the ports that were declared:
For devices with variable readings, filter to the configured outputs:
```python
declared = {spec.port for spec in self.output_ports if spec.name}
return {port: value for port, value in reading.items() if port in declared}
```
`spec.port` is the local, unqualified name and stays that way for the node's
whole life, so the set can be taken fresh each time. Say something in the log
when a declared port is not one the device has, once rather than once per poll: from
the canvas a renamed entity and a typo look the same, and both leave a port
silent forever.
Log missing device fields once so a typo does not leave a port silently empty.
## Polling
Set `poll_interval` and implement `poll()`; the base class runs the loop:
```python
async def poll(self) -> dict[str, Any] | None:
async def poll(self):
return {"reading": await self._read_device()}
```
- Return `None` when there is nothing new.
- **Only changed values are published.** A device polled every few seconds
usually says the same thing, and every publication wakes everything
downstream, so the loop compares against what it last published, meaning what it
actually published, so a publication that failed is retried next tick rather
than counting as said.
- Raising is not fatal: it is reported as a health problem and retried on the
next tick.
- The loop calls `inject`, which runs the graph, on a worker thread. `poll()`
itself runs on the event loop and must not block it.
Set `poll_interval` in seconds; `0` disables polling. Return `None` when
there is nothing new. The base loop publishes only changed values and retries
a failed reading or publication on the next tick. It reports connection health
around each poll.
A connector that is pushed to rather than polled leaves `poll_interval` at 0,
overrides `start`/`stop` to open and close its own subscription, and calls
`self.inject({...})` when something arrives.
`poll()` runs on the event loop. Use `asyncio.to_thread` for blocking reads.
The base loop already calls `inject` on a worker thread.
For pushed readings, leave `poll_interval=0`, manage the subscription in
`start()` and `stop()`, and call `self.inject({...})` when values arrive.
Offload `inject` too when calling it from the event loop.
## Writing
A connector that commands something rather than only reading it implements
`write()`. It is called when the node's inputs are satisfied, with the values
keyed by input port:
```python
def write(self, **ports: Any) -> dict[str, Any] | None:
def write(self, **ports):
self._device.set(ports["level"])
return None
```
- Settings are already parsed on `self.config`; `write()` is handed ports only.
- Return `None` unless the device answers something worth publishing, in which
case return it keyed by output port, exactly as `poll()` does.
- It runs on a worker thread rather than the event loop, but the thread is the
scheduler's, so a long block holds up the cascade. Queue the work if the
device is slow.
- A connector that only reads leaves it alone.
- `poll()` and `write()` are independent: a connector may have both, and a
node with inputs and no `poll_interval` never polls.
Settings stay on `self.config`; arguments contain input ports only.
Return `None`, or a dictionary keyed by output port if the device responds
with useful values.
A write is a command, not a value: set `idempotent = False` on the class so a
redelivery after a crash does not undo a newer command that already landed.
Writes run on a scheduler thread. Return promptly or queue slow work.
Polling and writing are independent; a connector may support both.
Set `idempotent = False` for commands that must not be redelivered after a
crash. Leave `write()` unchanged on a read-only connector.
## Devices whose readings are bytes
A camera frame or a recorded clip is far too big to be a message, so a
connector publishes a reference to it instead:
```python
async def poll(self) -> dict[str, Any] | None:
import asyncio
async def poll(self):
jpeg = await asyncio.to_thread(self._grab)
return {
"frame": self.save_artifact(
@@ -183,135 +133,110 @@ async def poll(self) -> dict[str, Any] | None:
}
```
`save_artifact` stores the bytes and returns what an `image`, `audio` or
`video` port carries: the media type has to match the port's type. It only
works once the node has started, since the store is the engine's and is handed
over then.
`save_artifact(data, name="", media_type="application/octet-stream",
volatile=False)` returns a reference for an artifact or media port.
Match the media type to an `image`, `audio`, or `video` port.
`volatile=True` is what a camera publishes with. The frame goes to a ring in
memory rather than the data volume and is pushed down the websocket to
whichever screens are drawing it, so a wall panel sees ten frames a second and
the SD card under it is never written to. Frames last until newer ones need the
room; leave the flag off for a reading somebody asked to keep.
The artifact store is available after `start()`. Use `volatile=True`
for disposable live frames: they go to a bounded memory ring and expire as
new frames arrive. Use durable storage for recordings.
Each reading is a new artifact, which the poll loop publishes because its
digest differs from the last. Set `poll_interval` to what somebody actually
wants to look at. Through a portal the bytes are fetched rather than pushed, so
a remote panel wants a frame every second or two; higher rates than that are
for the local network, and full-rate video still belongs on the camera's own
stream.
Local screens receive pushed live frames; portal screens fetch the bytes.
Choose a modest poll rate for remote viewing.
## Keeping state across restarts
A connection, a session or a device handle lives in the connector's own
attributes and goes with the process. The few things that must not, such as
the certificate a device presented on first contact or a token it paired with,
go in `self.store`:
Keep connections and device handles in node attributes. Store short strings
such as pairing tokens or pinned certificates in `self.store`:
```python
pem = await asyncio.to_thread(self.store.get, f"cert:{self.config.host}")
await asyncio.to_thread(self.store.set, f"cert:{self.config.host}", pem)
await asyncio.to_thread(self.store.delete, f"cert:{self.config.host}")
value = await asyncio.to_thread(self.store.get, "token:device")
await asyncio.to_thread(self.store.set, "token:device", "value")
await asyncio.to_thread(self.store.delete, "token:device")
```
- Keys and values are strings; `get` returns `None` for a key that is not
there, and `delete` returns whether there was one.
- The store belongs to the connector's **top-level package**, not to a node:
every node of the package sees the same keys, so two nodes on one bridge
share what was stored for it. Put the device in the key.
- It lives in the engine's database, so it survives a restart and needs
nothing `fluksio serve` does not have. It is available before `start`.
- Limits: a key is 1 to 200 characters without `/` or whitespace, a value at
most 64 kB, and a package at most 256 keys. Past any of them, `set` raises
`ValueError`. Bytes worth keeping belong in `save_artifact`.
- Every call is a database round trip, so on the event loop it goes through
`asyncio.to_thread`.
- Keys and values are strings. `get` returns `None` for a missing key;
`delete` returns whether it existed.
- All nodes from the connector's top-level package share the namespace.
Include a device identifier in each key.
- Storage is in the engine's database and is available before `start()`.
- Keys allow 1–200 characters, without `/` or whitespace. Values allow
64 KiB and each package allows 256 keys; exceeding a limit raises `ValueError`.
- Calls block on the database: use `asyncio.to_thread` on the event loop.
A superuser can see the keys (never the values) and clear them, all of them or
one at a time, under *Stored state* in a connector node's settings, or through
the API, which names the connector by any node type it installs:
Superusers can list keys and clear them under **Stored state** in the node
panel. Values are never returned. The equivalent API paths are:
```sh
curl -s $FLUKSIO/connectors/hue/state -H "Authorization: Bearer $TOKEN"
curl -X DELETE $FLUKSIO/connectors/hue/state/cert:10.0.0.2 -H "Authorization: Bearer $TOKEN"
curl -X DELETE $FLUKSIO/connectors/hue/state -H "Authorization: Bearer $TOKEN"
```
| Method | Path |
|---|---|
| `GET` | `/api/v1/connectors/{node_type}/state` |
| `DELETE` | `/api/v1/connectors/{node_type}/state/{key}` |
| `DELETE` | `/api/v1/connectors/{node_type}/state` |
So a connector has to cope with a key disappearing: clearing is how somebody
tells it to start over.
Handle missing keys: clearing one asks the connector to pair or learn again.
### Pinning a device's certificate
A device on the local network presents a certificate no system trusts and that
names a serial or a MAC rather than its address, so `verify=True` fails and
`verify=False` trusts anyone. `fluksio.flow.connector_tls` trusts the
certificate the device presented on first contact instead, kept in `self.store`
under `cert:<address>`:
`fluksio.flow.connector_tls` supports trusting a local device's certificate
on first contact:
```python
from fluksio.flow.connector_tls import context_for, pinned_certificate, raise_if_changed
import httpx
from fluksio.flow.connector_tls import (
context_for,
pinned_certificate,
raise_if_changed,
)
address = f"{self.config.host}:{self.config.port}"
pem = pinned_certificate(self.store, address, self.config.timeout) # blocking
pem = pinned_certificate(self.store, address, self.config.timeout)
try:
with httpx.Client(verify=context_for(pem)) as client:
...
response = client.get(f"https://{address}/status")
response.raise_for_status()
except httpx.ConnectError as exc:
raise_if_changed(exc, pem, address, self.config.timeout, "my_node_type")
raise_if_changed(exc, pem, address, self.config.timeout, "my_device")
raise
```
- First contact sends a TLS hello and nothing else; a device that does not
complete one raises `OSError` and nothing is pinned.
- Any other certificate is refused in the handshake, before a key or a command
is sent. `raise_if_changed` turns that refusal into `CertificateChanged`,
which names the new SHA-256 and the key to clear, under the node type given.
- The pinned certificate's dates are not checked: a pin trusts that exact
certificate, so one that expired, or an engine clock that is off, does not
lock the device out.
- Clearing the key accepts whatever the device presents next.
Run this blocking work off the event loop. First contact pins the certificate
under `cert:<address>`; a changed certificate is refused before sending a
credential or command. `raise_if_changed` reports `CertificateChanged`
with the fingerprint and key to clear.
## Lifecycle
Pins trust the exact certificate, including after its expiry. Clearing the
key accepts the certificate presented on the next connection.
## Lifecycle and health
```python
async def start(self, app=None) -> None: ...
async def stop(self, app=None) -> None: ...
async def start(self, app=None):
await super().start(app)
async def stop(self, app=None):
await super().stop(app)
```
- `start` is called when the connector's flow starts, and after every rebuild.
- `stop` is called before a rebuild and when the flow is stopped. **It must be
idempotent**: it is called whether or not `start` succeeded.
- Overriding either means calling `super()` if the polling loop is also wanted.
- The `app` argument is the FastAPI application, for the rare connector that
needs to mount a route. Most ignore it.
`start` runs when a flow starts or rebuilds. `stop` runs before rebuilding
or stopping, even after a failed start; make it idempotent. Call `super()`
when overriding these hooks if you want the base polling loop.
A stopped flow gets no `start` at all: that is what stopping it means.
The optional `app` is the FastAPI application, for connectors that mount
routes. Most connectors can ignore it.
## Health
Report connection changes if you manage your own subscription:
```python
self.report_health("ok")
self.report_health("degraded", "3 of 5 registers timed out")
self.report_health("down", str(exc))
self.report_health("degraded", "Some readings timed out")
self.report_health("down", "Connection lost")
```
Three values, `ok`, `degraded` and `down`, plus an optional detail string.
Reporting the same status twice is free, since only changes are published. A node
reporting `down` is named among its flow's issues and counted on the health
summary on Home; `degraded` means still working, and is not. The polling loop
already reports around `poll()`; a connector managing its own connection should
report when it connects and when it loses the connection.
Repeated statuses do not emit another event. `down` appears among flow issues
and on Home; `degraded` means the connection still works. Loading failures
are reported separately.
Health is about the connection, not about loading: a connector that fails to
load is reported separately and never runs.
## What a connector must not do
- **Hold state that belongs in the flow.** Logic nodes are stateless and run in
parallel; a connector may hold a connection, a session, or a device handle,
and keep what has to outlive a restart in `self.store`, but a value another
node needs belongs in a message.
- **Block the event loop.** `poll`, `start` and `stop` are async and run on the
loop. Use `asyncio.to_thread` for blocking I/O.
- **Read credentials from the environment.** They come through parameters, so
the person deploying a flow can change them without touching the host.
Keep shared flow values in messages, credentials in secret parameters, and
blocking I/O off the event loop.