docs
This commit is contained in:
@@ -1,180 +1,130 @@
|
||||
# The connector contract
|
||||
|
||||
A connector is the device-facing node class. It talks to something outside the
|
||||
engine (a device, a service, a protocol) and publishes what it finds as
|
||||
ordinary typed messages, so the rest of a flow cannot tell the difference
|
||||
between a reading from a heat pump and one from a `print` statement.
|
||||
|
||||
This document is normative: everything here is what the engine relies on and
|
||||
what a connector may rely on in return. The authoring walkthrough is in
|
||||
[Writing a connector](../code/connectors.md).
|
||||
Connectors integrate devices and services through typed messages.
|
||||
This reference covers the contract the engine supports; start with
|
||||
[Writing a connector](../code/connectors.md) for a working package.
|
||||
|
||||
## Contract version
|
||||
|
||||
```python
|
||||
from fluksio.flow.connector import CONTRACT_VERSION, ConnectorNode
|
||||
|
||||
|
||||
class MyConnector(ConnectorNode):
|
||||
contract = CONTRACT_VERSION
|
||||
title = "My connector"
|
||||
description = "Reads a device."
|
||||
```
|
||||
|
||||
`contract` must be declared **on the connector class itself**. The loader reads
|
||||
it from the class `__dict__` rather than through inheritance, so a connector
|
||||
written against an older base cannot pass itself off as current. A mismatch is
|
||||
logged and the connector is skipped; nothing else in the engine changes.
|
||||
Declare `contract` on the class itself: inheritance does not count.
|
||||
The current contract version is `1`. A mismatch is logged and the connector
|
||||
is skipped. Breaking changes bump the version; compatible additions do not.
|
||||
|
||||
The version is bumped only when a change would break connectors written against
|
||||
the old surface. Additions that leave existing connectors working do not bump
|
||||
it.
|
||||
`title` defaults to the entry point name; `description` may be empty.
|
||||
|
||||
## Packaging and discovery
|
||||
|
||||
A connector is an ordinary Python package. It advertises its node class through
|
||||
an entry point in the `fluksio.node_types` group:
|
||||
Register the class in your package's `pyproject.toml`:
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "fluksio-connector-skeleton"
|
||||
version = "0.1.0"
|
||||
|
||||
[project.entry-points."fluksio.node_types"]
|
||||
random_sensor = "fluksio_connector_skeleton:RandomSensor"
|
||||
my_device = "fluksio_connector_mydevice:MyConnector"
|
||||
```
|
||||
|
||||
- The **entry point name** is the node type as it appears on the canvas and in
|
||||
a stored flow. It must not collide with a built-in type (`python`, `mqtt`,
|
||||
`http`, `influxdb`, `delay`, `mlp`); a collision is logged and skipped.
|
||||
- The **package name and version** are the manifest. There is no second
|
||||
metadata file to keep in step, and the editor shows the provenance
|
||||
(`fluksio-connector-skeleton 0.1.0`) on the node type.
|
||||
The entry point name is the stored node type. Names already registered,
|
||||
including built-in types, are skipped. The editor shows the package name
|
||||
and version from its installed metadata.
|
||||
|
||||
Discovery happens once, when the engine starts. A connector's code is imported,
|
||||
and Python does not re-import a changed module, so installing or upgrading one
|
||||
means restarting the engine.
|
||||
|
||||
## Declaring what a connector is
|
||||
|
||||
```python
|
||||
class RandomSensor(ConnectorNode):
|
||||
contract = CONTRACT_VERSION
|
||||
title = "Random sensor" # shown in the add-node palette
|
||||
description = "Emits a random reading."
|
||||
```
|
||||
|
||||
`title` falls back to the entry point name; `description` may be empty.
|
||||
Discovery runs at engine startup. Install connectors in the engine's
|
||||
environment and restart after installation or code changes.
|
||||
|
||||
## Parameters
|
||||
|
||||
Settings are a pydantic model, and the editor builds a form from its JSON
|
||||
schema. Subclass the base so `poll_interval` stays available:
|
||||
Subclass `ConnectorNode.Params` to keep `poll_interval` and add your settings:
|
||||
|
||||
```python
|
||||
from pydantic import Field
|
||||
|
||||
|
||||
class Params(ConnectorNode.Params):
|
||||
host: str = "localhost"
|
||||
api_key: str | None = Field(default=None, json_schema_extra={"x-secret": True})
|
||||
api_key: str | None = Field(
|
||||
default=None, json_schema_extra={"x-secret": True}
|
||||
)
|
||||
```
|
||||
|
||||
- Booleans render as switches, numbers and strings as inputs. Objects and
|
||||
arrays are not rendered, so keep settings flat.
|
||||
- A field carrying **`x-secret`** renders as a picker over the stored secrets
|
||||
and writes a reference, `{"$secret": "name"}`, rather than the value. The
|
||||
engine resolves it when the node is built, so a credential never lands in
|
||||
`flow.json`. Always mark credentials this way.
|
||||
- The parsed model is available as `self.config`.
|
||||
The node panel builds a form from this model's JSON schema: switches for
|
||||
booleans, inputs for numbers and strings, and JSON fields for objects and
|
||||
arrays. Parsed settings are available as `self.config`.
|
||||
|
||||
Mark credentials with `x-secret`. The editor stores a reference such as
|
||||
`{"$secret": "device-key"}`, and the engine resolves it while building the node.
|
||||
|
||||
## Ports and messages
|
||||
|
||||
Ports are declared by whoever places the node, not by the connector, and they
|
||||
are `MessageSpec`s like any other node's: a message name, a `DType`, and an
|
||||
optional `interval`.
|
||||
The flow author declares ports as `MessageSpec` values. Each has a message
|
||||
name, a `DType`, and optional delivery settings.
|
||||
|
||||
- `poll()` returns a dict **keyed by output port**, and the base class maps it
|
||||
onto message names; `write()` receives a dict **keyed by input port** the
|
||||
same way.
|
||||
- Every value is checked against the port's declared `DType` on the way out. A
|
||||
wrong type is an error, not a hint.
|
||||
- `interval` on a port is enforced by the engine, not by the connector: an
|
||||
output port publishes at most every *n* seconds, an input port wakes its node
|
||||
at most that often. A connector should poll at the rate the device is
|
||||
comfortable with and leave delivery rates to whoever wires it up.
|
||||
- A key no port declares is an error, and it fails the whole reading rather
|
||||
than the one value. A mistyped metric name is how a training curve goes
|
||||
missing.
|
||||
- `poll()` returns values keyed by **output port**.
|
||||
- `write()` receives values keyed by **input port**.
|
||||
- The base class maps local port names to message names and validates types.
|
||||
- Unknown output keys fail the reading. Return only declared ports.
|
||||
- Port `interval` limits are enforced by the engine; the connector controls
|
||||
how often it reads the device.
|
||||
|
||||
### When the device decides what the ports are
|
||||
|
||||
A connector for a device whose readings vary by model (which components a
|
||||
relay has, which entities were flashed onto a board) cannot know its ports in
|
||||
advance, and returning everything the device reports would fail on the first
|
||||
value nobody bound. Narrow the reading to the ports that were declared:
|
||||
For devices with variable readings, filter to the configured outputs:
|
||||
|
||||
```python
|
||||
declared = {spec.port for spec in self.output_ports if spec.name}
|
||||
return {port: value for port, value in reading.items() if port in declared}
|
||||
```
|
||||
|
||||
`spec.port` is the local, unqualified name and stays that way for the node's
|
||||
whole life, so the set can be taken fresh each time. Say something in the log
|
||||
when a declared port is not one the device has, once rather than once per poll: from
|
||||
the canvas a renamed entity and a typo look the same, and both leave a port
|
||||
silent forever.
|
||||
Log missing device fields once so a typo does not leave a port silently empty.
|
||||
|
||||
## Polling
|
||||
|
||||
Set `poll_interval` and implement `poll()`; the base class runs the loop:
|
||||
|
||||
```python
|
||||
async def poll(self) -> dict[str, Any] | None:
|
||||
async def poll(self):
|
||||
return {"reading": await self._read_device()}
|
||||
```
|
||||
|
||||
- Return `None` when there is nothing new.
|
||||
- **Only changed values are published.** A device polled every few seconds
|
||||
usually says the same thing, and every publication wakes everything
|
||||
downstream, so the loop compares against what it last published, meaning what it
|
||||
actually published, so a publication that failed is retried next tick rather
|
||||
than counting as said.
|
||||
- Raising is not fatal: it is reported as a health problem and retried on the
|
||||
next tick.
|
||||
- The loop calls `inject`, which runs the graph, on a worker thread. `poll()`
|
||||
itself runs on the event loop and must not block it.
|
||||
Set `poll_interval` in seconds; `0` disables polling. Return `None` when
|
||||
there is nothing new. The base loop publishes only changed values and retries
|
||||
a failed reading or publication on the next tick. It reports connection health
|
||||
around each poll.
|
||||
|
||||
A connector that is pushed to rather than polled leaves `poll_interval` at 0,
|
||||
overrides `start`/`stop` to open and close its own subscription, and calls
|
||||
`self.inject({...})` when something arrives.
|
||||
`poll()` runs on the event loop. Use `asyncio.to_thread` for blocking reads.
|
||||
The base loop already calls `inject` on a worker thread.
|
||||
|
||||
For pushed readings, leave `poll_interval=0`, manage the subscription in
|
||||
`start()` and `stop()`, and call `self.inject({...})` when values arrive.
|
||||
Offload `inject` too when calling it from the event loop.
|
||||
|
||||
## Writing
|
||||
|
||||
A connector that commands something rather than only reading it implements
|
||||
`write()`. It is called when the node's inputs are satisfied, with the values
|
||||
keyed by input port:
|
||||
|
||||
```python
|
||||
def write(self, **ports: Any) -> dict[str, Any] | None:
|
||||
def write(self, **ports):
|
||||
self._device.set(ports["level"])
|
||||
return None
|
||||
```
|
||||
|
||||
- Settings are already parsed on `self.config`; `write()` is handed ports only.
|
||||
- Return `None` unless the device answers something worth publishing, in which
|
||||
case return it keyed by output port, exactly as `poll()` does.
|
||||
- It runs on a worker thread rather than the event loop, but the thread is the
|
||||
scheduler's, so a long block holds up the cascade. Queue the work if the
|
||||
device is slow.
|
||||
- A connector that only reads leaves it alone.
|
||||
- `poll()` and `write()` are independent: a connector may have both, and a
|
||||
node with inputs and no `poll_interval` never polls.
|
||||
Settings stay on `self.config`; arguments contain input ports only.
|
||||
Return `None`, or a dictionary keyed by output port if the device responds
|
||||
with useful values.
|
||||
|
||||
A write is a command, not a value: set `idempotent = False` on the class so a
|
||||
redelivery after a crash does not undo a newer command that already landed.
|
||||
Writes run on a scheduler thread. Return promptly or queue slow work.
|
||||
Polling and writing are independent; a connector may support both.
|
||||
|
||||
Set `idempotent = False` for commands that must not be redelivered after a
|
||||
crash. Leave `write()` unchanged on a read-only connector.
|
||||
|
||||
## Devices whose readings are bytes
|
||||
|
||||
A camera frame or a recorded clip is far too big to be a message, so a
|
||||
connector publishes a reference to it instead:
|
||||
|
||||
```python
|
||||
async def poll(self) -> dict[str, Any] | None:
|
||||
import asyncio
|
||||
|
||||
|
||||
async def poll(self):
|
||||
jpeg = await asyncio.to_thread(self._grab)
|
||||
return {
|
||||
"frame": self.save_artifact(
|
||||
@@ -183,135 +133,110 @@ async def poll(self) -> dict[str, Any] | None:
|
||||
}
|
||||
```
|
||||
|
||||
`save_artifact` stores the bytes and returns what an `image`, `audio` or
|
||||
`video` port carries: the media type has to match the port's type. It only
|
||||
works once the node has started, since the store is the engine's and is handed
|
||||
over then.
|
||||
`save_artifact(data, name="", media_type="application/octet-stream",
|
||||
volatile=False)` returns a reference for an artifact or media port.
|
||||
Match the media type to an `image`, `audio`, or `video` port.
|
||||
|
||||
`volatile=True` is what a camera publishes with. The frame goes to a ring in
|
||||
memory rather than the data volume and is pushed down the websocket to
|
||||
whichever screens are drawing it, so a wall panel sees ten frames a second and
|
||||
the SD card under it is never written to. Frames last until newer ones need the
|
||||
room; leave the flag off for a reading somebody asked to keep.
|
||||
The artifact store is available after `start()`. Use `volatile=True`
|
||||
for disposable live frames: they go to a bounded memory ring and expire as
|
||||
new frames arrive. Use durable storage for recordings.
|
||||
|
||||
Each reading is a new artifact, which the poll loop publishes because its
|
||||
digest differs from the last. Set `poll_interval` to what somebody actually
|
||||
wants to look at. Through a portal the bytes are fetched rather than pushed, so
|
||||
a remote panel wants a frame every second or two; higher rates than that are
|
||||
for the local network, and full-rate video still belongs on the camera's own
|
||||
stream.
|
||||
Local screens receive pushed live frames; portal screens fetch the bytes.
|
||||
Choose a modest poll rate for remote viewing.
|
||||
|
||||
## Keeping state across restarts
|
||||
|
||||
A connection, a session or a device handle lives in the connector's own
|
||||
attributes and goes with the process. The few things that must not, such as
|
||||
the certificate a device presented on first contact or a token it paired with,
|
||||
go in `self.store`:
|
||||
Keep connections and device handles in node attributes. Store short strings
|
||||
such as pairing tokens or pinned certificates in `self.store`:
|
||||
|
||||
```python
|
||||
pem = await asyncio.to_thread(self.store.get, f"cert:{self.config.host}")
|
||||
await asyncio.to_thread(self.store.set, f"cert:{self.config.host}", pem)
|
||||
await asyncio.to_thread(self.store.delete, f"cert:{self.config.host}")
|
||||
value = await asyncio.to_thread(self.store.get, "token:device")
|
||||
await asyncio.to_thread(self.store.set, "token:device", "value")
|
||||
await asyncio.to_thread(self.store.delete, "token:device")
|
||||
```
|
||||
|
||||
- Keys and values are strings; `get` returns `None` for a key that is not
|
||||
there, and `delete` returns whether there was one.
|
||||
- The store belongs to the connector's **top-level package**, not to a node:
|
||||
every node of the package sees the same keys, so two nodes on one bridge
|
||||
share what was stored for it. Put the device in the key.
|
||||
- It lives in the engine's database, so it survives a restart and needs
|
||||
nothing `fluksio serve` does not have. It is available before `start`.
|
||||
- Limits: a key is 1 to 200 characters without `/` or whitespace, a value at
|
||||
most 64 kB, and a package at most 256 keys. Past any of them, `set` raises
|
||||
`ValueError`. Bytes worth keeping belong in `save_artifact`.
|
||||
- Every call is a database round trip, so on the event loop it goes through
|
||||
`asyncio.to_thread`.
|
||||
- Keys and values are strings. `get` returns `None` for a missing key;
|
||||
`delete` returns whether it existed.
|
||||
- All nodes from the connector's top-level package share the namespace.
|
||||
Include a device identifier in each key.
|
||||
- Storage is in the engine's database and is available before `start()`.
|
||||
- Keys allow 1–200 characters, without `/` or whitespace. Values allow
|
||||
64 KiB and each package allows 256 keys; exceeding a limit raises `ValueError`.
|
||||
- Calls block on the database: use `asyncio.to_thread` on the event loop.
|
||||
|
||||
A superuser can see the keys (never the values) and clear them, all of them or
|
||||
one at a time, under *Stored state* in a connector node's settings, or through
|
||||
the API, which names the connector by any node type it installs:
|
||||
Superusers can list keys and clear them under **Stored state** in the node
|
||||
panel. Values are never returned. The equivalent API paths are:
|
||||
|
||||
```sh
|
||||
curl -s $FLUKSIO/connectors/hue/state -H "Authorization: Bearer $TOKEN"
|
||||
curl -X DELETE $FLUKSIO/connectors/hue/state/cert:10.0.0.2 -H "Authorization: Bearer $TOKEN"
|
||||
curl -X DELETE $FLUKSIO/connectors/hue/state -H "Authorization: Bearer $TOKEN"
|
||||
```
|
||||
| Method | Path |
|
||||
|---|---|
|
||||
| `GET` | `/api/v1/connectors/{node_type}/state` |
|
||||
| `DELETE` | `/api/v1/connectors/{node_type}/state/{key}` |
|
||||
| `DELETE` | `/api/v1/connectors/{node_type}/state` |
|
||||
|
||||
So a connector has to cope with a key disappearing: clearing is how somebody
|
||||
tells it to start over.
|
||||
Handle missing keys: clearing one asks the connector to pair or learn again.
|
||||
|
||||
### Pinning a device's certificate
|
||||
|
||||
A device on the local network presents a certificate no system trusts and that
|
||||
names a serial or a MAC rather than its address, so `verify=True` fails and
|
||||
`verify=False` trusts anyone. `fluksio.flow.connector_tls` trusts the
|
||||
certificate the device presented on first contact instead, kept in `self.store`
|
||||
under `cert:<address>`:
|
||||
`fluksio.flow.connector_tls` supports trusting a local device's certificate
|
||||
on first contact:
|
||||
|
||||
```python
|
||||
from fluksio.flow.connector_tls import context_for, pinned_certificate, raise_if_changed
|
||||
import httpx
|
||||
|
||||
from fluksio.flow.connector_tls import (
|
||||
context_for,
|
||||
pinned_certificate,
|
||||
raise_if_changed,
|
||||
)
|
||||
|
||||
address = f"{self.config.host}:{self.config.port}"
|
||||
pem = pinned_certificate(self.store, address, self.config.timeout) # blocking
|
||||
pem = pinned_certificate(self.store, address, self.config.timeout)
|
||||
try:
|
||||
with httpx.Client(verify=context_for(pem)) as client:
|
||||
...
|
||||
response = client.get(f"https://{address}/status")
|
||||
response.raise_for_status()
|
||||
except httpx.ConnectError as exc:
|
||||
raise_if_changed(exc, pem, address, self.config.timeout, "my_node_type")
|
||||
raise_if_changed(exc, pem, address, self.config.timeout, "my_device")
|
||||
raise
|
||||
```
|
||||
|
||||
- First contact sends a TLS hello and nothing else; a device that does not
|
||||
complete one raises `OSError` and nothing is pinned.
|
||||
- Any other certificate is refused in the handshake, before a key or a command
|
||||
is sent. `raise_if_changed` turns that refusal into `CertificateChanged`,
|
||||
which names the new SHA-256 and the key to clear, under the node type given.
|
||||
- The pinned certificate's dates are not checked: a pin trusts that exact
|
||||
certificate, so one that expired, or an engine clock that is off, does not
|
||||
lock the device out.
|
||||
- Clearing the key accepts whatever the device presents next.
|
||||
Run this blocking work off the event loop. First contact pins the certificate
|
||||
under `cert:<address>`; a changed certificate is refused before sending a
|
||||
credential or command. `raise_if_changed` reports `CertificateChanged`
|
||||
with the fingerprint and key to clear.
|
||||
|
||||
## Lifecycle
|
||||
Pins trust the exact certificate, including after its expiry. Clearing the
|
||||
key accepts the certificate presented on the next connection.
|
||||
|
||||
## Lifecycle and health
|
||||
|
||||
```python
|
||||
async def start(self, app=None) -> None: ...
|
||||
async def stop(self, app=None) -> None: ...
|
||||
async def start(self, app=None):
|
||||
await super().start(app)
|
||||
|
||||
|
||||
async def stop(self, app=None):
|
||||
await super().stop(app)
|
||||
```
|
||||
|
||||
- `start` is called when the connector's flow starts, and after every rebuild.
|
||||
- `stop` is called before a rebuild and when the flow is stopped. **It must be
|
||||
idempotent**: it is called whether or not `start` succeeded.
|
||||
- Overriding either means calling `super()` if the polling loop is also wanted.
|
||||
- The `app` argument is the FastAPI application, for the rare connector that
|
||||
needs to mount a route. Most ignore it.
|
||||
`start` runs when a flow starts or rebuilds. `stop` runs before rebuilding
|
||||
or stopping, even after a failed start; make it idempotent. Call `super()`
|
||||
when overriding these hooks if you want the base polling loop.
|
||||
|
||||
A stopped flow gets no `start` at all: that is what stopping it means.
|
||||
The optional `app` is the FastAPI application, for connectors that mount
|
||||
routes. Most connectors can ignore it.
|
||||
|
||||
## Health
|
||||
Report connection changes if you manage your own subscription:
|
||||
|
||||
```python
|
||||
self.report_health("ok")
|
||||
self.report_health("degraded", "3 of 5 registers timed out")
|
||||
self.report_health("down", str(exc))
|
||||
self.report_health("degraded", "Some readings timed out")
|
||||
self.report_health("down", "Connection lost")
|
||||
```
|
||||
|
||||
Three values, `ok`, `degraded` and `down`, plus an optional detail string.
|
||||
Reporting the same status twice is free, since only changes are published. A node
|
||||
reporting `down` is named among its flow's issues and counted on the health
|
||||
summary on Home; `degraded` means still working, and is not. The polling loop
|
||||
already reports around `poll()`; a connector managing its own connection should
|
||||
report when it connects and when it loses the connection.
|
||||
Repeated statuses do not emit another event. `down` appears among flow issues
|
||||
and on Home; `degraded` means the connection still works. Loading failures
|
||||
are reported separately.
|
||||
|
||||
Health is about the connection, not about loading: a connector that fails to
|
||||
load is reported separately and never runs.
|
||||
|
||||
## What a connector must not do
|
||||
|
||||
- **Hold state that belongs in the flow.** Logic nodes are stateless and run in
|
||||
parallel; a connector may hold a connection, a session, or a device handle,
|
||||
and keep what has to outlive a restart in `self.store`, but a value another
|
||||
node needs belongs in a message.
|
||||
- **Block the event loop.** `poll`, `start` and `stop` are async and run on the
|
||||
loop. Use `asyncio.to_thread` for blocking I/O.
|
||||
- **Read credentials from the environment.** They come through parameters, so
|
||||
the person deploying a flow can change them without touching the host.
|
||||
Keep shared flow values in messages, credentials in secret parameters, and
|
||||
blocking I/O off the event loop.
|
||||
|
||||
Reference in New Issue
Block a user