Live view over WebRTC (openpanel L1, server and browser)

A device reporting a `live` port streams its camera to the browser's own
RTCPeerConnection. The flows socket carries the signalling as text, so it
crosses the portal: `live` messages on a person's socket, `signal` on a
device's (start, sdp, candidate, bye, busy). The server routes by session,
holds a device to two viewers, ends a session when either socket closes or
no answer comes within 10 s, and gives each socket an outbox of its own
that ends a session rather than drop a message. TURN credentials are
minted per session in coturn's REST scheme (STUN_URLS, TURN_URLS,
TURN_SECRET), the viewer's with the device's offer; GET /devices/{id}/ice
hands them out too. The tunnel keeps a stream's early frames instead of
dropping them past eight and forgets them when the stream or link goes.

A media tile on a device's camera, and the Devices tab's preview, play
live when tapped, pause frames meanwhile, and fall back to frames with a
short note on busy, failure or no picture within 10 s. coturn is an
optional compose service behind the `turn` profile.
This commit is contained in:
Melvin Strobl
2026-10-17 08:37:39 +02:00
parent efd883d092
commit 6a98dcd23f
20 changed files with 1535 additions and 24 deletions
+32
View File
@@ -201,6 +201,7 @@ on its own, so this is for reclaiming space sooner.
| `POST` | `/devices/{id}/pair` | adopt by `{code}`, or by `{request}` from the pending list for a request from your network |
| `POST` | `/devices/{id}/unpair` | revoke the device credential, one the portal issued included |
| `PUT` | `/devices/{id}/report` | what the device has, from its own credential only; at most 16 KB, replaces the last report, read leniently (see [What a device reports](../interface/dashboards.md#what-a-device-reports)) |
| `GET` | `/devices/{id}/ice` | STUN and TURN servers for a live view of the device, as `{ice_servers: [{urls, username?, credential?}], ttl}`, with TURN credentials of the caller's own; not for a device credential |
`/panels/pair`, `/panels/pair/{code}` and `GET /panels/{id}` answer as their
`/devices/` counterparts, for firmware that has not updated yet.
@@ -275,6 +276,37 @@ logged and dropped and the socket stays open. With `http` it uploads with
`POST /messages/<message>`. A device that is unpaired or deleted while
connected is closed with 1008.
### Live view
A device that reports a `live` port streams its camera over WebRTC. The
video goes peer to peer, or through a TURN relay, and the socket carries only
the signalling, as text, so it crosses the portal too. A person's socket
sends and hears `{"type": "live", "device": "<id>", "session": "<id>",
"action": ...}`, a device's socket `{"type": "signal", "session": "<id>",
"action": ...}`. A session id is 8 to 64 letters, digits, `-` or `_`, chosen by
the viewer. In order:
| Step | Viewer's socket | Device's socket |
|---|---|---|
| 1 | sends `start` | hears `start` with `ice_servers` minted for the device |
| 2 | hears `sdp` (the offer) with `ice_servers` minted for the viewer | sends `sdp` with its offer |
| 3 | sends `sdp` with its answer | hears `sdp` |
| 4 | sends and hears `candidate`, the bare `candidate:...` line | hears and sends `candidate` |
| 5 | sends `bye`, or hears `bye` or `busy` with a `reason` | hears `bye` with a `reason`, or sends `bye` or `busy` |
Any signed-in person may watch any device. A device speaks only for the
sessions it was handed on that socket. The server answers `busy` itself once a
device has 2 sessions, and `bye` with `offline` or `unsupported` when the
device is not connected or reports no `live` port. A session the viewer has not
answered within 10 seconds ends with `timeout`, and either socket closing ends
its sessions with `closed` or `offline`. A socket that falls 64 signalling
messages behind ends the session with `overflow` instead of losing one.
`ice_servers` follow `RTCIceServer`: `{urls: [...]}` for STUN,
`{urls, username, credential}` for TURN, in coturn's REST scheme with a
username of `<expiry>:<who>`, good for an hour. A device that sends its camera's
frames on the same socket pauses them while a session negotiates, so its
offer and candidates are not queued behind frames.
## Errors and client generation
| Code | Meaning |
+22 -2
View File
@@ -208,7 +208,8 @@ at, and stops when it scrolls away, the tab is hidden or the tile closes. A
clip keeps playing either way. One page is pushed frames for at most 8
messages at once. A ninth tile fetches its frames one by one and does not turn
a camera on. A [device's camera](#a-devices-camera) sends only while a tile like
this wants it.
this wants it. A tile on a device's camera that streams live shows **Live**:
tap it for [live video](#live-view).
For continuous video, set **Live stream** to a browser-compatible camera URL.
The browser must be able to reach that URL; enrollment does not expose the
@@ -600,7 +601,7 @@ keeps the switch as it was set.
### A device's camera
A camera device (openpanel on a NANO) reports four ports:
A camera device (openpanel on a NANO) reports these ports:
| Port | Type | Settings | What it sends |
|---|---|---|---|
@@ -608,6 +609,7 @@ A camera device (openpanel on a NANO) reports four ports:
| `person` | `bool` | `hold_s` (10) | whether someone is in view, held for `hold_s` after they leave |
| `detections` | `json` | | `{count, boxes: [{score, x0, y0, x1, y1}]}`, boxes scaled 0 to 1, on change and at most once a second |
| `snapshot` | `image` | | one stored frame each time `person` turns true |
| `live` | `video`, on demand | | H.264 over WebRTC while a [live view](#live-view) runs; never wired in a flow |
`camera` sends only while it has demand: while a page wants its frames (a
dashboard tile, the preview on the device's page, a screen through the
@@ -633,6 +635,24 @@ A device may fetch the last 8 artifacts each of its messages pointed at, not
only the current one, so a screen fetching a camera frame by frame through the
portal is not refused a frame the camera has just replaced.
#### Live view
A media tile showing a device's `camera`, and the preview on the device's
page, offer **Live** while the device is online and reports `live`. Tap the
tile and the browser connects to the device with WebRTC, without a plugin,
and plays its video in the tile under a **Live** badge. Tap again to stop. It
stops by itself when the tile scrolls away, the tab is hidden or the page
closes. While it plays the tile asks for no frames, so the camera encodes one
picture rather than two, and frames come back when it ends.
A device serves 2 live viewers. A third tile, a failed connection or no
picture within 10 seconds stays on frames and says so briefly. On one network
the two ends find each other directly. Between networks they need STUN and,
behind strict NATs, a TURN relay: set `STUN_URLS`, `TURN_URLS` and
`TURN_SECRET` (see [Configuration](../reference/configuration.md#live-view)).
The server only passes on the few messages that set a connection up, the
video never crosses it. A wall panel's own screen does not offer live view.
### Before device nodes
Before device nodes a device carried its ports itself. On its first start an
+18
View File
@@ -189,6 +189,24 @@ under Docker, raise `shm_size` with the ring size.
See [Agents over MCP](../code/agents.md).
## Live view
A camera's [live view](../interface/dashboards.md#live-view) connects the
browser and the device directly. These name the servers that help when the two
are on different networks. Unset, only direct connections are tried, and no
third party learns who watches what.
| Variable | Default | Purpose |
|---|---|---|
| `STUN_URLS` | unset | STUN servers, comma separated, such as `stun:turn.example.com:3478` |
| `TURN_URLS` | unset | TURN relays, comma separated, such as `turn:turn.example.com:3478,turns:turn.example.com:5349?transport=tcp` |
| `TURN_SECRET` | unset | the secret shared with the TURN server (coturn's `static-auth-secret`); TURN is used only with it |
Each viewer and device gets TURN credentials of its own per live view, good for
an hour, which coturn checks against the shared secret. A view relayed through
TURN therefore ends after an hour. coturn also answers STUN, so one server
covers both.
## Mail and monitoring
Password-reset mail requires `SMTP_HOST` and `EMAILS_FROM_EMAIL`.