Cameras on devices (openpanel C1): frames in memory, a binary uplink, demand
- Volatile frames are held in memory with a byte-bounded queue instead of the /dev/shm ring scanned on every put; workers keep writing to the directory, which is trimmed on a timer. A volatile upload no longer spools to disk. ARTIFACT_VOLATILE_BYTES=0 turns the ring off. - A device sends camera frames as binary messages on its socket, held and published in one step and credited to its device node; refused frames are logged and dropped. Its snapshot carries demand and the uplink (binary, or http through the portal). - One demand registry for every socket's wanted media, the tunnel's included, plus what running flows read, mapped onto on-demand ports and published as device_demand; at most 8 media names per socket. - Per-message deflate off in serve and dev; a device may fetch the last 8 digests of a message; published dashboards cached per store revision; a revoked device socket is closed with 1008. - Media tiles want bytes only while on screen; the Devices tab previews a wired camera; the device node warns when a flow reads an on-demand port.
This commit is contained in:
@@ -173,6 +173,8 @@ curl "$FLUKSIO/artifacts/<digest>" \
|
||||
|
||||
The upload returns a reference with the content digest. Identical bytes are
|
||||
stored once. Node code uses `save_artifact` and `load_artifact` instead.
|
||||
`?volatile=true` holds the bytes in the engine's memory for a live frame, where
|
||||
newer frames push them out, instead of storing them.
|
||||
|
||||
A superuser can remove stored bytes with `DELETE /artifacts/<digest>`. While
|
||||
a run, a current message or a flow input's initial value still references the
|
||||
@@ -249,6 +251,30 @@ Browsers first call `POST /flows/ws-ticket`, then connect with
|
||||
`?ticket=<ticket>`. Tickets are single-use and expire after 30 seconds.
|
||||
The socket sends a snapshot followed by node, value, health, and run events.
|
||||
|
||||
A client asks for the bytes of the media it shows with
|
||||
`{"type": "media", "names": [...]}`, which replaces what it asked for before,
|
||||
for at most 8 messages. Each frame held in memory then arrives as a binary
|
||||
message before the value naming it: four bytes of header length (big endian),
|
||||
a JSON header `{"type": "media", "name", "digest", "media_type", "ts"}`, then
|
||||
the bytes. Messages are sent without per-message compression.
|
||||
|
||||
A device's socket is bounded to its own messages. Its snapshot carries
|
||||
`"demand"`, the ports it reported as `on_demand` that someone wants now, and
|
||||
`"uplink"`, `"binary"` on its own socket and `"http"` through the portal. When
|
||||
that set changes it is sent `{"type": "demand", "ports": [...]}`. A port has
|
||||
demand while a socket asks for the media of the message it is wired to, or a
|
||||
node of a running flow reads that message.
|
||||
|
||||
With a `binary` uplink the device sends a camera frame as a binary message in
|
||||
the same layout, with the header `{"type": "frame", "name": "<message>",
|
||||
"media_type": "image/jpeg", "ts": <device ms>}`. The message has to be one of
|
||||
its `in` ports wired as `image`. The frame is held in memory and published
|
||||
in one step, credited to its device node. A frame that does not qualify is
|
||||
logged and dropped and the socket stays open. With `http` it uploads with
|
||||
`PUT /artifacts?volatile=true&name=<message>` and publishes with
|
||||
`POST /messages/<message>`. A device that is unpaired or deleted while
|
||||
connected is closed with 1008.
|
||||
|
||||
## Errors and client generation
|
||||
|
||||
| Code | Meaning |
|
||||
|
||||
@@ -203,6 +203,13 @@ For local streams, `volatile=True` artifacts can be pushed over the websocket
|
||||
with their message. Stored artifacts are fetched individually, as are frames
|
||||
viewed through a portal. Use occasional snapshots for remote screens.
|
||||
|
||||
A tile asks for frames only while it is on screen in a tab that is being looked
|
||||
at, and stops when it scrolls away, the tab is hidden or the tile closes. A
|
||||
clip keeps playing either way. One page is pushed frames for at most 8
|
||||
messages at once. A ninth tile fetches its frames one by one and does not turn
|
||||
a camera on. A [device's camera](#a-devices-camera) sends only while a tile like
|
||||
this wants it.
|
||||
|
||||
For continuous video, set **Live stream** to a browser-compatible camera URL.
|
||||
The browser must be able to reach that URL; enrollment does not expose the
|
||||
camera itself. See [Media payloads](../reference/payload-types.md#image-audio-video).
|
||||
@@ -591,6 +598,34 @@ the dashboard settings, named after the device, and the first report of a
|
||||
touch screen turns **Touch friendly** on. A device paired before this version
|
||||
keeps the switch as it was set.
|
||||
|
||||
### A device's camera
|
||||
|
||||
A camera device (openpanel on a NANO) reports four ports:
|
||||
|
||||
| Port | Type | Settings | What it sends |
|
||||
|---|---|---|---|
|
||||
| `camera` | `image`, on demand | `width` (640), `fps` (5), `idle_fps` (0, off), `quality` (65) | JPEG frames, held in memory and never stored |
|
||||
| `person` | `bool` | `hold_s` (10) | whether someone is in view, held for `hold_s` after they leave |
|
||||
| `detections` | `json` | | `{count, boxes: [{score, x0, y0, x1, y1}]}`, boxes scaled 0 to 1, on change and at most once a second |
|
||||
| `snapshot` | `image` | | one stored frame each time `person` turns true |
|
||||
|
||||
`camera` sends only while it has demand: while a page wants its frames (a
|
||||
dashboard tile, the preview on the device's page, a screen through the
|
||||
portal), or a node of a running flow reads its message. Once its flow wires
|
||||
`camera`, the device's page shows a **Camera** preview. A flow that reads
|
||||
`camera` keeps it sending all the time, about 300 GB a month through a cloud
|
||||
instance, so the Device node warns when a node in its flow reads a port that
|
||||
sends on demand. Have flows read `snapshot` or `person` instead.
|
||||
|
||||
The device is told which of its ports have demand on its live socket, and sends
|
||||
its frames there too, as described under [Live events](../code/api.md#live-events).
|
||||
Through the portal the socket carries text only, so a device paired there
|
||||
uploads each frame over HTTP instead.
|
||||
|
||||
A device may fetch the last 8 artifacts each of its messages pointed at, not
|
||||
only the current one, so a screen fetching a camera frame by frame through the
|
||||
portal is not refused a frame the camera has just replaced.
|
||||
|
||||
### Before device nodes
|
||||
|
||||
Before device nodes a device carried its ports itself. On its first start an
|
||||
|
||||
@@ -167,14 +167,16 @@ MIG entries are passed on as written.
|
||||
| `MAX_ARTIFACT_BYTES` | `2147483648` (2 GiB) | upload limit; `0` removes it |
|
||||
| `ARTIFACT_GC_INTERVAL_S` | `3600` | cleanup interval; `0` disables |
|
||||
| `ARTIFACT_GC_GRACE_S` | `3600` | minimum age before cleanup |
|
||||
| `ARTIFACT_VOLATILE_DIR` | auto-detected | live-media directory under `/dev/shm`, or the temporary directory |
|
||||
| `ARTIFACT_VOLATILE_BYTES` | `50331648` (48 MiB) | live-media ring size; `0` stores volatile saves durably |
|
||||
| `ARTIFACT_VOLATILE_DIR` | auto-detected | where local workers write live frames, under `/dev/shm` or the temporary directory |
|
||||
| `ARTIFACT_VOLATILE_BYTES` | `50331648` (48 MiB) | live-media ring size, in the engine's memory and again in that directory; `0` stores volatile saves durably |
|
||||
|
||||
Cleanup removes durable artifacts no longer referenced by recorded runs or
|
||||
current messages, and pauses while a run is active. Volatile artifacts use a
|
||||
bounded ring; the oldest disappear as it fills. Recorded run outputs are
|
||||
copied into durable storage. If you increase the ring size in Docker, increase
|
||||
`shm_size` too.
|
||||
current messages, and pauses while a run is active. Volatile artifacts live in
|
||||
a bounded ring in the engine's memory, and the oldest disappear as it fills.
|
||||
Recorded run outputs are copied into durable storage. A worker in the engine's
|
||||
container writes its frames to `ARTIFACT_VOLATILE_DIR` instead, which the
|
||||
engine trims to the same size every few seconds. If workers publish frames
|
||||
under Docker, raise `shm_size` with the ring size.
|
||||
|
||||
## Agents
|
||||
|
||||
|
||||
Reference in New Issue
Block a user