Media dtypes: image, audio and video as narrowed artifact references
Docs / docs (push) Successful in 30s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m7s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m54s
pre-commit / pre-commit (push) Failing after 4m24s
Test Backend / test-backend (push) Successful in 3m8s
Compose Smoke Test / test-compose (push) Successful in 40s
Playwright Tests / merge-reports (push) Successful in 1m33s

A port may now declare `image`, `audio` or `video`. Each is the artifact
reference the engine already had, narrowed by the `media_type` on it, so a
speech recogniser declares what it eats rather than taking any bytes at all and
finding out. Bytes still never travel as a message and nothing on the wire
stops being JSON: a camera publishes one reference per frame, a microphone one
per chunk, and a reference may carry a `meta` dict nothing here interprets.

Streaming media is therefore an ordinary streaming port — with one change to
what that means. An emission used to journal an item with no payload, so
downstream read whatever was current when the item was claimed; a consumer
slower than its producer saw only the newest chunk and the ones between were
lost. That is right for a training curve and wrong for a second of speech, so
an emission now journals a `kind="emission"` item carrying its values, and the
executor hands them to the nodes reading that message instead of writing them
to state again. The value in state stays the latest, which is what everything
else reads, and the wave is filtered by what actually changed rather than
walking everything reachable. No queue serialization change — the existing
`outputs` field carries it.

Continuous media makes the store's missing GC a real problem, so this closes
it: `sweep_artifacts` runs hourly, keeps every digest a `run_artifact` row
records or a live message holds, spares anything written in the last hour, and
stands aside entirely while a run is in flight, since a node may store a
checkpoint long before it returns the reference to it. That also collects the
orphans a deleted flow has always left behind. `ARTIFACT_GC_INTERVAL_S=0` turns
it off.

Around the edges: `GET /artifacts/{digest}` serves the media type the caller
passes and answers ranged requests, so a browser plays a clip rather than
downloading it; `PUT` spools to disk instead of holding the whole body in
memory, as does `save_artifact` given a path; a Media widget draws whatever its
message points at, and a wall panel may fetch the bytes its own tiles are
showing and nothing else; and a connector gets `save_artifact`, for a device
whose readings are bytes.

What this cannot do is live video: a frame every second or two is a glance, and
the honest answer above that is the camera's own stream, which the widget takes
as a URL and the browser plays from source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-26 23:44:55 +02:00
co-authored by Claude Opus 5
parent 8be7e424ba
commit 0ffcabfdb9
37 changed files with 1271 additions and 62 deletions
+8
View File
@@ -128,6 +128,14 @@ warning into a refusal to start.
| `FLOW_CPUS` | `0` | cores nodes that declare `resources` may be given; 0 works it out as every core but two, which are what keeps the engine answering while the machine is busy |
| `FLOW_GPUS` | `0` | GPUs on this machine, each held by one node at a time. Not detected — say how many there are |
| `OBS_RETENTION_DAYS` | `30` | how long metrics, events and run records are kept |
| `ARTIFACT_GC_INTERVAL_S` | `3600` | how often artifact bytes nothing refers to are swept away; 0 never sweeps |
| `ARTIFACT_GC_GRACE_S` | `3600` | how long a freshly written artifact is spared, whatever refers to it |
An artifact is referred to by a run that recorded it or by a message currently
holding it; anything else is what a camera published four hours ago, and the
sweep is what keeps a flow streaming media from filling the disk. It stands
aside entirely while a run is in flight, since a node may store a checkpoint
long before it returns the reference to it.
A node that declares nothing is not accounted against `FLOW_CPUS`; it runs on
the shared pool and is given `FLOW_CPUS / FLOW_MAX_WORKERS` as a thread cap, so
+23
View File
@@ -166,6 +166,29 @@ def write(self, **ports: Any) -> dict[str, Any] | None:
A write is a command, not a value: set `idempotent = False` on the class so a
redelivery after a crash does not undo a newer command that already landed.
## Devices whose readings are bytes
A camera frame or a recorded clip is far too big to be a message, so a
connector publishes a reference to it instead:
```python
async def poll(self) -> dict[str, Any] | None:
jpeg = await asyncio.to_thread(self._grab)
return {
"frame": self.save_artifact(jpeg, "frame.jpg", media_type="image/jpeg")
}
```
`save_artifact` stores the bytes and returns what an `image`, `audio` or
`video` port carries — the media type has to match the port's type. It only
works once the node has started, since the store is the engine's and is handed
over then.
Each reading is a new artifact, which the poll loop publishes because its
digest differs from the last. Set `poll_interval` to what somebody actually
wants to look at: a frame every second or two is a glance, and live video
belongs on the camera's own stream rather than in the graph.
## Lifecycle
```python
+34
View File
@@ -99,6 +99,39 @@ stays valid wherever the store is reachable from, including on another machine.
Node code produces one with `fluksio.save_artifact` and opens one with
`fluksio.load_artifact`. See [Writing node code](../code/nodes.md#bytes-artifacts).
### `image`, `audio`, `video`
The same reference, narrowed to a kind of media by its `media_type`.
```json
{"digest": "sha256:…", "size": 61344, "media_type": "image/jpeg", "name": "frame.jpg",
"meta": {"width": 1280, "height": 720, "seq": 41}}
```
An `audio` port takes `audio/*` and refuses anything else, so a speech
recogniser declares what it eats rather than taking any bytes at all and
finding out. An `artifact` port still accepts all three — media narrows
artifact, not the other way round.
`meta` is optional and nothing here reads it: sample rates, dimensions and
sequence numbers are for whoever consumes the media.
Bytes still never travel as a message. A camera publishes one reference per
frame and a microphone one per chunk — which makes a media stream an ordinary
[streaming port](../concepts/flows.md#streaming-ports), and each frame an
artifact. What that costs is worth knowing before pointing a camera at it:
| Rate | Where it works |
|---|---|
| A clip a second (speech) | anywhere, including through the portal |
| A frame every second or two (a glance at a door) | locally; through the portal, every few seconds |
| Live video, 1530 fps | not here — see below |
Real-time video is not a message-plane problem: every frame would be an
artifact, an event and a fetch. Point a media widget's **stream URL** at
whatever the camera already serves and the browser plays it from source; the
messages then carry the occasional still, and the flow reacts to those.
### `json`
Anything JSON-serializable. The escape hatch, and the right answer when a
@@ -129,6 +162,7 @@ carry one of them literally is asking for a value this engine reads as a name.
| Notification | `record` |
| Value | anything |
| Icon | weather strings, booleans and numbers alike |
| Media | `image`, `audio`, `video`, `artifact` |
| Clock, Text | nothing — they bind to no message |
Enforced on the server as well as in the editor.