Media dtypes: image, audio and video as narrowed artifact references
Docs / docs (push) Successful in 30s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m7s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m54s
pre-commit / pre-commit (push) Failing after 4m24s
Test Backend / test-backend (push) Successful in 3m8s
Compose Smoke Test / test-compose (push) Successful in 40s
Playwright Tests / merge-reports (push) Successful in 1m33s
Docs / docs (push) Successful in 30s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m7s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m54s
pre-commit / pre-commit (push) Failing after 4m24s
Test Backend / test-backend (push) Successful in 3m8s
Compose Smoke Test / test-compose (push) Successful in 40s
Playwright Tests / merge-reports (push) Successful in 1m33s
A port may now declare `image`, `audio` or `video`. Each is the artifact
reference the engine already had, narrowed by the `media_type` on it, so a
speech recogniser declares what it eats rather than taking any bytes at all and
finding out. Bytes still never travel as a message and nothing on the wire
stops being JSON: a camera publishes one reference per frame, a microphone one
per chunk, and a reference may carry a `meta` dict nothing here interprets.
Streaming media is therefore an ordinary streaming port — with one change to
what that means. An emission used to journal an item with no payload, so
downstream read whatever was current when the item was claimed; a consumer
slower than its producer saw only the newest chunk and the ones between were
lost. That is right for a training curve and wrong for a second of speech, so
an emission now journals a `kind="emission"` item carrying its values, and the
executor hands them to the nodes reading that message instead of writing them
to state again. The value in state stays the latest, which is what everything
else reads, and the wave is filtered by what actually changed rather than
walking everything reachable. No queue serialization change — the existing
`outputs` field carries it.
Continuous media makes the store's missing GC a real problem, so this closes
it: `sweep_artifacts` runs hourly, keeps every digest a `run_artifact` row
records or a live message holds, spares anything written in the last hour, and
stands aside entirely while a run is in flight, since a node may store a
checkpoint long before it returns the reference to it. That also collects the
orphans a deleted flow has always left behind. `ARTIFACT_GC_INTERVAL_S=0` turns
it off.
Around the edges: `GET /artifacts/{digest}` serves the media type the caller
passes and answers ranged requests, so a browser plays a clip rather than
downloading it; `PUT` spools to disk instead of holding the whole body in
memory, as does `save_artifact` given a path; a Media widget draws whatever its
message points at, and a wall panel may fetch the bytes its own tiles are
showing and nothing else; and a connector gets `save_artifact`, for a device
whose readings are bytes.
What this cannot do is live video: a frame every second or two is a glance, and
the honest answer above that is the camera's own stream, which the widget takes
as a URL and the browser plays from source.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -33,6 +33,13 @@ class DType(str, Enum):
|
||||
thirty-megabyte checkpoint never sits in Redis. Inline codecs would only be
|
||||
needed for payloads too small to be worth a round trip, and nothing asks
|
||||
for that yet.
|
||||
|
||||
``image``, ``audio`` and ``video`` are that same reference narrowed to a
|
||||
media family, so a speech recogniser declares what it eats rather than
|
||||
taking any bytes at all and finding out. Bytes still never travel as a
|
||||
message: a camera publishes one reference per frame, a microphone one per
|
||||
chunk. A reference may carry a ``meta`` dict — sample rate, dimensions, a
|
||||
sequence number — which nothing here interprets.
|
||||
"""
|
||||
|
||||
FLOAT = "float"
|
||||
@@ -51,6 +58,12 @@ class DType(str, Enum):
|
||||
#: A reference to stored bytes:
|
||||
#: ``{"digest": "sha256:…", "size": int, "media_type": str, "name": str}``.
|
||||
ARTIFACT = "artifact"
|
||||
#: An artifact reference whose ``media_type`` is ``image/*``.
|
||||
IMAGE = "image"
|
||||
#: An artifact reference whose ``media_type`` is ``audio/*``.
|
||||
AUDIO = "audio"
|
||||
#: An artifact reference whose ``media_type`` is ``video/*``.
|
||||
VIDEO = "video"
|
||||
|
||||
|
||||
_JSON_TYPES = (dict, list, str, int, float, bool, type(None))
|
||||
@@ -65,6 +78,14 @@ _ITEM_TYPES = frozenset(
|
||||
{DType.FLOAT, DType.INT, DType.STR, DType.BOOL, DType.JSON, DType.RECORD}
|
||||
)
|
||||
|
||||
#: The media dtypes, and the ``media_type`` family a reference must declare to
|
||||
#: satisfy each. A stream of them is a stream of references, one per chunk.
|
||||
MEDIA_FAMILIES = {
|
||||
DType.IMAGE: "image/",
|
||||
DType.AUDIO: "audio/",
|
||||
DType.VIDEO: "video/",
|
||||
}
|
||||
|
||||
|
||||
#: How deep to look for a non-finite number. Deeper than any payload that
|
||||
#: reads well on a canvas, and a bound on a value that refers to itself.
|
||||
@@ -147,6 +168,11 @@ def _is_artifact(value: Any) -> bool:
|
||||
)
|
||||
|
||||
|
||||
def _media_family(value: Any) -> str:
|
||||
"""The ``media_type`` an artifact reference declares, lowercased."""
|
||||
return str(value.get("media_type") or "").lower()
|
||||
|
||||
|
||||
def _matches(dtype: DType, value: Any) -> bool:
|
||||
"""Whether one value satisfies a scalar or record type."""
|
||||
if dtype is DType.BOOL:
|
||||
@@ -161,6 +187,10 @@ def _matches(dtype: DType, value: Any) -> bool:
|
||||
return _is_record(value)
|
||||
if dtype is DType.ARTIFACT:
|
||||
return _is_artifact(value)
|
||||
if dtype in MEDIA_FAMILIES:
|
||||
return _is_artifact(value) and _media_family(value).startswith(
|
||||
MEDIA_FAMILIES[dtype]
|
||||
)
|
||||
return isinstance(value, _JSON_TYPES)
|
||||
|
||||
|
||||
@@ -243,6 +273,14 @@ class MessageSpec(BaseModel):
|
||||
else:
|
||||
ok = _matches(self.dtype, value)
|
||||
if not ok:
|
||||
if self.dtype in MEDIA_FAMILIES and _is_artifact(value):
|
||||
# It is a reference, just to the wrong kind of bytes — saying so
|
||||
# beats "expected audio, got dict" on a media_type typo.
|
||||
raise TypeError(
|
||||
f"{where}: expected {self.dtype.value} "
|
||||
f"({MEDIA_FAMILIES[self.dtype]}*), got an artifact of "
|
||||
f"'{_media_family(value) or 'no media type'}'"
|
||||
)
|
||||
raise TypeError(
|
||||
f"{where}: expected {self.dtype.value}, got {type(value).__name__}"
|
||||
)
|
||||
@@ -263,7 +301,13 @@ class MessageSpec(BaseModel):
|
||||
return str(value).lower() in ("true", "1", "yes", "on")
|
||||
if self.dtype is DType.STR:
|
||||
return value if isinstance(value, str) else json.dumps(value)
|
||||
if self.dtype in (DType.SERIES, DType.RECORD, DType.LIST, DType.ARTIFACT):
|
||||
if self.dtype in (
|
||||
DType.SERIES,
|
||||
DType.RECORD,
|
||||
DType.LIST,
|
||||
DType.ARTIFACT,
|
||||
*MEDIA_FAMILIES,
|
||||
):
|
||||
# A structured payload arriving as text is the same hint a numeric
|
||||
# one is; the shape itself is still checked afterwards.
|
||||
return json.loads(value) if isinstance(value, str) else value
|
||||
|
||||
Reference in New Issue
Block a user