Media dtypes: image, audio and video as narrowed artifact references

A port may now declare `image`, `audio` or `video`. Each is the artifact
reference the engine already had, narrowed by the `media_type` on it, so a
speech recogniser declares what it eats rather than taking any bytes at all and
finding out. Bytes still never travel as a message and nothing on the wire
stops being JSON: a camera publishes one reference per frame, a microphone one
per chunk, and a reference may carry a `meta` dict nothing here interprets.

Streaming media is therefore an ordinary streaming port — with one change to
what that means. An emission used to journal an item with no payload, so
downstream read whatever was current when the item was claimed; a consumer
slower than its producer saw only the newest chunk and the ones between were
lost. That is right for a training curve and wrong for a second of speech, so
an emission now journals a `kind="emission"` item carrying its values, and the
executor hands them to the nodes reading that message instead of writing them
to state again. The value in state stays the latest, which is what everything
else reads, and the wave is filtered by what actually changed rather than
walking everything reachable. No queue serialization change — the existing
`outputs` field carries it.

Continuous media makes the store's missing GC a real problem, so this closes
it: `sweep_artifacts` runs hourly, keeps every digest a `run_artifact` row
records or a live message holds, spares anything written in the last hour, and
stands aside entirely while a run is in flight, since a node may store a
checkpoint long before it returns the reference to it. That also collects the
orphans a deleted flow has always left behind. `ARTIFACT_GC_INTERVAL_S=0` turns
it off.

Around the edges: `GET /artifacts/{digest}` serves the media type the caller
passes and answers ranged requests, so a browser plays a clip rather than
downloading it; `PUT` spools to disk instead of holding the whole body in
memory, as does `save_artifact` given a path; a Media widget draws whatever its
message points at, and a wall panel may fetch the bytes its own tiles are
showing and nothing else; and a connector gets `save_artifact`, for a device
whose readings are bytes.

What this cannot do is live video: a frame every second or two is a glance, and
the honest answer above that is the camera's own stream, which the widget takes
as a URL and the browser plays from source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

frontend/scripts/capture-screenshots.mjs

@@ -89,6 +89,7 @@ for (const theme of ["light", "dark"]) {
 
     await captureFlows(page, dir)
     await captureDashboards(page, dir)
+    await captureMedia(page, dir)
     await captureRuns(page, dir)
 
     await context.close()
@@ -192,6 +193,41 @@ async function captureDashboards(page, dir) {
   await page.screenshot({ path: `${dir}/app-panel.png` })
 }
 
+/**
+ * A media tile drawing what a camera published, where there is one.
+ *
+ * Skipped unless the media example is seeded (root `make seed-example-media`),
+ * since it is the one shot that needs a source of frames. The bytes arrive as
+ * a blob — the tile fetches them with the session's credential, which no `img`
+ * could carry on its own — so a `blob:` source is the proof the whole path ran
+ * rather than that a picture is merely present.
+ */
+async function captureMedia(page, dir) {
+  const answer = await page.goto(`${APP_URL}/view/camera`, {
+    waitUntil: "networkidle",
+  })
+  if (!answer?.ok()) return
+
+  const picture = page.locator("img[alt='Test camera']")
+  try {
+    await picture.waitFor({ timeout: 15000 })
+    await page.waitForFunction(
+      () =>
+        document
+          .querySelector("img[alt='Test camera']")
+          ?.src?.startsWith("blob:") ?? false,
+      { timeout: 15000 },
+    )
+  } catch {
+    console.warn(
+      "  media tile drew nothing — is `make seed-example-media` run?",
+    )
+    return
+  }
+  await page.waitForTimeout(500)
+  await page.screenshot({ path: `${dir}/app-media.png` })
+}
+
 /**
  * The flow editor, empty-handed if the instance has no flows yet: seeds one
  * with a node so the canvas and the node panel are both worth looking at.

frontend/src/client/schemas.gen.ts

@@ -373,7 +373,7 @@ export const ChannelSchema = {
 
 export const DTypeSchema = {
     type: 'string',
-    enum: ['float', 'int', 'str', 'bool', 'json', 'series', 'record', 'list', 'artifact'],
+    enum: ['float', 'int', 'str', 'bool', 'json', 'series', 'record', 'list', 'artifact', 'image', 'audio', 'video'],
     title: 'DType',
     description: `Serializable payload types.
 
@@ -388,7 +388,14 @@ them. That keeps everything on the wire JSON, which is what the state
 backend, the queue and the worker protocol all rely on, and it means a
 thirty-megabyte checkpoint never sits in Redis. Inline codecs would only be
 needed for payloads too small to be worth a round trip, and nothing asks
-for that yet.`
+for that yet.
+
+\`\`image\`\`, \`\`audio\`\` and \`\`video\`\` are that same reference narrowed to a
+media family, so a speech recogniser declares what it eats rather than
+taking any bytes at all and finding out. Bytes still never travel as a
+message: a camera publishes one reference per frame, a microphone one per
+chunk. A reference may carry a \`\`meta\`\` dict — sample rate, dimensions, a
+sequence number — which nothing here interprets.`
 } as const;
 
 export const DashboardDef_InputSchema = {
@@ -3327,7 +3334,7 @@ export const WidgetDefSchema = {
         },
         type: {
             type: 'string',
-            enum: ['stat', 'gauge', 'chart', 'markdown', 'agenda', 'notification', 'bar', 'icon', 'forecast', 'clock', 'button', 'switch', 'slider', 'input', 'dropdown', 'color'],
+            enum: ['stat', 'gauge', 'chart', 'markdown', 'agenda', 'notification', 'bar', 'icon', 'forecast', 'clock', 'media', 'button', 'switch', 'slider', 'input', 'dropdown', 'color'],
             title: 'Type'
         },
         title: {

frontend/src/client/sdk.gen.ts

@@ -65,6 +65,10 @@ export class ArtifactsService {
     /**
      * Put Artifact
      * Store the request body and answer with the reference to it.
+     *
+     * Spooled to disk as it arrives rather than buffered: a video segment is as
+     * legitimate a body here as a checkpoint, and neither should have to fit in
+     * memory twice.
      * @param data The data for the request.
      * @param data.name
      * @param data.mediaType
@@ -87,9 +91,15 @@ export class ArtifactsService {
     
     /**
      * Get Artifact
-     * Stream one artifact back.
+     * Serve one artifact back.
+     *
+     * The caller passes the media type off the reference it holds, which is what
+     * lets a browser play a clip rather than download it; the store itself keeps
+     * only bytes. Ranged requests are answered because an audio or video element
+     * scrubbing through a file asks for them.
      * @param data The data for the request.
      * @param data.digest
+     * @param data.mediaType
      * @returns unknown Successful Response
      * @throws ApiError
      */
@@ -100,6 +110,9 @@ export class ArtifactsService {
             path: {
                 digest: data.digest
             },
+            query: {
+                media_type: data.mediaType
+            },
             errors: {
                 422: 'Validation Error'
             }

frontend/src/client/types.gen.ts

@@ -182,8 +182,15 @@ export type DeadLetter = {
  * thirty-megabyte checkpoint never sits in Redis. Inline codecs would only be
  * needed for payloads too small to be worth a round trip, and nothing asks
  * for that yet.
+ *
+ * ``image``, ``audio`` and ``video`` are that same reference narrowed to a
+ * media family, so a speech recogniser declares what it eats rather than
+ * taking any bytes at all and finding out. Bytes still never travel as a
+ * message: a camera publishes one reference per frame, a microphone one per
+ * chunk. A reference may carry a ``meta`` dict — sample rate, dimensions, a
+ * sequence number — which nothing here interprets.
  */
-export type DType = 'float' | 'int' | 'str' | 'bool' | 'json' | 'series' | 'record' | 'list' | 'artifact';
+export type DType = 'float' | 'int' | 'str' | 'bool' | 'json' | 'series' | 'record' | 'list' | 'artifact' | 'image' | 'audio' | 'video';
 
 /**
  * Something wired into this flow that is not a node in it.
@@ -1156,7 +1163,7 @@ export type ValidationResult = {
  */
 export type WidgetDef = {
     id: string;
-    type: 'stat' | 'gauge' | 'chart' | 'markdown' | 'agenda' | 'notification' | 'bar' | 'icon' | 'forecast' | 'clock' | 'button' | 'switch' | 'slider' | 'input' | 'dropdown' | 'color';
+    type: 'stat' | 'gauge' | 'chart' | 'markdown' | 'agenda' | 'notification' | 'bar' | 'icon' | 'forecast' | 'clock' | 'media' | 'button' | 'switch' | 'slider' | 'input' | 'dropdown' | 'color';
     title?: string;
     layout?: {
         [key: string]: Placement;
@@ -1166,7 +1173,7 @@ export type WidgetDef = {
     };
 };
 
-export type type = 'stat' | 'gauge' | 'chart' | 'markdown' | 'agenda' | 'notification' | 'bar' | 'icon' | 'forecast' | 'clock' | 'button' | 'switch' | 'slider' | 'input' | 'dropdown' | 'color';
+export type type = 'stat' | 'gauge' | 'chart' | 'markdown' | 'agenda' | 'notification' | 'bar' | 'icon' | 'forecast' | 'clock' | 'media' | 'button' | 'switch' | 'slider' | 'input' | 'dropdown' | 'color';
 
 export type WorkerInfo = {
     name: string;
@@ -1202,6 +1209,7 @@ export type ArtifactsPutArtifactResponse = (ArtifactRef);
 
 export type ArtifactsGetArtifactData = {
     digest: string;
+    mediaType?: string;
 };
 
 export type ArtifactsGetArtifactResponse = (unknown);

frontend/src/components/Dashboard/MediaWidget.tsx

@@ -0,0 +1,153 @@
+import { useEffect, useState } from "react"
+
+import { OpenAPI } from "@/client"
+import { apiToken } from "@/lib/portal"
+import { cn } from "@/lib/utils"
+import { useBoundValue } from "./dataContext"
+import { config, text } from "./ui/core/config"
+import type { WidgetProps } from "./widgets"
+
+/** An artifact reference, as a message carries one. */
+type MediaRef = {
+  digest?: string
+  media_type?: string
+  name?: string
+  size?: number
+}
+
+const isRef = (value: unknown): value is MediaRef =>
+  typeof value === "object" &&
+  value !== null &&
+  typeof (value as MediaRef).digest === "string" &&
+  (value as MediaRef).digest!.startsWith("sha256:")
+
+/**
+ * A local URL for an artifact's bytes, refreshed whenever the digest changes.
+ *
+ * Not the endpoint itself: `/artifacts/{digest}` takes a bearer token, and no
+ * `<img>` or `<audio>` can carry a header. So the bytes come through fetch and
+ * are handed to the element as an object URL — which also means playback never
+ * asks the server for a range, since the blob is already here.
+ *
+ * The URL is revoked when it is replaced, or the tab would hold every frame a
+ * camera has ever sent for as long as the page is open.
+ */
+function useArtifactUrl(ref: MediaRef | null): string {
+  const digest = ref?.digest ?? ""
+  const mediaType = ref?.media_type ?? ""
+  const [url, setUrl] = useState("")
+
+  useEffect(() => {
+    if (!digest) {
+      setUrl("")
+      return
+    }
+    let live = true
+    let made = ""
+    const token = apiToken()
+    const query = mediaType
+      ? `?media_type=${encodeURIComponent(mediaType)}`
+      : ""
+    fetch(`${OpenAPI.BASE}/api/v1/artifacts/${digest}${query}`, {
+      headers: token ? { Authorization: `Bearer ${token}` } : {},
+    })
+      .then((answer) => (answer.ok ? answer.blob() : Promise.reject(answer)))
+      .then((blob) => {
+        if (!live) return
+        made = URL.createObjectURL(blob)
+        setUrl(made)
+      })
+      .catch(() => {
+        if (live) setUrl("")
+      })
+    return () => {
+      live = false
+      if (made) URL.revokeObjectURL(made)
+    }
+  }, [digest, mediaType])
+
+  return url
+}
+
+/**
+ * What a message's bytes look like: a frame, a clip, a segment.
+ *
+ * Media never travels as a message — the reference does, and the bytes are
+ * fetched from the artifact store. What that means for a wall panel is a
+ * refresh per published frame, which suits a camera glancing every few seconds
+ * rather than a live view; for that, point `stream_url` at whatever the camera
+ * already serves and the browser plays it directly.
+ */
+export function MediaWidget({ widget }: WidgetProps) {
+  const cfg = config(widget)
+  const message = text(cfg.message)
+  const stream = text(cfg.stream_url)
+  const live = useBoundValue(message || undefined)
+  const value = live?.value
+  const ref = isRef(value) ? value : null
+  const url = useArtifactUrl(ref)
+
+  const kind = (ref?.media_type ?? text(cfg.dtype)).split("/")[0]
+  const fit = text(cfg.fit) === "contain" ? "object-contain" : "object-cover"
+  const label = widget.title || ref?.name || message
+
+  // A camera serving its own stream is played from source: the engine carries
+  // references at whatever rate the flow publishes them, which is not video.
+  if (stream && kind !== "audio") {
+    return (
+      <img
+        src={stream}
+        alt={label}
+        className={cn("h-full w-full rounded-md", fit)}
+      />
+    )
+  }
+
+  if (!message && !stream) {
+    return <p className="text-muted-foreground">Pick a message.</p>
+  }
+  if (!ref) {
+    return <p className="text-muted-foreground">Nothing published yet.</p>
+  }
+  if (!url) {
+    return <p className="text-muted-foreground">Loading…</p>
+  }
+
+  if (kind === "audio") {
+    return (
+      <audio
+        // Keyed on the digest so a new clip replaces the element rather than
+        // leaving the last one's playhead on it.
+        key={ref.digest}
+        src={url}
+        controls
+        autoPlay={Boolean(cfg.autoplay)}
+        className="w-full"
+      >
+        <track kind="captions" />
+      </audio>
+    )
+  }
+
+  if (kind === "video") {
+    return (
+      <video
+        key={ref.digest}
+        src={url}
+        controls
+        autoPlay={Boolean(cfg.autoplay)}
+        className={cn("h-full w-full rounded-md", fit)}
+      >
+        <track kind="captions" />
+      </video>
+    )
+  }
+
+  return (
+    <img
+      src={url}
+      alt={label}
+      className={cn("h-full w-full rounded-md", fit)}
+    />
+  )
+}

frontend/src/components/Dashboard/panels.tsx

@@ -1231,6 +1231,53 @@ export function WidgetPanel({
           </div>
         ) : null}
 
+        {widget.type === "media" ? (
+          <div className="grid gap-3">
+            <div className="grid gap-1.5">
+              <Label className="text-sm font-normal">Fills the tile</Label>
+              <Segmented
+                value={str(cfg.fit) || "cover"}
+                options={[
+                  ["cover", "Crop"],
+                  ["contain", "Fit"],
+                ]}
+                label="How the picture fills its tile"
+                testId="widget-fit"
+                onChange={(fit) => set({ fit })}
+              />
+            </div>
+            <div className="grid gap-1.5">
+              <Label className="text-sm font-normal" htmlFor="media-stream">
+                Live stream
+              </Label>
+              <Input
+                id="media-stream"
+                value={str(cfg.stream_url)}
+                placeholder="https://camera.local/stream.mjpg"
+                onChange={(event) => set({ stream_url: event.target.value })}
+              />
+              <p className="text-xs text-muted-foreground">
+                A camera's own stream, played straight from it. Messages carry a
+                frame at a time, which suits a glance every few seconds rather
+                than live video.
+              </p>
+            </div>
+            <div className="flex items-center justify-between gap-2 text-sm">
+              Play as it arrives
+              <Switch
+                checked={Boolean(cfg.autoplay)}
+                aria-label="Play as it arrives"
+                data-testid="widget-autoplay"
+                onCheckedChange={(autoplay) => set({ autoplay })}
+              />
+            </div>
+            <p className="text-xs text-muted-foreground">
+              A browser only plays sound by itself once someone has touched the
+              page, so a screen nobody has tapped stays silent.
+            </p>
+          </div>
+        ) : null}
+
         {widget.type === "switch" || widget.type === "dropdown" ? (
           <div className="grid gap-1.5">
             <Label className="text-sm font-normal">Style</Label>

frontend/src/components/Dashboard/widgets.tsx

@@ -12,6 +12,7 @@ import { useBoundValue } from "./dataContext"
 import "./dashboard.css"
 import { ForecastWidget } from "./ForecastWidget"
 import { IconWidget } from "./IconWidget"
+import { MediaWidget } from "./MediaWidget"
 import { usePublish } from "./publish"
 import { useUi } from "./ui"
 import { COLOR_DTYPES, colorFormatOf } from "./ui/core/color"
@@ -51,6 +52,10 @@ export const WIDGET_DTYPES: Partial<Record<WidgetKind, string[]>> = {
   // Either shape a colour can travel as; its `format` decides which of the two
   // this widget means, which `widgetIssue` holds the binding to.
   color: ["list", "str"],
+  // A camera frame, a clip, a segment. What it draws follows the type it is
+  // bound to; a plain artifact is taken as well, since the media type on the
+  // reference is what says what the bytes are.
+  media: ["image", "audio", "video", "artifact"],
   // An icon maps weather strings, bool hints and numbers alike, and a clock
   // binds nothing at all, so neither has a row to be held to.
 }
@@ -72,6 +77,7 @@ export const WIDGET_LABELS: Record<WidgetKind, string> = {
   icon: "Icon",
   forecast: "Forecast",
   clock: "Clock",
+  media: "Media",
   button: "Button",
   switch: "Switch",
   slider: "Slider",
@@ -92,6 +98,7 @@ export const WIDGET_SIZES: Record<WidgetKind, { w: number; h: number }> = {
   icon: { w: 2, h: 2 },
   forecast: { w: 6, h: 2 },
   clock: { w: 3, h: 2 },
+  media: { w: 4, h: 4 },
   button: { w: 3, h: 2 },
   switch: { w: 3, h: 2 },
   slider: { w: 4, h: 2 },
@@ -176,6 +183,12 @@ export function widgetIssue(widget: WidgetDef): string | null {
     return null
   }
 
+  // A media tile playing a camera's own stream binds no message: the browser
+  // fetches it from the source, and the engine is not in the way of it.
+  if (widget.type === "media" && text(cfg.stream_url) && !text(cfg.message)) {
+    return null
+  }
+
   const input = INPUT_WIDGETS.has(widget.type)
   const bound = text(cfg[input ? "target" : "message"])
   if (!bound) {
@@ -598,6 +611,7 @@ const RENDERERS: Partial<
   icon: IconWidget,
   forecast: ForecastWidget,
   clock: ClockWidget,
+  media: MediaWidget,
   button: ButtonWidget,
   switch: SwitchWidget,
   slider: SliderWidget,

frontend/src/components/Flow/NodePanel.tsx

@@ -62,6 +62,9 @@ export const DTYPES: DType[] = [
   "record",
   "list",
   "artifact",
+  "image",
+  "audio",
+  "video",
 ]
 
 /** What a list may hold. One declared level: no list of lists. */
@@ -938,8 +941,12 @@ const PLACEHOLDER: Record<DType, string> = {
   record: "{}",
   list: "[]",
   // Bytes never travel as a message: the node stores them and returns what
-  // the next one opens.
+  // the next one opens. A media port is the same reference, saying what kind
+  // of bytes they are.
   artifact: 'fluksio.save_artifact(b"", "result.bin")',
+  image: 'fluksio.save_artifact(b"", "frame.png", media_type="image/png")',
+  audio: 'fluksio.save_artifact(b"", "clip.wav", media_type="audio/wav")',
+  video: 'fluksio.save_artifact(b"", "clip.mp4", media_type="video/mp4")',
 }
 
 const SCAFFOLD_DOC =

frontend/src/components/Flow/ValuePreview.tsx

@@ -42,7 +42,16 @@ function summarize(value: object, dtype?: DType): string {
     const name =
       typeof record.name === "string" && record.name ? record.name : ""
     const size = typeof record.size === "number" ? `${si(record.size)}B` : ""
-    return ["artifact", name, size].filter(Boolean).join(" · ")
+    // The media type when there is one: what kind of bytes these are is the
+    // first thing worth knowing about a frame or a clip, and it is what the
+    // port's own type was declared against.
+    const media =
+      typeof record.media_type === "string" &&
+      record.media_type &&
+      record.media_type !== "application/octet-stream"
+        ? record.media_type
+        : "artifact"
+    return [media, name, size].filter(Boolean).join(" · ")
   }
 
   if (Array.isArray(record.lines)) {