Push a frame instead of storing and fetching it

The rate the media dtypes could carry was one frame every second or two: each
was a file on the data volume, an event on the socket, and a request back for
the bytes. This closes both halves of that, and they are one feature.

`save_artifact(..., volatile=True)` writes to a `VolatileStore` — the same
content-addressed store, in `/dev/shm`, bounded by size with the oldest falling
out (`ARTIFACT_VOLATILE_BYTES`, 48 MB under the container's raised `shm_size`).
Nothing sweeps it: a frame nobody kept is not worth walking the store to find.
`ArtifactStore.path` falls through to it, which is what lets a volatile frame be
an ordinary reference everywhere else — the dtype check, a panel's digest scope,
`load_artifact` in a node, and the widget's own fetch all work on one unchanged.
`adopt` copies one into the store when a run records it, so "returned media is
kept, emitted media is not" stays true.

The bytes then go down the flows websocket as a length-prefixed binary frame,
sent just ahead of the `message_value` naming them, so a tile has the frame when
it hears the value moved. Nothing is pushed unasked: a client names the messages
it is drawing (`{"type":"media","names":[…]}`), a panel's list is intersected
with the scope it already had, and only the newest frame per name in a batch is
sent — a client that fell behind is not handed frames it would draw over. The
tunnel relays text only, so a screen reached through a portal falls back to
fetching, which is why the rate table now has two rows.

Around the edges: the remote worker's fetch cache is bounded at last
(`FLUKSIO_ARTIFACT_CACHE_BYTES`), since content addressing means nothing in it
ever expires and a media stream fills it with chunks nothing asks for twice; a
port carrying an image draws the frame in the node panel rather than only
saying `image/png · frame.png · 1.79kB`; and an edge chip says that much instead
of a line of hash. The media screenshot stops waiting for `networkidle` — a
camera is a socket that never goes quiet, which is the point of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YC4u66vjzW54fnHu5Juhh9
This commit is contained in:
2026-09-02 10:15:14 +02:00
co-authored by Claude Opus 5
parent 518231aa39
commit d471614e6a
29 changed files with 1101 additions and 147 deletions
@@ -1,82 +1,17 @@
import { useEffect, useState } from "react"
import { OpenAPI } from "@/client"
import { apiToken } from "@/lib/portal"
import { isRef, useArtifactUrl } from "@/lib/media"
import { cn } from "@/lib/utils"
import { useBoundValue } from "./dataContext"
import { config, text } from "./ui/core/config"
import type { WidgetProps } from "./widgets"
/** An artifact reference, as a message carries one. */
type MediaRef = {
digest?: string
media_type?: string
name?: string
size?: number
}
const isRef = (value: unknown): value is MediaRef =>
typeof value === "object" &&
value !== null &&
typeof (value as MediaRef).digest === "string" &&
(value as MediaRef).digest!.startsWith("sha256:")
/**
* A local URL for an artifact's bytes, refreshed whenever the digest changes.
*
* Not the endpoint itself: `/artifacts/{digest}` takes a bearer token, and no
* `<img>` or `<audio>` can carry a header. So the bytes come through fetch and
* are handed to the element as an object URL — which also means playback never
* asks the server for a range, since the blob is already here.
*
* The URL is revoked when it is replaced, or the tab would hold every frame a
* camera has ever sent for as long as the page is open.
*/
function useArtifactUrl(ref: MediaRef | null): string {
const digest = ref?.digest ?? ""
const mediaType = ref?.media_type ?? ""
const [url, setUrl] = useState("")
useEffect(() => {
if (!digest) {
setUrl("")
return
}
let live = true
let made = ""
const token = apiToken()
const query = mediaType
? `?media_type=${encodeURIComponent(mediaType)}`
: ""
fetch(`${OpenAPI.BASE}/api/v1/artifacts/${digest}${query}`, {
headers: token ? { Authorization: `Bearer ${token}` } : {},
})
.then((answer) => (answer.ok ? answer.blob() : Promise.reject(answer)))
.then((blob) => {
if (!live) return
made = URL.createObjectURL(blob)
setUrl(made)
})
.catch(() => {
if (live) setUrl("")
})
return () => {
live = false
if (made) URL.revokeObjectURL(made)
}
}, [digest, mediaType])
return url
}
/**
* What a message's bytes look like: a frame, a clip, a segment.
*
* Media never travels as a message — the reference does, and the bytes are
* fetched from the artifact store. What that means for a wall panel is a
* refresh per published frame, which suits a camera glancing every few seconds
* rather than a live view; for that, point `stream_url` at whatever the camera
* already serves and the browser plays it directly.
* Media never travels as a message — the reference does. The bytes come down
* the socket in front of it where the engine is holding the frame in memory,
* and are fetched from the artifact store otherwise; either way this tile only
* asks for what it is drawing. A camera serving its own stream is still played
* from source: point `stream_url` at it and the browser does the work.
*/
export function MediaWidget({ widget }: WidgetProps) {
const cfg = config(widget)
@@ -85,7 +20,7 @@ export function MediaWidget({ widget }: WidgetProps) {
const live = useBoundValue(message || undefined)
const value = live?.value
const ref = isRef(value) ? value : null
const url = useArtifactUrl(ref)
const url = useArtifactUrl(ref, message || undefined)
const kind = (ref?.media_type ?? text(cfg.dtype)).split("/")[0]
const fit = text(cfg.fit) === "contain" ? "object-contain" : "object-cover"