stroblmeandClaude Opus 5
d471614e6a
The rate the media dtypes could carry was one frame every second or two: each
was a file on the data volume, an event on the socket, and a request back for
the bytes. This closes both halves of that, and they are one feature.
`save_artifact(..., volatile=True)` writes to a `VolatileStore` — the same
content-addressed store, in `/dev/shm`, bounded by size with the oldest falling
out (`ARTIFACT_VOLATILE_BYTES`, 48 MB under the container's raised `shm_size`).
Nothing sweeps it: a frame nobody kept is not worth walking the store to find.
`ArtifactStore.path` falls through to it, which is what lets a volatile frame be
an ordinary reference everywhere else — the dtype check, a panel's digest scope,
`load_artifact` in a node, and the widget's own fetch all work on one unchanged.
`adopt` copies one into the store when a run records it, so "returned media is
kept, emitted media is not" stays true.
The bytes then go down the flows websocket as a length-prefixed binary frame,
sent just ahead of the `message_value` naming them, so a tile has the frame when
it hears the value moved. Nothing is pushed unasked: a client names the messages
it is drawing (`{"type":"media","names":[…]}`), a panel's list is intersected
with the scope it already had, and only the newest frame per name in a batch is
sent — a client that fell behind is not handed frames it would draw over. The
tunnel relays text only, so a screen reached through a portal falls back to
fetching, which is why the rate table now has two rows.
Around the edges: the remote worker's fetch cache is bounded at last
(`FLUKSIO_ARTIFACT_CACHE_BYTES`), since content addressing means nothing in it
ever expires and a media stream fills it with chunks nothing asks for twice; a
port carrying an image draws the frame in the node panel rather than only
saying `image/png · frame.png · 1.79kB`; and an edge chip says that much instead
of a line of hash. The media screenshot stops waiting for `networkidle` — a
camera is a socket that never goes quiet, which is the point of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YC4u66vjzW54fnHu5Juhh9
89 lines
2.5 KiB
TypeScript
89 lines
2.5 KiB
TypeScript
import { isRef, useArtifactUrl } from "@/lib/media"
|
|
import { cn } from "@/lib/utils"
|
|
import { useBoundValue } from "./dataContext"
|
|
import { config, text } from "./ui/core/config"
|
|
import type { WidgetProps } from "./widgets"
|
|
|
|
/**
|
|
* What a message's bytes look like: a frame, a clip, a segment.
|
|
*
|
|
* Media never travels as a message — the reference does. The bytes come down
|
|
* the socket in front of it where the engine is holding the frame in memory,
|
|
* and are fetched from the artifact store otherwise; either way this tile only
|
|
* asks for what it is drawing. A camera serving its own stream is still played
|
|
* from source: point `stream_url` at it and the browser does the work.
|
|
*/
|
|
export function MediaWidget({ widget }: WidgetProps) {
|
|
const cfg = config(widget)
|
|
const message = text(cfg.message)
|
|
const stream = text(cfg.stream_url)
|
|
const live = useBoundValue(message || undefined)
|
|
const value = live?.value
|
|
const ref = isRef(value) ? value : null
|
|
const url = useArtifactUrl(ref, message || undefined)
|
|
|
|
const kind = (ref?.media_type ?? text(cfg.dtype)).split("/")[0]
|
|
const fit = text(cfg.fit) === "contain" ? "object-contain" : "object-cover"
|
|
const label = widget.title || ref?.name || message
|
|
|
|
// A camera serving its own stream is played from source: the engine carries
|
|
// references at whatever rate the flow publishes them, which is not video.
|
|
if (stream && kind !== "audio") {
|
|
return (
|
|
<img
|
|
src={stream}
|
|
alt={label}
|
|
className={cn("h-full w-full rounded-md", fit)}
|
|
/>
|
|
)
|
|
}
|
|
|
|
if (!message && !stream) {
|
|
return <p className="text-muted-foreground">Pick a message.</p>
|
|
}
|
|
if (!ref) {
|
|
return <p className="text-muted-foreground">Nothing published yet.</p>
|
|
}
|
|
if (!url) {
|
|
return <p className="text-muted-foreground">Loading…</p>
|
|
}
|
|
|
|
if (kind === "audio") {
|
|
return (
|
|
<audio
|
|
// Keyed on the digest so a new clip replaces the element rather than
|
|
// leaving the last one's playhead on it.
|
|
key={ref.digest}
|
|
src={url}
|
|
controls
|
|
autoPlay={Boolean(cfg.autoplay)}
|
|
className="w-full"
|
|
>
|
|
<track kind="captions" />
|
|
</audio>
|
|
)
|
|
}
|
|
|
|
if (kind === "video") {
|
|
return (
|
|
<video
|
|
key={ref.digest}
|
|
src={url}
|
|
controls
|
|
autoPlay={Boolean(cfg.autoplay)}
|
|
className={cn("h-full w-full rounded-md", fit)}
|
|
>
|
|
<track kind="captions" />
|
|
</video>
|
|
)
|
|
}
|
|
|
|
return (
|
|
<img
|
|
src={url}
|
|
alt={label}
|
|
className={cn("h-full w-full rounded-md", fit)}
|
|
/>
|
|
)
|
|
}
|