Push a frame instead of storing and fetching it
The rate the media dtypes could carry was one frame every second or two: each
was a file on the data volume, an event on the socket, and a request back for
the bytes. This closes both halves of that, and they are one feature.
`save_artifact(..., volatile=True)` writes to a `VolatileStore` — the same
content-addressed store, in `/dev/shm`, bounded by size with the oldest falling
out (`ARTIFACT_VOLATILE_BYTES`, 48 MB under the container's raised `shm_size`).
Nothing sweeps it: a frame nobody kept is not worth walking the store to find.
`ArtifactStore.path` falls through to it, which is what lets a volatile frame be
an ordinary reference everywhere else — the dtype check, a panel's digest scope,
`load_artifact` in a node, and the widget's own fetch all work on one unchanged.
`adopt` copies one into the store when a run records it, so "returned media is
kept, emitted media is not" stays true.
The bytes then go down the flows websocket as a length-prefixed binary frame,
sent just ahead of the `message_value` naming them, so a tile has the frame when
it hears the value moved. Nothing is pushed unasked: a client names the messages
it is drawing (`{"type":"media","names":[…]}`), a panel's list is intersected
with the scope it already had, and only the newest frame per name in a batch is
sent — a client that fell behind is not handed frames it would draw over. The
tunnel relays text only, so a screen reached through a portal falls back to
fetching, which is why the rate table now has two rows.
Around the edges: the remote worker's fetch cache is bounded at last
(`FLUKSIO_ARTIFACT_CACHE_BYTES`), since content addressing means nothing in it
ever expires and a media stream fills it with chunks nothing asks for twice; a
port carrying an image draws the frame in the node panel rather than only
saying `image/png · frame.png · 1.79kB`; and an edge chip says that much instead
of a line of hash. The media screenshot stops waiting for `networkidle` — a
camera is a socket that never goes quiet, which is the point of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YC4u66vjzW54fnHu5Juhh9
This commit is contained in:
@@ -199,13 +199,18 @@ async function captureDashboards(page, dir) {
|
||||
*
|
||||
* Skipped unless the media example is seeded (root `make seed-example-media`),
|
||||
* since it is the one shot that needs a source of frames. The bytes arrive as
|
||||
* a blob — the tile fetches them with the session's credential, which no `img`
|
||||
* could carry on its own — so a `blob:` source is the proof the whole path ran
|
||||
* rather than that a picture is merely present.
|
||||
* a blob — pushed down the socket, or fetched with the session's credential,
|
||||
* which no `img` could carry on its own — so a `blob:` source is the proof the
|
||||
* whole path ran rather than that a picture is merely present.
|
||||
*
|
||||
* The one page that cannot wait for `networkidle`: a camera publishing several
|
||||
* frames a second is a socket that never goes quiet, which is the point of it.
|
||||
* The blob source below is a stronger wait anyway — it says a frame arrived,
|
||||
* where idleness only ever said the page stopped asking.
|
||||
*/
|
||||
async function captureMedia(page, dir) {
|
||||
const answer = await page.goto(`${APP_URL}/view/camera`, {
|
||||
waitUntil: "networkidle",
|
||||
waitUntil: "domcontentloaded",
|
||||
})
|
||||
if (!answer?.ok()) return
|
||||
|
||||
|
||||
@@ -1,82 +1,17 @@
|
||||
import { useEffect, useState } from "react"
|
||||
|
||||
import { OpenAPI } from "@/client"
|
||||
import { apiToken } from "@/lib/portal"
|
||||
import { isRef, useArtifactUrl } from "@/lib/media"
|
||||
import { cn } from "@/lib/utils"
|
||||
import { useBoundValue } from "./dataContext"
|
||||
import { config, text } from "./ui/core/config"
|
||||
import type { WidgetProps } from "./widgets"
|
||||
|
||||
/** An artifact reference, as a message carries one. */
|
||||
type MediaRef = {
|
||||
digest?: string
|
||||
media_type?: string
|
||||
name?: string
|
||||
size?: number
|
||||
}
|
||||
|
||||
const isRef = (value: unknown): value is MediaRef =>
|
||||
typeof value === "object" &&
|
||||
value !== null &&
|
||||
typeof (value as MediaRef).digest === "string" &&
|
||||
(value as MediaRef).digest!.startsWith("sha256:")
|
||||
|
||||
/**
|
||||
* A local URL for an artifact's bytes, refreshed whenever the digest changes.
|
||||
*
|
||||
* Not the endpoint itself: `/artifacts/{digest}` takes a bearer token, and no
|
||||
* `<img>` or `<audio>` can carry a header. So the bytes come through fetch and
|
||||
* are handed to the element as an object URL — which also means playback never
|
||||
* asks the server for a range, since the blob is already here.
|
||||
*
|
||||
* The URL is revoked when it is replaced, or the tab would hold every frame a
|
||||
* camera has ever sent for as long as the page is open.
|
||||
*/
|
||||
function useArtifactUrl(ref: MediaRef | null): string {
|
||||
const digest = ref?.digest ?? ""
|
||||
const mediaType = ref?.media_type ?? ""
|
||||
const [url, setUrl] = useState("")
|
||||
|
||||
useEffect(() => {
|
||||
if (!digest) {
|
||||
setUrl("")
|
||||
return
|
||||
}
|
||||
let live = true
|
||||
let made = ""
|
||||
const token = apiToken()
|
||||
const query = mediaType
|
||||
? `?media_type=${encodeURIComponent(mediaType)}`
|
||||
: ""
|
||||
fetch(`${OpenAPI.BASE}/api/v1/artifacts/${digest}${query}`, {
|
||||
headers: token ? { Authorization: `Bearer ${token}` } : {},
|
||||
})
|
||||
.then((answer) => (answer.ok ? answer.blob() : Promise.reject(answer)))
|
||||
.then((blob) => {
|
||||
if (!live) return
|
||||
made = URL.createObjectURL(blob)
|
||||
setUrl(made)
|
||||
})
|
||||
.catch(() => {
|
||||
if (live) setUrl("")
|
||||
})
|
||||
return () => {
|
||||
live = false
|
||||
if (made) URL.revokeObjectURL(made)
|
||||
}
|
||||
}, [digest, mediaType])
|
||||
|
||||
return url
|
||||
}
|
||||
|
||||
/**
|
||||
* What a message's bytes look like: a frame, a clip, a segment.
|
||||
*
|
||||
* Media never travels as a message — the reference does, and the bytes are
|
||||
* fetched from the artifact store. What that means for a wall panel is a
|
||||
* refresh per published frame, which suits a camera glancing every few seconds
|
||||
* rather than a live view; for that, point `stream_url` at whatever the camera
|
||||
* already serves and the browser plays it directly.
|
||||
* Media never travels as a message — the reference does. The bytes come down
|
||||
* the socket in front of it where the engine is holding the frame in memory,
|
||||
* and are fetched from the artifact store otherwise; either way this tile only
|
||||
* asks for what it is drawing. A camera serving its own stream is still played
|
||||
* from source: point `stream_url` at it and the browser does the work.
|
||||
*/
|
||||
export function MediaWidget({ widget }: WidgetProps) {
|
||||
const cfg = config(widget)
|
||||
@@ -85,7 +20,7 @@ export function MediaWidget({ widget }: WidgetProps) {
|
||||
const live = useBoundValue(message || undefined)
|
||||
const value = live?.value
|
||||
const ref = isRef(value) ? value : null
|
||||
const url = useArtifactUrl(ref)
|
||||
const url = useArtifactUrl(ref, message || undefined)
|
||||
|
||||
const kind = (ref?.media_type ?? text(cfg.dtype)).split("/")[0]
|
||||
const fit = text(cfg.fit) === "contain" ? "object-contain" : "object-cover"
|
||||
|
||||
@@ -159,7 +159,12 @@ export function EdgeInspector({
|
||||
Nothing has come through yet. Run the flow to see a value here.
|
||||
</p>
|
||||
) : scalar === null ? (
|
||||
<ValuePreview value={live.value} className="mt-2" defaultOpen />
|
||||
<ValuePreview
|
||||
value={live.value}
|
||||
name={edge.message}
|
||||
className="mt-2"
|
||||
defaultOpen
|
||||
/>
|
||||
) : null}
|
||||
</PopoverContent>
|
||||
</Popover>
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
} from "@xyflow/react"
|
||||
import { memo, useEffect, useRef, useState } from "react"
|
||||
|
||||
import { describeArtifact, isRef } from "@/lib/media"
|
||||
import { duration } from "@/lib/motion"
|
||||
import { cn } from "@/lib/utils"
|
||||
import type { FlowEdgeData } from "./deriveEdges"
|
||||
@@ -66,6 +67,9 @@ function formatValue(value: unknown): string {
|
||||
return Number.isInteger(value) ? String(value) : value.toFixed(2)
|
||||
}
|
||||
if (typeof value === "string") return value
|
||||
// A media reference serialised whole is a line of hash: what belongs in a
|
||||
// chip this size is what kind of bytes crossed the edge.
|
||||
if (isRef(value)) return describeArtifact(value as Record<string, unknown>)
|
||||
return JSON.stringify(value) ?? ""
|
||||
}
|
||||
|
||||
|
||||
@@ -119,7 +119,13 @@ export function MessageSparkline({
|
||||
</p>
|
||||
)
|
||||
}
|
||||
return <ValuePreview value={live.value} dtype={dtype} />
|
||||
return (
|
||||
<ValuePreview
|
||||
value={live.value}
|
||||
name={qualify(flow, message)}
|
||||
dtype={dtype}
|
||||
/>
|
||||
)
|
||||
}
|
||||
|
||||
// The value is live here, so the dot on the newest reading is earned. The
|
||||
|
||||
@@ -3,7 +3,8 @@ import { useState } from "react"
|
||||
|
||||
import type { DType } from "@/client"
|
||||
import { Marquee } from "@/components/Common/Marquee"
|
||||
import { cn, si } from "@/lib/utils"
|
||||
import { describeArtifact, isRef, useArtifactUrl } from "@/lib/media"
|
||||
import { cn } from "@/lib/utils"
|
||||
|
||||
/**
|
||||
* How much of a structured value is worth unfolding in a side panel.
|
||||
@@ -19,11 +20,6 @@ function count(n: number, one: string, many = `${one}s`): string {
|
||||
return `${n} ${n === 1 ? one : many}`
|
||||
}
|
||||
|
||||
/** Whether this is an artifact reference rather than data of its own. */
|
||||
function isArtifact(value: Record<string, unknown>): boolean {
|
||||
return typeof value.digest === "string" && value.digest.startsWith("sha256:")
|
||||
}
|
||||
|
||||
/**
|
||||
* What a structured value *is*, in the space a value would have taken.
|
||||
*
|
||||
@@ -38,21 +34,7 @@ function summarize(value: object, dtype?: DType): string {
|
||||
}
|
||||
|
||||
const record = value as Record<string, unknown>
|
||||
if (isArtifact(record)) {
|
||||
const name =
|
||||
typeof record.name === "string" && record.name ? record.name : ""
|
||||
const size = typeof record.size === "number" ? `${si(record.size)}B` : ""
|
||||
// The media type when there is one: what kind of bytes these are is the
|
||||
// first thing worth knowing about a frame or a clip, and it is what the
|
||||
// port's own type was declared against.
|
||||
const media =
|
||||
typeof record.media_type === "string" &&
|
||||
record.media_type &&
|
||||
record.media_type !== "application/octet-stream"
|
||||
? record.media_type
|
||||
: "artifact"
|
||||
return [media, name, size].filter(Boolean).join(" · ")
|
||||
}
|
||||
if (isRef(record)) return describeArtifact(record)
|
||||
|
||||
if (Array.isArray(record.lines)) {
|
||||
const points = record.lines.reduce(
|
||||
@@ -77,13 +59,39 @@ function summarize(value: object, dtype?: DType): string {
|
||||
* which is what keeps a panel four hundred pixels wide from being pushed open
|
||||
* by one checkpoint reference.
|
||||
*/
|
||||
/**
|
||||
* The frame itself, where the value is one.
|
||||
*
|
||||
* A port carrying an image says `image/png · frame.png · 48kB`, which is the
|
||||
* right answer while wiring and the wrong one while pointing a camera. Drawn
|
||||
* small: this is a side panel, and the tile is where a frame is looked at.
|
||||
*/
|
||||
function Thumbnail({ value, name }: { value: object; name?: string }) {
|
||||
const url = useArtifactUrl(value, name)
|
||||
if (!url) return null
|
||||
return (
|
||||
<img
|
||||
src={url}
|
||||
alt={String((value as { name?: string }).name ?? "frame")}
|
||||
className="mt-1 max-h-24 w-full rounded-sm object-contain"
|
||||
data-testid="value-preview-thumbnail"
|
||||
/>
|
||||
)
|
||||
}
|
||||
|
||||
export function ValuePreview({
|
||||
value,
|
||||
name,
|
||||
dtype,
|
||||
defaultOpen = false,
|
||||
className,
|
||||
}: {
|
||||
value: unknown
|
||||
/**
|
||||
* The message this value is on. Only needed to draw a live frame: it is what
|
||||
* asks the engine to push the bytes for it.
|
||||
*/
|
||||
name?: string
|
||||
dtype?: DType
|
||||
/** Start unfolded, where there is room for it — an inspector, not a row. */
|
||||
defaultOpen?: boolean
|
||||
@@ -91,6 +99,8 @@ export function ValuePreview({
|
||||
}) {
|
||||
const [open, setOpen] = useState(defaultOpen)
|
||||
const structured = value !== null && typeof value === "object"
|
||||
const image =
|
||||
structured && isRef(value) && (value.media_type ?? "").startsWith("image/")
|
||||
|
||||
if (!structured) {
|
||||
return (
|
||||
@@ -118,6 +128,7 @@ export function ValuePreview({
|
||||
/>
|
||||
<Marquee text={summarize(value, dtype)} className="flex-1 font-mono" />
|
||||
</button>
|
||||
{image ? <Thumbnail value={value as object} name={name} /> : null}
|
||||
{open ? (
|
||||
<pre
|
||||
className={cn(
|
||||
|
||||
@@ -83,6 +83,26 @@ const listeners = new Map<string, Set<Listener>>()
|
||||
let connected = false
|
||||
const connectionListeners = new Set<Listener>()
|
||||
|
||||
/**
|
||||
* Object URLs for frames the engine pushed, keyed by digest.
|
||||
*
|
||||
* Small on purpose: a camera's frames are worth exactly as long as the next
|
||||
* one takes to arrive, and a tab that held every one of them would grow
|
||||
* without bound. The oldest is revoked when the room runs out.
|
||||
*/
|
||||
const BYTES_LIMIT = 8
|
||||
const bytes = new Map<string, string>()
|
||||
|
||||
/**
|
||||
* Which messages this page is drawing bytes for, and how many tiles each.
|
||||
*
|
||||
* Refcounted because two tiles may show one camera and the first to unmount
|
||||
* must not stop the second's frames. The socket sends the key set whenever it
|
||||
* changes; nothing is pushed for a name nobody is looking at.
|
||||
*/
|
||||
const wanted = new Map<string, number>()
|
||||
const wantedListeners = new Set<Listener>()
|
||||
|
||||
function notify(key: string) {
|
||||
for (const listener of listeners.get(key) ?? []) listener()
|
||||
}
|
||||
@@ -247,6 +267,53 @@ export const liveStore = {
|
||||
isConnected() {
|
||||
return connected
|
||||
},
|
||||
/**
|
||||
* Take a frame the socket pushed, as an object URL the tile can draw.
|
||||
*
|
||||
* Content addressing means a digest already here is the same bytes, so the
|
||||
* blob is dropped rather than replacing an identical one.
|
||||
*/
|
||||
setBytes(digest: string, blob: Blob) {
|
||||
if (bytes.has(digest)) return
|
||||
bytes.set(digest, URL.createObjectURL(blob))
|
||||
while (bytes.size > BYTES_LIMIT) {
|
||||
const oldest = bytes.keys().next().value
|
||||
if (oldest === undefined) break
|
||||
const url = bytes.get(oldest)
|
||||
bytes.delete(oldest)
|
||||
if (url) URL.revokeObjectURL(url)
|
||||
notify(`bytes:${oldest}`)
|
||||
}
|
||||
notify(`bytes:${digest}`)
|
||||
},
|
||||
getBytes(digest: string) {
|
||||
return bytes.get(digest)
|
||||
},
|
||||
/** Ask for this message's frames while the caller is drawing it. */
|
||||
wantBytes(name: string) {
|
||||
wanted.set(name, (wanted.get(name) ?? 0) + 1)
|
||||
if (wanted.get(name) === 1)
|
||||
for (const listener of wantedListeners) listener()
|
||||
return () => {
|
||||
const left = (wanted.get(name) ?? 1) - 1
|
||||
if (left > 0) {
|
||||
wanted.set(name, left)
|
||||
return
|
||||
}
|
||||
wanted.delete(name)
|
||||
for (const listener of wantedListeners) listener()
|
||||
}
|
||||
},
|
||||
wantedNames() {
|
||||
return [...wanted.keys()]
|
||||
},
|
||||
/** Told when the set changes, so the socket can say so. */
|
||||
onWanted(listener: Listener) {
|
||||
wantedListeners.add(listener)
|
||||
return () => {
|
||||
wantedListeners.delete(listener)
|
||||
}
|
||||
},
|
||||
reset() {
|
||||
for (const key of values.keys()) notify(`value:${key}`)
|
||||
values.clear()
|
||||
@@ -264,6 +331,13 @@ export const liveStore = {
|
||||
notify("logs")
|
||||
for (const flow of paused) notify(`paused:${flow}`)
|
||||
paused.clear()
|
||||
for (const [digest, url] of bytes) {
|
||||
URL.revokeObjectURL(url)
|
||||
notify(`bytes:${digest}`)
|
||||
}
|
||||
bytes.clear()
|
||||
// `wanted` is deliberately kept: the tiles asking are still mounted, and a
|
||||
// reconnecting socket has to say what they want all over again.
|
||||
},
|
||||
}
|
||||
|
||||
@@ -274,6 +348,15 @@ export function useLiveValue(name: string | undefined): LiveValue | undefined {
|
||||
)
|
||||
}
|
||||
|
||||
/** The pushed bytes for a digest, if the socket carried them. */
|
||||
export function useLiveBytes(digest: string | undefined): string | undefined {
|
||||
return useSyncExternalStore(
|
||||
(listener) =>
|
||||
digest ? subscribeKey(`bytes:${digest}`, listener) : () => {},
|
||||
() => (digest ? bytes.get(digest) : undefined),
|
||||
)
|
||||
}
|
||||
|
||||
export function useNodeStatus(nodeId: string): LiveStatus | undefined {
|
||||
return useSyncExternalStore(
|
||||
(listener) => subscribeKey(`status:${nodeId}`, listener),
|
||||
|
||||
@@ -7,6 +7,7 @@ import { dashboardKeys, panelKeys } from "@/components/Dashboard/queries"
|
||||
import { healthKeys } from "@/components/Health/queries"
|
||||
import { runKeys } from "@/components/Runs/queries"
|
||||
import { connectionStore } from "@/lib/connectionStore"
|
||||
import { parseMediaFrame } from "@/lib/media"
|
||||
import { apiToken } from "@/lib/portal"
|
||||
import { type LogLine, liveStore, type ValueSource } from "./liveStore"
|
||||
import { flowKeys } from "./queries"
|
||||
@@ -160,9 +161,25 @@ function schedule() {
|
||||
retry = Math.min(retry * 2, RECONNECT_MAX)
|
||||
}
|
||||
|
||||
/**
|
||||
* Say which messages this page wants frames for.
|
||||
*
|
||||
* Bytes are the one thing the socket does not send unasked: a camera is
|
||||
* hundreds of kilobytes a second and most tabs are drawing no media at all. So
|
||||
* a Media tile or a thumbnail registers its message, and this tells the engine
|
||||
* — again on every reconnect, since the new socket knows nothing.
|
||||
*/
|
||||
function sendWanted() {
|
||||
if (socket?.readyState !== WebSocket.OPEN) return
|
||||
socket.send(JSON.stringify({ type: "media", names: liveStore.wantedNames() }))
|
||||
}
|
||||
|
||||
liveStore.onWanted(sendWanted)
|
||||
|
||||
function connect() {
|
||||
if (watchers === 0 || socket || timer) return
|
||||
const ws = new WebSocket(socketUrl())
|
||||
ws.binaryType = "arraybuffer"
|
||||
socket = ws
|
||||
|
||||
ws.onopen = () => {
|
||||
@@ -185,6 +202,7 @@ function connect() {
|
||||
]) {
|
||||
client?.invalidateQueries({ queryKey })
|
||||
}
|
||||
sendWanted()
|
||||
}
|
||||
|
||||
const handle = (message: FlowEvent) => {
|
||||
@@ -318,6 +336,13 @@ function connect() {
|
||||
// frame rather than the connection: an exception thrown here escapes into
|
||||
// `window.onerror` and leaves whatever it had already applied behind.
|
||||
try {
|
||||
if (event.data instanceof ArrayBuffer) {
|
||||
// Media, sent in front of the value that names it — so the tile has
|
||||
// the bytes by the time it hears the message changed.
|
||||
const frame = parseMediaFrame(event.data)
|
||||
if (frame) liveStore.setBytes(frame.header.digest, frame.bytes)
|
||||
return
|
||||
}
|
||||
const payload = JSON.parse(event.data)
|
||||
if (payload?.type === "batch") {
|
||||
// A cascade publishes a dozen events at once and the engine coalesces
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
/**
|
||||
* The framing a pushed frame arrives in, and what a malformed one must do.
|
||||
*
|
||||
* Run: `bun src/lib/media.check.ts` (there is no unit runner; the suite in
|
||||
* `tests/` drives a running stack).
|
||||
*/
|
||||
import assert from "node:assert/strict"
|
||||
|
||||
import { describeArtifact, isRef, parseMediaFrame } from "./media"
|
||||
|
||||
function frame(header: object, payload: Uint8Array): ArrayBuffer {
|
||||
const encoded = new TextEncoder().encode(JSON.stringify(header))
|
||||
const out = new Uint8Array(4 + encoded.length + payload.length)
|
||||
new DataView(out.buffer).setUint32(0, encoded.length)
|
||||
out.set(encoded, 4)
|
||||
out.set(payload, 4 + encoded.length)
|
||||
return out.buffer
|
||||
}
|
||||
|
||||
const bytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47])
|
||||
const parsed = parseMediaFrame(
|
||||
frame(
|
||||
{
|
||||
type: "media",
|
||||
name: "cam.frame",
|
||||
digest: `sha256:${"a".repeat(64)}`,
|
||||
media_type: "image/png",
|
||||
ts: 1,
|
||||
},
|
||||
bytes,
|
||||
),
|
||||
)
|
||||
assert.ok(parsed)
|
||||
assert.equal(parsed.header.name, "cam.frame")
|
||||
assert.equal(parsed.header.media_type, "image/png")
|
||||
assert.equal(parsed.bytes.size, 4)
|
||||
assert.equal(parsed.bytes.type, "image/png")
|
||||
|
||||
// A frame the client cannot read is dropped, never thrown on: this runs on
|
||||
// every socket message.
|
||||
assert.equal(parseMediaFrame(new ArrayBuffer(0)), null)
|
||||
assert.equal(parseMediaFrame(new ArrayBuffer(2)), null)
|
||||
assert.equal(parseMediaFrame(frame({ name: "n" }, bytes)), null)
|
||||
// A length longer than the frame is truncation, not a payload.
|
||||
const short = new Uint8Array(8)
|
||||
new DataView(short.buffer).setUint32(0, 999)
|
||||
assert.equal(parseMediaFrame(short.buffer), null)
|
||||
|
||||
assert.ok(isRef({ digest: `sha256:${"b".repeat(64)}` }))
|
||||
assert.equal(isRef({ digest: "nope" }), false)
|
||||
assert.equal(isRef(null), false)
|
||||
|
||||
assert.equal(
|
||||
describeArtifact({ media_type: "image/png", name: "f.png", size: 48000 }),
|
||||
"image/png · f.png · 48kB",
|
||||
)
|
||||
// No media type is still worth summarising as something.
|
||||
assert.equal(
|
||||
describeArtifact({ media_type: "application/octet-stream", size: 12 }),
|
||||
"artifact · 12B",
|
||||
)
|
||||
|
||||
console.log("media.check.ts ok")
|
||||
@@ -0,0 +1,152 @@
|
||||
/**
|
||||
* Media references, and the bytes behind them.
|
||||
*
|
||||
* A frame never travels as a message — the reference does. What the bytes cost
|
||||
* to get is what limits the rate: fetched, it is a round trip per frame and a
|
||||
* camera is a glance; pushed down the socket that already carries the value,
|
||||
* it is a view. Both paths are here, and a caller does not choose between
|
||||
* them: the hook takes whichever arrived.
|
||||
*/
|
||||
import { useEffect, useState } from "react"
|
||||
|
||||
import { OpenAPI } from "@/client"
|
||||
import { liveStore, useLiveBytes } from "@/components/Flow/liveStore"
|
||||
import { apiToken } from "@/lib/portal"
|
||||
import { si } from "@/lib/utils"
|
||||
|
||||
/** An artifact reference, as a message carries one. */
|
||||
export type MediaRef = {
|
||||
digest?: string
|
||||
media_type?: string
|
||||
name?: string
|
||||
size?: number
|
||||
}
|
||||
|
||||
export const isRef = (value: unknown): value is MediaRef =>
|
||||
typeof value === "object" &&
|
||||
value !== null &&
|
||||
typeof (value as MediaRef).digest === "string" &&
|
||||
(value as MediaRef).digest!.startsWith("sha256:")
|
||||
|
||||
/**
|
||||
* What a reference *is*, in the space a value would have taken: what kind of
|
||||
* bytes, what they were called, how many of them.
|
||||
*/
|
||||
export function describeArtifact(record: Record<string, unknown>): string {
|
||||
const name = typeof record.name === "string" && record.name ? record.name : ""
|
||||
const size = typeof record.size === "number" ? `${si(record.size)}B` : ""
|
||||
// The media type when there is one: what kind of bytes these are is the
|
||||
// first thing worth knowing about a frame or a clip, and it is what the
|
||||
// port's own type was declared against.
|
||||
const media =
|
||||
typeof record.media_type === "string" &&
|
||||
record.media_type &&
|
||||
record.media_type !== "application/octet-stream"
|
||||
? record.media_type
|
||||
: "artifact"
|
||||
return [media, name, size].filter(Boolean).join(" · ")
|
||||
}
|
||||
|
||||
/** The header a pushed frame carries in front of its bytes. */
|
||||
export type MediaFrame = {
|
||||
name: string
|
||||
digest: string
|
||||
media_type: string
|
||||
ts?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Split a binary socket frame into what it says and what it carries.
|
||||
*
|
||||
* Four bytes of header length, the header as JSON, then the bytes. Returns
|
||||
* null for anything that does not parse, since a frame the client cannot read
|
||||
* is one to drop rather than one to crash on.
|
||||
*/
|
||||
export function parseMediaFrame(
|
||||
buffer: ArrayBuffer,
|
||||
): { header: MediaFrame; bytes: Blob } | null {
|
||||
if (buffer.byteLength < 4) return null
|
||||
const length = new DataView(buffer).getUint32(0)
|
||||
if (length <= 0 || 4 + length > buffer.byteLength) return null
|
||||
try {
|
||||
const header = JSON.parse(
|
||||
new TextDecoder().decode(new Uint8Array(buffer, 4, length)),
|
||||
) as MediaFrame
|
||||
if (
|
||||
typeof header?.digest !== "string" ||
|
||||
typeof header?.name !== "string"
|
||||
) {
|
||||
return null
|
||||
}
|
||||
return {
|
||||
header,
|
||||
bytes: new Blob([new Uint8Array(buffer, 4 + length)], {
|
||||
type: header.media_type || "application/octet-stream",
|
||||
}),
|
||||
}
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A local URL for an artifact's bytes, refreshed whenever the digest changes.
|
||||
*
|
||||
* Pushed bytes are used where the socket sent them, which is the fast path and
|
||||
* costs no request at all. Otherwise they are fetched: `/artifacts/{digest}`
|
||||
* takes a bearer token and no `<img>` or `<audio>` can carry a header, so the
|
||||
* bytes come through fetch and are handed to the element as an object URL.
|
||||
*
|
||||
* Passing `name` registers interest in that message, which is what tells the
|
||||
* engine to push its frames at all — nothing is sent to a screen that is not
|
||||
* drawing it.
|
||||
*/
|
||||
export function useArtifactUrl(ref: MediaRef | null, name?: string): string {
|
||||
const digest = ref?.digest ?? ""
|
||||
const mediaType = ref?.media_type ?? ""
|
||||
const pushed = useLiveBytes(digest)
|
||||
const [fetched, setFetched] = useState("")
|
||||
|
||||
useEffect(() => {
|
||||
if (!name) return
|
||||
return liveStore.wantBytes(name)
|
||||
}, [name])
|
||||
|
||||
useEffect(() => {
|
||||
// Already here: the socket carried the bytes in front of the value.
|
||||
if (!digest || pushed) {
|
||||
setFetched("")
|
||||
return
|
||||
}
|
||||
let live = true
|
||||
let made = ""
|
||||
const token = apiToken()
|
||||
const query = mediaType
|
||||
? `?media_type=${encodeURIComponent(mediaType)}`
|
||||
: ""
|
||||
const request = new AbortController()
|
||||
fetch(`${OpenAPI.BASE}/api/v1/artifacts/${digest}${query}`, {
|
||||
headers: token ? { Authorization: `Bearer ${token}` } : {},
|
||||
signal: request.signal,
|
||||
})
|
||||
.then((answer) => (answer.ok ? answer.blob() : Promise.reject(answer)))
|
||||
.then((blob) => {
|
||||
if (!live) return
|
||||
made = URL.createObjectURL(blob)
|
||||
setFetched(made)
|
||||
})
|
||||
.catch(() => {
|
||||
if (live) setFetched("")
|
||||
})
|
||||
return () => {
|
||||
live = false
|
||||
// Abandoned rather than merely ignored: a tile showing a camera starts a
|
||||
// request per frame, and the ones it no longer wants should not still be
|
||||
// arriving.
|
||||
request.abort()
|
||||
if (made) URL.revokeObjectURL(made)
|
||||
}
|
||||
}, [digest, mediaType, pushed])
|
||||
|
||||
return pushed || fetched
|
||||
}
|
||||
Reference in New Issue
Block a user