Push a frame instead of storing and fetching it

The rate the media dtypes could carry was one frame every second or two: each
was a file on the data volume, an event on the socket, and a request back for
the bytes. This closes both halves of that, and they are one feature.

`save_artifact(..., volatile=True)` writes to a `VolatileStore` — the same
content-addressed store, in `/dev/shm`, bounded by size with the oldest falling
out (`ARTIFACT_VOLATILE_BYTES`, 48 MB under the container's raised `shm_size`).
Nothing sweeps it: a frame nobody kept is not worth walking the store to find.
`ArtifactStore.path` falls through to it, which is what lets a volatile frame be
an ordinary reference everywhere else — the dtype check, a panel's digest scope,
`load_artifact` in a node, and the widget's own fetch all work on one unchanged.
`adopt` copies one into the store when a run records it, so "returned media is
kept, emitted media is not" stays true.

The bytes then go down the flows websocket as a length-prefixed binary frame,
sent just ahead of the `message_value` naming them, so a tile has the frame when
it hears the value moved. Nothing is pushed unasked: a client names the messages
it is drawing (`{"type":"media","names":[…]}`), a panel's list is intersected
with the scope it already had, and only the newest frame per name in a batch is
sent — a client that fell behind is not handed frames it would draw over. The
tunnel relays text only, so a screen reached through a portal falls back to
fetching, which is why the rate table now has two rows.

Around the edges: the remote worker's fetch cache is bounded at last
(`FLUKSIO_ARTIFACT_CACHE_BYTES`), since content addressing means nothing in it
ever expires and a media stream fills it with chunks nothing asks for twice; a
port carrying an image draws the frame in the node panel rather than only
saying `image/png · frame.png · 1.79kB`; and an edge chip says that much instead
of a line of hash. The media screenshot stops waiting for `networkidle` — a
camera is a socket that never goes quiet, which is the point of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YC4u66vjzW54fnHu5Juhh9

backend/fluksio/flow/artifacts.py

@@ -52,6 +52,10 @@ class ArtifactStore:
     def __init__(self, root: Path) -> None:
         self.root = root
         self.root.mkdir(parents=True, exist_ok=True)
+        #: A ring of frames held in memory, for media a flow only shows live.
+        #: Set by the engine at startup; every lookup below falls through to
+        #: it, which is what lets a volatile reference be an ordinary one.
+        self.volatile: VolatileStore | None = None
 
     def _path(self, digest: str) -> Path:
         body = digest[len(DIGEST_PREFIX) :]
@@ -60,14 +64,25 @@ class ArtifactStore:
         return self.root / body[:2] / body
 
     def put(
-        self, chunks: Iterable[bytes], name: str = "", media_type: str = ""
+        self,
+        chunks: Iterable[bytes],
+        name: str = "",
+        media_type: str = "",
+        volatile: bool = False,
     ) -> dict[str, Any]:
         """Store a stream and return the reference to it.
 
         Written to a temporary file first and moved into place once the digest
         is known, so a half-written artifact never has a name anyone can find.
         A file already there is left alone: identical content is identical.
+
+        ``volatile`` puts it in the ring instead, where it is held in memory
+        and falls out once newer frames need the room. The reference is the
+        same shape either way — what differs is how long the bytes last, and a
+        caller that wants one kept returns it from a run.
         """
+        if volatile and self.volatile is not None:
+            return self.volatile.put(chunks, name=name, media_type=media_type)
         digester = hashlib.sha256()
         size = 0
         handle = tempfile.NamedTemporaryFile(dir=self.root, delete=False)
@@ -98,21 +113,52 @@ class ArtifactStore:
         }
 
     def put_file(
-        self, path: Path, media_type: str = "", name: str = ""
+        self,
+        path: Path,
+        media_type: str = "",
+        name: str = "",
+        volatile: bool = False,
     ) -> dict[str, Any]:
         with path.open("rb") as handle:
             return self.put(
                 iter(lambda: handle.read(CHUNK), b""),
                 name=name or path.name,
                 media_type=media_type,
+                volatile=volatile,
             )
 
     def path(self, digest: str) -> Path | None:
-        """Where the bytes are, or None if this store does not have them."""
+        """Where the bytes are, or None if this store does not have them.
+
+        The ring is looked in second, so everything that resolves a digest —
+        serving one over HTTP, checking a run input still exists, a panel's
+        scope — reaches a live frame without knowing there are two stores.
+        """
         if not valid_digest(digest):
             return None
         target = self._path(digest)
-        return target if target.exists() else None
+        if target.exists():
+            return target
+        if self.volatile is not None:
+            return self.volatile.path(digest)
+        return None
+
+    def adopt(self, digest: str) -> bool:
+        """Copy a volatile artifact into this store, so it outlives the ring.
+
+        What makes "emitted media is not kept, returned media is" true: a run
+        recording a reference calls this, and the frame stops being one the
+        next few seconds can evict.
+        """
+        if self.volatile is None or not valid_digest(digest):
+            return False
+        if self._path(digest).exists():
+            return True
+        source = self.volatile.path(digest)
+        if source is None:
+            return False
+        self.put_file(source)
+        return True
 
     def read(self, digest: str) -> Iterator[bytes]:
         target = self.path(digest)
@@ -150,3 +196,71 @@ class ArtifactStore:
             except OSError:
                 logger.warning("Could not remove artifact %s", entry.name)
         return removed
+
+
+class VolatileStore(ArtifactStore):
+    """A bounded ring of artifacts, held wherever memory is cheaper than disk.
+
+    A camera publishing ten frames a second is ten files a second, and on the
+    wall panel this is built for that disk is an SD card. So the frames go to a
+    memory-backed directory instead and the oldest fall out once the ring is
+    full: nothing sweeps it, because a frame nobody kept is not worth a pass
+    over the store to find.
+
+    It is an ``ArtifactStore``, digest layout and all, which is what lets a
+    frame be an ordinary reference — the dtype check, the panel's scope, a
+    node opening one with ``load_artifact`` and the widget fetching one all
+    work on it unchanged.
+    """
+
+    def __init__(self, root: Path, limit_bytes: int) -> None:
+        super().__init__(root)
+        self.limit_bytes = limit_bytes
+
+    def put(
+        self,
+        chunks: Iterable[bytes],
+        name: str = "",
+        media_type: str = "",
+        volatile: bool = False,
+    ) -> dict[str, Any]:
+        reference = super().put(chunks, name=name, media_type=media_type)
+        self.trim()
+        return reference
+
+    def trim(self) -> int:
+        """Drop the oldest until the ring is inside its bound.
+
+        Also called on a timer, because a worker in this container writes here
+        itself and the engine never sees that ``put``.
+
+        # ponytail: a scandir per trim. An in-memory index if a ring of
+        # thousands of frames ever shows up in a profile.
+        """
+        if self.limit_bytes <= 0:
+            return 0
+        entries: list[tuple[float, int, Path]] = []
+        total = 0
+        for entry in self.root.glob("*/*"):
+            try:
+                stat = entry.stat()
+            except OSError:
+                continue
+            if not entry.is_file():
+                continue
+            entries.append((stat.st_mtime, stat.st_size, entry))
+            total += stat.st_size
+        if total <= self.limit_bytes:
+            return 0
+
+        removed = 0
+        for _mtime, size, entry in sorted(entries):
+            if total <= self.limit_bytes:
+                break
+            try:
+                entry.unlink()
+            except OSError:
+                continue
+            total -= size
+            removed += 1
+        return removed

backend/fluksio/flow/connector.py

@@ -141,6 +141,7 @@ class ConnectorNode(Node):
         data: bytes,
         name: str = "",
         media_type: str = "application/octet-stream",
+        volatile: bool = False,
     ) -> dict[str, Any]:
         """Store bytes and return the reference to publish on a media port.
 
@@ -149,6 +150,11 @@ class ConnectorNode(Node):
         store and the reference names them, which is what an ``image``,
         ``audio`` or ``video`` port carries.
 
+        ``volatile`` is what a camera publishes with: the bytes go to a ring
+        held in memory rather than to the data volume, are pushed to whatever
+        screen is watching, and last seconds. Use it for a frame; leave it off
+        for a recording somebody asked to keep.
+
         Only available once the node has started — the store belongs to the
         engine, and is handed over then.
         """
@@ -156,7 +162,9 @@ class ConnectorNode(Node):
             raise RuntimeError(
                 "no artifact store: a connector can only save bytes once it has started"
             )
-        return self._artifacts.put([data], name=name, media_type=media_type)
+        return self._artifacts.put(
+            [data], name=name, media_type=media_type, volatile=volatile
+        )
 
     async def start(self, app: FastAPI | None = None) -> None:
         self._artifacts = getattr(app.state, "artifact_store", None) if app else None

backend/fluksio/flow/runs.py

@@ -1313,6 +1313,11 @@ class RunService:
             with Session(db_engine) as session:
                 session.merge(row)
                 for message, ref in outcome.artifacts.items():
+                    # A run recording a reference is what "returned media is
+                    # kept" means: copied out of the volatile ring, or the row
+                    # would outlive the bytes it names by a few seconds.
+                    if self._artifacts is not None:
+                        self._artifacts.adopt(str(ref.get("digest") or ""))
                     session.merge(
                         RunArtifact(
                             run_id=run_id,