Push a frame instead of storing and fetching it
The rate the media dtypes could carry was one frame every second or two: each
was a file on the data volume, an event on the socket, and a request back for
the bytes. This closes both halves of that, and they are one feature.
`save_artifact(..., volatile=True)` writes to a `VolatileStore` — the same
content-addressed store, in `/dev/shm`, bounded by size with the oldest falling
out (`ARTIFACT_VOLATILE_BYTES`, 48 MB under the container's raised `shm_size`).
Nothing sweeps it: a frame nobody kept is not worth walking the store to find.
`ArtifactStore.path` falls through to it, which is what lets a volatile frame be
an ordinary reference everywhere else — the dtype check, a panel's digest scope,
`load_artifact` in a node, and the widget's own fetch all work on one unchanged.
`adopt` copies one into the store when a run records it, so "returned media is
kept, emitted media is not" stays true.
The bytes then go down the flows websocket as a length-prefixed binary frame,
sent just ahead of the `message_value` naming them, so a tile has the frame when
it hears the value moved. Nothing is pushed unasked: a client names the messages
it is drawing (`{"type":"media","names":[…]}`), a panel's list is intersected
with the scope it already had, and only the newest frame per name in a batch is
sent — a client that fell behind is not handed frames it would draw over. The
tunnel relays text only, so a screen reached through a portal falls back to
fetching, which is why the rate table now has two rows.
Around the edges: the remote worker's fetch cache is bounded at last
(`FLUKSIO_ARTIFACT_CACHE_BYTES`), since content addressing means nothing in it
ever expires and a media stream fills it with chunks nothing asks for twice; a
port carrying an image draws the frame in the node panel rather than only
saying `image/png · frame.png · 1.79kB`; and an edge chip says that much instead
of a line of hash. The media screenshot stops waiting for `networkidle` — a
camera is a socket that never goes quiet, which is the point of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YC4u66vjzW54fnHu5Juhh9
This commit is contained in:
@@ -730,3 +730,84 @@ def test_cancelling_reaches_a_node_running_in_a_child(pool):
|
||||
assert pool.cancel("demo.slow"), "the parent must find a child's node"
|
||||
thread.join(timeout=10)
|
||||
assert failed
|
||||
|
||||
|
||||
def test_a_node_saves_a_frame_to_the_ring(tmp_path):
|
||||
"""A camera writes to memory, and the next node reads it from there.
|
||||
|
||||
The point of the ring being an ordinary store: node code says one word
|
||||
more, and everything downstream — the dtype, ``load_artifact``, the widget
|
||||
— carries on not knowing there are two of them.
|
||||
"""
|
||||
from fluksio_worker.worker_main import ARTIFACT_VOLATILE_DIR_ENV
|
||||
|
||||
from fluksio.flow.artifacts import VolatileStore
|
||||
|
||||
store = ArtifactStore(tmp_path / "artifacts")
|
||||
store.volatile = VolatileStore(tmp_path / "ring", limit_bytes=1_000_000)
|
||||
pool = PythonWorkerPool(
|
||||
python=sys.executable,
|
||||
size=1,
|
||||
env={
|
||||
ARTIFACT_DIR_ENV: str(store.root),
|
||||
ARTIFACT_VOLATILE_DIR_ENV: str(store.volatile.root),
|
||||
},
|
||||
)
|
||||
pool.start()
|
||||
try:
|
||||
ref = pool.run(
|
||||
"demo",
|
||||
"frame",
|
||||
"import fluksio\n"
|
||||
"def process():\n"
|
||||
" return {'frame': fluksio.save_artifact(\n"
|
||||
" b'\\x89PNG' + b'p' * 64, 'f.png',\n"
|
||||
" media_type='image/png', volatile=True)}\n",
|
||||
{},
|
||||
"demo.frame",
|
||||
timeout=10,
|
||||
)["frame"]
|
||||
|
||||
assert MessageSpec(name="frame", dtype=DType.IMAGE).check(ref) is None
|
||||
# In the ring, and nowhere near the data volume.
|
||||
assert store.volatile.path(ref["digest"]) is not None
|
||||
assert not (store.root / ref["digest"][7:9]).exists()
|
||||
|
||||
read = pool.run(
|
||||
"demo",
|
||||
"read",
|
||||
"import fluksio\n"
|
||||
"def process(frame):\n"
|
||||
" with open(fluksio.load_artifact(frame), 'rb') as f:\n"
|
||||
" return {'size': len(f.read())}\n",
|
||||
{"frame": ref},
|
||||
"demo.read",
|
||||
timeout=10,
|
||||
)
|
||||
assert read == {"size": 68}
|
||||
finally:
|
||||
pool.stop()
|
||||
|
||||
|
||||
def test_the_fetch_cache_is_bounded(tmp_path):
|
||||
"""A worker downloading a media stream fills its cache with chunks nothing
|
||||
will ask for twice, and content addressing means nothing ever expires."""
|
||||
import os
|
||||
|
||||
from fluksio_worker.worker_main import ARTIFACT_CACHE_BYTES_ENV, _trim_cache
|
||||
|
||||
cache = tmp_path / "cache"
|
||||
cache.mkdir()
|
||||
for index in range(5):
|
||||
entry = cache / f"chunk{index}"
|
||||
entry.write_bytes(b"c" * 1000)
|
||||
os.utime(entry, (index, index))
|
||||
|
||||
os.environ[ARTIFACT_CACHE_BYTES_ENV] = "3000"
|
||||
try:
|
||||
_trim_cache(str(cache))
|
||||
finally:
|
||||
del os.environ[ARTIFACT_CACHE_BYTES_ENV]
|
||||
|
||||
left = sorted(entry.name for entry in cache.iterdir())
|
||||
assert left == ["chunk2", "chunk3", "chunk4"]
|
||||
|
||||
Reference in New Issue
Block a user