The rate the media dtypes could carry was one frame every second or two: each
was a file on the data volume, an event on the socket, and a request back for
the bytes. This closes both halves of that, and they are one feature.
`save_artifact(..., volatile=True)` writes to a `VolatileStore` — the same
content-addressed store, in `/dev/shm`, bounded by size with the oldest falling
out (`ARTIFACT_VOLATILE_BYTES`, 48 MB under the container's raised `shm_size`).
Nothing sweeps it: a frame nobody kept is not worth walking the store to find.
`ArtifactStore.path` falls through to it, which is what lets a volatile frame be
an ordinary reference everywhere else — the dtype check, a panel's digest scope,
`load_artifact` in a node, and the widget's own fetch all work on one unchanged.
`adopt` copies one into the store when a run records it, so "returned media is
kept, emitted media is not" stays true.
The bytes then go down the flows websocket as a length-prefixed binary frame,
sent just ahead of the `message_value` naming them, so a tile has the frame when
it hears the value moved. Nothing is pushed unasked: a client names the messages
it is drawing (`{"type":"media","names":[…]}`), a panel's list is intersected
with the scope it already had, and only the newest frame per name in a batch is
sent — a client that fell behind is not handed frames it would draw over. The
tunnel relays text only, so a screen reached through a portal falls back to
fetching, which is why the rate table now has two rows.
Around the edges: the remote worker's fetch cache is bounded at last
(`FLUKSIO_ARTIFACT_CACHE_BYTES`), since content addressing means nothing in it
ever expires and a media stream fills it with chunks nothing asks for twice; a
port carrying an image draws the frame in the node panel rather than only
saying `image/png · frame.png · 1.79kB`; and an edge chip says that much instead
of a line of hash. The media screenshot stops waiting for `networkidle` — a
camera is a socket that never goes quiet, which is the point of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YC4u66vjzW54fnHu5Juhh9
A data-science environment installing fluksio waited for lxml, aiohttp and
the rest of a connector stack it has nothing to talk to. Outbound mail,
error reporting and the MQTT and InfluxDB clients moved to
`fluksio[server]`, which the image installs; each import is guarded and
names the extra. `tenacity` had no import site at all and is gone.
23 fewer packages and the compiled ones among them — a bare `pip install
fluksio` still serves, runs every python node, and registers the mqtt and
influxdb node types, which only need the library when one is actually
built. sentry-sdk arrives anyway underneath `fastapi[standard]`; what
changed there is that nothing of ours requires it.
The dev environment keeps every extra: the suite exercises the connectors
and strict mypy checks their call sites.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc
Inside a worker `import fluksio` is the reporter, which had emit and the
artifact calls but no logger — so `fluksio.logger.info(...)`, written
against the SDK's top-level export, died with AttributeError after the
training it was reporting on had already succeeded. Its records go to the
same capture a print does; the handler resolves sys.stderr per record
because a call runs under redirect_stderr.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc
A port may now declare `image`, `audio` or `video`. Each is the artifact
reference the engine already had, narrowed by the `media_type` on it, so a
speech recogniser declares what it eats rather than taking any bytes at all and
finding out. Bytes still never travel as a message and nothing on the wire
stops being JSON: a camera publishes one reference per frame, a microphone one
per chunk, and a reference may carry a `meta` dict nothing here interprets.
Streaming media is therefore an ordinary streaming port — with one change to
what that means. An emission used to journal an item with no payload, so
downstream read whatever was current when the item was claimed; a consumer
slower than its producer saw only the newest chunk and the ones between were
lost. That is right for a training curve and wrong for a second of speech, so
an emission now journals a `kind="emission"` item carrying its values, and the
executor hands them to the nodes reading that message instead of writing them
to state again. The value in state stays the latest, which is what everything
else reads, and the wave is filtered by what actually changed rather than
walking everything reachable. No queue serialization change — the existing
`outputs` field carries it.
Continuous media makes the store's missing GC a real problem, so this closes
it: `sweep_artifacts` runs hourly, keeps every digest a `run_artifact` row
records or a live message holds, spares anything written in the last hour, and
stands aside entirely while a run is in flight, since a node may store a
checkpoint long before it returns the reference to it. That also collects the
orphans a deleted flow has always left behind. `ARTIFACT_GC_INTERVAL_S=0` turns
it off.
Around the edges: `GET /artifacts/{digest}` serves the media type the caller
passes and answers ranged requests, so a browser plays a clip rather than
downloading it; `PUT` spools to disk instead of holding the whole body in
memory, as does `save_artifact` given a path; a Media widget draws whatever its
message points at, and a wall panel may fetch the bytes its own tiles are
showing and nothing else; and a connector gets `save_artifact`, for a device
whose readings are bytes.
What this cannot do is live video: a frame every second or two is a glance, and
the honest answer above that is the camera's own stream, which the widget takes
as a URL and the browser plays from source.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The worker held each yield one behind, because the last one is the node's
result when the generator returns nothing of its own. Only the engine knows
what ports a node declared, so the check happened when the *next* yield
arrived — a pass late, which for a training loop is however long one epoch
takes.
The worker now sends every yield as it happens and returns whatever its
generator returned; EmitSink holds the last one back and decides at the end
of the call what it was. Old "emit" frames are still handled, so a remote
agent that has not been restarted keeps working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four things the python SDK turned up, each fixed where every client sees it.
A key no port declares is now an error rather than a silent drop, on the
return, the yield and the emit alike — the contract the docs already stated.
The SDK reads literal yields at sync time, so a typo fails before anything
runs, and an emission of one fails the call rather than being logged where
nobody looks.
NaN and infinity are refused at the port. JSON cannot spell either, so one
that travelled came back as a 500, a socket frame that stopped the canvas, or
a metric batch the database dropped whole.
An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the
engine — so the CLI, the run dialog and a python caller mean the same thing,
and a sweep can pass one at all.
Node timeouts are off by default. The clock measured silence, which a training
node is full of, and remote workers had already stopped enforcing it — their
heartbeat reset it. Now a heartbeat proves the agent rather than the node,
ninety seconds of nothing fails the call either way, and the engine touches
work it is still running so a long node is not redelivered at sixty seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
A data scientist keeps their code where it is and decorates it: `@node`
declares a function's ports beside the function, `Flow(name, nodes=[...])`
says which of them make a flow, and `use(fn, wire=..., **settings)` rebinds
one for a single flow. `fluksio sync` uploads the document plus a generated
import shim per node, so the store still holds a complete, runnable,
git-versioned definition while the code it imports stays theirs.
`fluksio login|run|runs` and `flow.submit().wait()` are the client half, over
the run endpoints that already existed. Runs record the user repository's
commit beside the store's, so "what code produced this number" is answerable
on the side that now holds the code.
- `fluksio/sdk/`: ports, decorators, the flow builder and its checks, the shim
generator, an HTTP client and sync. Standard library only at import, so
`from fluksio import node` in a training script pulls in no engine.
- `FlowDef.origin` marks a flow code-defined; `Run.origin_commit` carries the
repository's commit; `POST /modules/refresh` retires the workers without an
install, which every sync calls — a worker holds the imported package in
memory, so an edit to it is invisible until the process goes.
- The canvas shows a generated body read-only and names the repository to edit
instead; a body edited there stops the next sync rather than being discarded.
- The worker's reporter carries inert `Port`, `node`, `use` and `Flow`, since
the shim imports a module whose first line declares them.
- `examples/myresearch` is the worked example, `make sync-example` uploads it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
A node's timeout now covers its body only: the pool loads the source into the
worker it picked, off the node's budget, so imports that outlast the timeout no
longer make a node impossible to run. Draft checks compile without caching, so
saving does not evict what a busy node is serving calls from. Requests carry an
id the worker echoes and the pool checks, a reply is encoded once, and the
remote-exception cache is bounded.
DELETE /flows/{name} answers 409 while the flow has a running or queued run,
which is what was letting run_node rows outlive their run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StpRc2C6au1WJ1EUU7fsfu
A cluster or GPU host installs `pip install fluksio-worker` and gets the
agent and the runner, not psycopg, numpy and the MCP SDK. The engine
depends on it as a workspace member, so the file it launches node code
with is the same file a remote worker runs — which is what keeps a node
unable to tell the difference. Copying the two files by hand still works.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
stroblmeandClaude Opus 5