Files
app/docs/reference/payload-types.md
T
stroblmeandClaude Opus 5 0ffcabfdb9
Docs / docs (push) Successful in 30s
Playwright Tests / test-playwright (1, 2) (push) Successful in 3m7s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m54s
pre-commit / pre-commit (push) Failing after 4m24s
Test Backend / test-backend (push) Successful in 3m8s
Compose Smoke Test / test-compose (push) Successful in 40s
Playwright Tests / merge-reports (push) Successful in 1m33s
Media dtypes: image, audio and video as narrowed artifact references
A port may now declare `image`, `audio` or `video`. Each is the artifact
reference the engine already had, narrowed by the `media_type` on it, so a
speech recogniser declares what it eats rather than taking any bytes at all and
finding out. Bytes still never travel as a message and nothing on the wire
stops being JSON: a camera publishes one reference per frame, a microphone one
per chunk, and a reference may carry a `meta` dict nothing here interprets.

Streaming media is therefore an ordinary streaming port — with one change to
what that means. An emission used to journal an item with no payload, so
downstream read whatever was current when the item was claimed; a consumer
slower than its producer saw only the newest chunk and the ones between were
lost. That is right for a training curve and wrong for a second of speech, so
an emission now journals a `kind="emission"` item carrying its values, and the
executor hands them to the nodes reading that message instead of writing them
to state again. The value in state stays the latest, which is what everything
else reads, and the wave is filtered by what actually changed rather than
walking everything reachable. No queue serialization change — the existing
`outputs` field carries it.

Continuous media makes the store's missing GC a real problem, so this closes
it: `sweep_artifacts` runs hourly, keeps every digest a `run_artifact` row
records or a live message holds, spares anything written in the last hour, and
stands aside entirely while a run is in flight, since a node may store a
checkpoint long before it returns the reference to it. That also collects the
orphans a deleted flow has always left behind. `ARTIFACT_GC_INTERVAL_S=0` turns
it off.

Around the edges: `GET /artifacts/{digest}` serves the media type the caller
passes and answers ranged requests, so a browser plays a clip rather than
downloading it; `PUT` spools to disk instead of holding the whole body in
memory, as does `save_artifact` given a path; a Media widget draws whatever its
message points at, and a wall panel may fetch the bytes its own tiles are
showing and nothing else; and a connector gets `save_artifact`, for a device
whose readings are bytes.

What this cannot do is live video: a frame every second or two is a glance, and
the honest answer above that is the camera's own stream, which the widget takes
as a URL and the browser plays from source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 23:44:55 +02:00

6.4 KiB
Raw Blame History

Payload types

Every port declares a dtype, and every value that passes through it is checked against that declaration.

This is not decoration. It is what lets the dashboard editor offer you only the messages a gauge can actually draw, what lets the canvas refuse a binding before anything runs, and what lets a downstream node know the shape of what it is getting before the flow starts.

Everything on the wire is JSON. That is what lets the same value pass through the state backend, the work queue and the worker protocol unchanged.

The scalars

dtype Accepts
float any number — int or float, but not bool
int a whole number, not bool
str a string
bool exactly true or false

bool is an int subclass in Python and deliberately not a number here: a flag is not a measurement, and a switch bound to a temperature is a mistake worth catching.

NaN and infinity are refused, wherever they sit — including inside a json, record, series or list. JSON cannot spell either, so one that travelled would come back as a response nobody can parse and a row the database rejects, a long way from the node that made it. An empty subset or a division with no denominator is what usually produces one; publish None instead.

The structured ones

These are declared shapes rather than "some JSON", which is what makes a widget binding checkable.

record

Flat named scalars.

{"title": "Boiler", "body": "Pressure low", "severity": "warning"}

Nesting is deliberately out: a record that can contain a record is a schema language, and the shape stops being readable from the declaration alone.

Read by the Notification widget. It is also what an alert channel of kind dashboard writes.

list

Ordered items of one declared shape. The port also declares item:

item Meaning
unset record — what the agenda and forecast widgets read
float, int, str, bool a list of scalars
json anything

A list of lists, or a list of series, is refused. One declared level is the point.

series

Labelled lines of (timestamp, value) pairs — what a chart draws.

{
  "lines": [
    {"label": "living", "points": [[1717000000, 21.4], [1717000060, 21.5]]}
  ],
  "range": "-24h"
}

Keys beside lines are carried through untouched, which is how a querying chart puts the window and resolution it asked for on the request and reads them back off the answer. That is what stops an answer to a different question from overwriting the picture.

GET /runs/series/compare answers in this shape, which is why comparing three training curves is a widget binding rather than a screen of its own.

artifact

A reference to stored bytes.

{"digest": "sha256:…", "size": 4194304, "media_type": "application/octet-stream", "name": "weights.pt"}

Binary payloads — tensors, checkpoints, images — never travel as a message. The bytes go to a content-addressed store and the message carries this. A thirty-megabyte checkpoint never sits in the state backend, and the reference stays valid wherever the store is reachable from, including on another machine.

Node code produces one with fluksio.save_artifact and opens one with fluksio.load_artifact. See Writing node code.

image, audio, video

The same reference, narrowed to a kind of media by its media_type.

{"digest": "sha256:…", "size": 61344, "media_type": "image/jpeg", "name": "frame.jpg",
 "meta": {"width": 1280, "height": 720, "seq": 41}}

An audio port takes audio/* and refuses anything else, so a speech recogniser declares what it eats rather than taking any bytes at all and finding out. An artifact port still accepts all three — media narrows artifact, not the other way round.

meta is optional and nothing here reads it: sample rates, dimensions and sequence numbers are for whoever consumes the media.

Bytes still never travel as a message. A camera publishes one reference per frame and a microphone one per chunk — which makes a media stream an ordinary streaming port, and each frame an artifact. What that costs is worth knowing before pointing a camera at it:

Rate Where it works
A clip a second (speech) anywhere, including through the portal
A frame every second or two (a glance at a door) locally; through the portal, every few seconds
Live video, 1530 fps not here — see below

Real-time video is not a message-plane problem: every frame would be an artifact, an event and a fetch. Point a media widget's stream URL at whatever the camera already serves and the browser plays it from source; the messages then carry the occasional still, and the flow reacts to those.

json

Anything JSON-serializable. The escape hatch, and the right answer when a payload genuinely has no fixed shape.

Reach for it last. A json port tells the canvas, the widget picker and the next author nothing.

Naming a run's output

Any run parameter is also accepted as text, since nobody wants to paste an object into a shell. @run:<id>.<output> names what an earlier run produced — whatever its type, an artifact reference or a json config alike — and a bare sha256:… digest names content in the artifact store. Both resolve before the run starts, so the CLI, the run dialog and a python caller all mean the same thing by the same string.

Both spellings are reserved on every input, str included: an input that has to carry one of them literally is asking for a value this engine reads as a name.

What a widget will bind to

Widget Accepts
Gauge, Chart, Slider, Bar float, int
Switch bool
Agenda, Forecast list
Notification record
Value anything
Icon weather strings, booleans and numbers alike
Media image, audio, video, artifact
Clock, Text nothing — they bind to no message

Enforced on the server as well as in the editor.

Type failures

A value that does not match its port's declaration raises on the node that published it, naming the port and what arrived. It does not get published, and it does not reach anything downstream — a wrong value stopping at its source is much easier to diagnose than one propagating.

See also