- Volatile frames are held in memory with a byte-bounded queue instead of
the /dev/shm ring scanned on every put; workers keep writing to the
directory, which is trimmed on a timer. A volatile upload no longer spools
to disk. ARTIFACT_VOLATILE_BYTES=0 turns the ring off.
- A device sends camera frames as binary messages on its socket, held and
published in one step and credited to its device node; refused frames are
logged and dropped. Its snapshot carries demand and the uplink (binary, or
http through the portal).
- One demand registry for every socket's wanted media, the tunnel's
included, plus what running flows read, mapped onto on-demand ports and
published as device_demand; at most 8 media names per socket.
- Per-message deflate off in serve and dev; a device may fetch the last 8
digests of a message; published dashboards cached per store revision; a
revoked device socket is closed with 1008.
- Media tiles want bytes only while on screen; the Devices tab previews a
wired camera; the device node warns when a flow reads an on-demand port.
- POST /devices/pair answers a 2-character `check`, listed by /devices/pending
and shown beside each waiting entry; five codes a minute per address, and at
50 waiting the oldest gives way instead of a 429
- portal-minted device tokens are asked for with the device's nonce (`pnc`) and
held against it, so unpair and delete revoke them; one without `pnc` stands
only while the nonce is 0
- a device credential acts as the device (`device_principal`), not as the
approving superuser, so deleting that account unpairs nothing
- the report is validated port by port: unknown enum values read as `unknown`,
unreadable ports and sections are left out and logged, `config` may name an
entry of `config_schemas`; 422 only for no schema 1 or no object
- devices.json is parsed once per change (cached by stat), delete drops the
report and keeps the nonce, a busy recheck runs in the background
- scope and io only from enabled flows (start/stop tells the devices), a port
bound twice gives one io row
- the Devices tab's saves cancel and hold back device refetches until settled
A device reports its board, screen, camera and ports from its own
credential (16 KB, schema 1, replacing the last report), and the flows
are validated against it again. The node panel turns a reported port's
config schema into a form, a reported screen is a canvas preset and a
touch screen's first report turns Touch friendly on. An offline device
node is degraded: amber on the canvas, a warning on Home, never a fault.
- NodeDef `device`/`device_policy` become `worker`/`worker_policy`, the
issue `ignored_device` becomes `ignored_worker`, and the SDK's `@node`
takes `worker=` and `worker_policy=`
- FlowStore._migrate renames the keys in published and draft documents
once, bumping their version; a queued run's `needs` reads the old key
- placement passes the label as `label`, client regenerated
- docs: worker placement wording, MCP issuer without DOMAIN, device rows in
the CLI data directory and the API table, device links in compose comments
- flow/nodes/device.py: `device` node with params `device` and `config`;
provides the device's `in` ports, requires its `out` ports without
waking anything, never run (Node.passive), down "Offline" from presence
- DeviceDef.io and DevicePort go; GET /devices/{id} computes `io` and
`bindings` from the device nodes of published flows, in a fixed order,
cached by FlowStore.revision; the credential's scope comes from them
- issues `unknown_device` and `device_port` (wired twice; against a report
the wrong direction or dtype, or unlisted as advisory), rechecked when a
device is created or deleted; ValidationIssue.advisory is a field
- a device publishing on a bound port is credited to its node by the
server; the panel endpoint kind is gone from `_endpoints` and the canvas
- a rebuild that moves a device's bindings sends `dashboard_changed` with
an empty name, which the firmware refetches on
- startup moves each device's `io` rows into device nodes in the flows
declaring their messages, draft and published, one commit; rows no
flow declares are logged and left out
- frontend: Device node settings (device, Publishes, Follows, a port
picker from a report), palette shortcuts per reported port, DeviceDetail
lists bindings with "Use in flow" in place of the I/O editor
- docs: devices, flow editor, node types, issues, API
- flow/devices.py: DeviceDef in FLOWS_DIR/_devices/<id>.json, committed under
the flow store's write lock, 409 on a stale version; nonces in DEVICES_FILE
outside git; in-memory presence announced as device_status
- api/routes/devices.py: CRUD, pairing with an optional {name, board,
firmware}, a pending list adopted without a code from the LAN only, unpair;
/panels/pair, /panels/pair/{code} and GET /panels/{id} kept as aliases,
PUT /panels/ gone
- _device_may replaces _panel_may, JWT claims unchanged
- panels.json moves into devices once at startup, then .migrated
- MCP list_devices, get_device; search category device
- Devices tab (list, detail with presence, pairing, ordered dashboards,
touch, I/O), PanelsDialog removed, serialised optimistic saves
CACHE_SHAPE_SINCE marks the last change that moved every cache key. The
run service's periodic sweep clears cache_key on rows of runs created
before it, which nothing can match any more. Their outputs stay.
Its entries are the cards present, and a local child's allocation is
named through them, so under =2,3 card 0 is device 2. Remote workers
keep mapping against their own mask.
mqtt and http take websocket's vocabulary: their Mode is LISTENER/SENDER,
default names read mqtt_listener_… and http_listener_…, and mqtt's
publish_queue_size is send_queue_size. No alias stays behind, so the flow
store rewrites the old setting in every published and draft mqtt node when it
opens, in the same migration as trigger → debounce. Every mqtt and http
setting carries a one-line description for the editor, and the node-types
reference uses one wording for direction and says that an http sender
publishes a JSON object response on its outputs.
The store rewrites `"type": "trigger"` to `debounce` in every published and
draft flow when it opens, bumps the version and commits once. No alias stays
behind, so `trigger` is now an unknown type.
ws:// and wss:// URLs dial out and hold a supervised connection; a path is
served under /hooks/<flow>/ with an optional secret. Outputs only listen,
inputs send. Frames are JSON keyed by port, with json_key and on_connect.
Writing node source, installing modules and any flow save that adds,
retypes or repoints a code node (Function, Command), edits a Command
node's settings or changes a node's resources.env now answer 403 for
everyone but a superuser. MCP tools and the assistant reach the same
routes, so they inherit the gate. Node types report runs_code.
The editor shows code read-only with a note, disables a Command node's
settings, leaves code nodes and reusable nodes out of the palette and
paste, and the Modules page offers no install.
A connector that misses one poll turned the health badge amber and the node
on the canvas red until it recovered. `LoadedNode.health_since` marks when the
status last actually changed — a rotating error detail under an unchanged
status does not move it, or a connector retrying with a different errno each
poll would never debounce — and `unhealthy_nodes` applies a 60 s floor to it.
The floor is on the surfacing, not on the transition: the stored health is the
truth the moment a node reports it, so a flow reading node health is never
told a stale story. Only going down is held back; recovery clears at once.
A python node's ports and settings arrive as keyword arguments, so a declared
name its `process` does not take was a TypeError on every call — and a node
that loads fine and fails every time it runs is the quiet kind of broken: the
hosted demo did it 720 times an hour for two days and the health badge read
ok throughout. `_build_node` now reads a written body with `ast` and refuses
the mismatch at load, so the node is an error on the canvas and an issue on
publish. Skipped for `**kwargs`, a decorated or absent `process`, and the
template a new node opens with. The SDK's generated shim always takes
`**settings`, so synced flows are untouched.
`/observability/summary` names a node that has failed in the last fifteen
minutes and reads degraded while it does, which is what would have made the
badge amber. `nodes.failing` carries the count.
Follows the portal: the noun is "instance" everywhere the app says it —
UI strings, CLI output, error details, docs and comments. The wire keys
(`instance_id`, `instance_token`) and the hub route this calls move with it.
An existing cloud.json is adopted rather than refused: without the key
alias the dataclass fails to parse, which the caller swallows and reads as
"never enrolled" instead of "reconnect".
`instance_key` on a node type becomes `target_key`. It means the outside
thing a node points at, which is a different sense of the word, and keeping
both would put two meanings of "instance" in one codebase.
`brain_graph` read `self.issues` alone, which is what a build found — so a
node that loaded and then lost its device was a well neuron on Home, and
that is what someone comes to this view to find.
Measured with `make bench-engine` against a real Redis: 103.6 -> 164.4
messages a second on a five-node chain (p50 latency 2125 -> 1171 ms) and
34.8 -> 63.2 on a fan-out of twenty. Against the memory backend, which is
what a pip install runs on, 262 -> 626.
The two that bought most of it:
- `StateBackend.record` puts a published value, its timestamp, its series
and its version counter in one round trip. They were four calls building
four pipelines, and a value crossing an edge pays them twice. A released
rate-limit hold rides along instead of a DEL per port.
- the readiness check reads a node's inputs and hands them to the node,
rather than reading the triggering ones to count them and having the node
read the same keys again a moment later.
`apply_outputs` was a second copy of `_record_outputs` and is now the same
code plus the event that distinguishes it.
The rest, each small:
- `_derive` builds a node-by-id map and a `consumes` index, so dispatching
an item and publishing a value stop scanning every node in the
installation.
- `read_all` is memoised against the store revision — it sits on the
publish path, so a dashboard slider was reading and validating every
flow file per value. Same mechanism `_wiring` already uses.
- the `message_value` source block is built once per node instead of per
emission.
- both timer threads ask the queue to promote only when something is
actually due, which takes an idle engine from ~4 Redis round trips a
second to one.
- the shared httpx client is bounded (32 connections, one retry); its
default pool is 100 with no per-host cap, so one slow endpoint could
take it and every other sender node with it.
- the MQTT and delay nodes no longer log a line per message at INFO.
Robustness, in the same pass:
- `MemoryWorkQueue._done` was a set nothing ever removed from — one entry
per non-idempotent node per item, for the life of the process, in the
default configuration. Capped, the way the Redis side expires its
markers.
- a saturated engine can claim from the due lane past the cascade limit.
The capacity gate sits in front of the claim, so the due lane's priority
— decided inside it — did not apply while every slot was held: a motor's
stop was not behind the long nodes, it was unread. Only after a slot has
genuinely failed to free for half a second, and briefly, so the backlog
is not starved in turn.
- `reclaim_stale` dispatches through that same gate. It could return sixty
entries and push in-flight far past the limit the gate exists to hold.
- a flow's nodes are stopped together rather than one after another. Each
gets `NODE_STOP_TIMEOUT`, so a flow whose broker was unreachable took
five seconds per node — long enough to outlast `REBUILD_WAIT` and 503
the deploy.
- the worker pool and the HTTP client are closed on a thread, not on the
event loop, and a run closes the state backend it built (on Redis, a
client and a connection pool per run).
- the five background tasks say something when they die. Each catches
exceptions inside its loop, so one raised anywhere else left the engine
serving with no metrics, no alerts or no artifact sweep, silently.
`tests/flow/test_round_trips.py` counts the state operations one message
costs — four, where it was about eleven — because none of the above would
fail a behavioural test if it were undone.
Both calls sit in a closure where a narrowing of `Placer | None` and
`PythonWorkerPool | None` will not carry across the function boundary. The
`remote.run_on` line below them already carried the same ignore; these two
close out `make lint-backend`.
The poll loop remembered what it read rather than what it published, so a
value the node could not publish counted as said: the next poll skipped it,
succeeded, and health went back to ok with the port still dark. Remember it
only after inject returns, and report ok last.
A node reporting itself down is now derived into its flow's issues on read
and counted on the health summary, so the canvas marks it and Home says so.
Being down does not stop the flow, and the issue clears by itself when the
node reports well again. The repeating poll warning is logged once per
outage rather than once per tick.
`sync` follows each node function's imports through the project's own modules
— stopping at the standard library, at anything installed, and at Fluksio
itself, whose checkout would otherwise be most of every digest — and records
the file list with what it hashed to. The engine hashes those files again when
the run is claimed, so the fingerprint is live rather than a snapshot, and
falls back to what sync recorded when it cannot see them: a remote worker's
runs used to share one empty digest, and therefore one key.
Three things follow. Editing a helper a node calls into re-runs that node, as
before. Editing something the node never reaches no longer re-runs anything —
a notebook two directories away was invalidating every arm. And
`Run.code_digest` is now the hash of its nodes' digests, so it is neither
looser nor tighter than "the code behind these numbers", which is what makes
it worth joining an exported table on.
`sync` says so too: it compares the per-node digest against the stored one, so
a helper edit prints `train: updated (flow, fit)` instead of `unchanged`. The
digest is read when the document is built rather than when the flow is
declared, so a second `sync()` in one process sees an edit between them.
Also: `fluksio runs` shows only the inputs that differ from what the flow
declares, fitted to the terminal, so a flow taking a few kB of json no longer
wraps every line.
Every existing cache entry misses once — the fingerprint changed shape.
Concurrent runs sat at 4 whatever FLOW_MAX_CASCADES said: that setting bounds
cascades, and the run drivers read a hardcoded MAX_PARALLEL nobody could reach.
FLOW_MAX_RUNS is the knob they read now, --max-runs/--max-cascades/--max-workers
are the same three as flags on serve, and the engine says which numbers it
started with — which is the only way to tell that a settings file was read.
Events keep the run they happened in. The payload always carried it and the
persist path dropped it, so reading one run's failures meant filtering the
engine-wide list; a batch run's id reaches those events now too, since a run
has no journaled item to name itself by.
Also: a provisioner's 0 means "no deadline" rather than "cancel on the next
reconcile", and a command that reaches no engine says how to start one.
Raw cpus and gpus are a property of the machines an installation has, so a node
written against a cluster quietly stops meaning anything when the cluster is
replaced. A node says "gpu-small" instead, and what that is stored here —
editable, and read again every time the node is built, so changing the flavor
changes what the next run gets.
Memory joins the schema properly (`ram`, in MB, accepting "2G"), along with
`duration_s` for how long a node is expected to take. That one is recorded and
shown and nothing else yet: a statement for whoever is planning around the node,
not a limit — the limit is still `timeout`.
A flavor and a number for the same thing is refused, compared by value so an
editor writing the whole object back with its defaults still round-trips. A name
nothing stores is refused at the save, which covers the canvas and `fluksio
sync` at once, and deleting one a node still asks for says which node.
Four sizes are seeded on an installation that has none, and never re-seeded:
re-adding one somebody deliberately removed is an argument nobody wins.
The engine answered "where does this node run" twice, in two ways that could
not see each other: a device sent it to a worker carrying that label, and
resources were counted against the engine's own cores. Declaring both meant the
second answer won and nothing was counted at all — which the data-science
getting-started page and the worked example both do.
One question now, in flow/placement.py: of every machine attached, which could
grant what this node asked for, and which of those has it free. The books move
onto each machine — one accountant per worker, built from the inventory it
reported — and the waiting moves above them, where one condition variable can
be woken by a release anywhere or by a worker attaching. Locks go one way:
placer, then a machine's books, never back.
So a node asking for a card now finds the box that has one, rather than being
clamped down to none and run here. When nothing can grant the ask at all it is
still cut down and run — a flow written on a cluster has to work on a laptop —
but the ceiling is one real machine now, since taking the largest of each
dimension separately can describe a machine nobody has.
Two things fixed on the way. A device on a connector node held every batch run
of its flow forever, waiting for a worker that could never run an entry point.
And `prefer` falling back to the engine skipped the books, so the fallback held
nothing.
The bench flow's node has taken a `params` argument that with_settings has not
forwarded for some time, so the benchmark could not run at all: 62 ms median
submit-to-result with this, against the 61 ms on record.
The store answers the new-node template when nothing was ever written for a
node, so such a node ran — returning {} on every call, reporting active and
ok, and saying nothing anywhere. Unreachable through `fluksio sync`, which
writes every body before it publishes; the editor end was open.
A run of a flow holding one is now refused, and the flow carries a
missing_source issue so it is visible before anybody runs it. A draft is
exempt: a node being written legitimately has no published body yet.
The generated client is regenerated for the new issue code, which also
catches up the drift left by earlier backend work (resources, code_digest,
idempotency_key).
The worker held each yield one behind, because the last one is the node's
result when the generator returns nothing of its own. Only the engine knows
what ports a node declared, so the check happened when the *next* yield
arrived — a pass late, which for a training loop is however long one epoch
takes.
The worker now sends every yield as it happens and returns whatever its
generator returned; EmitSink holds the last one back and decides at the end
of the call what it was. Old "emit" frames are still handled, so a remote
agent that has not been restarted keeps working.
Five concurrent training nodes, each sizing its thread pool to every core,
left the engine's own event loop unscheduled: the API stopped answering
within 10 s and every client died. The same shape on a GPU deadlocked a run
for 21 minutes at 0% utilisation with nothing failing and nothing to read --
it just sat in `running`.
@node(resources={"cpus": 2}) is the declaration. The engine holds that much
for the length of the execution, so more of them than the machine has room
for wait their turn rather than oversubscribing it, and a `gpus` node holds
its card exclusively. FLOW_CPUS defaults to every core but two, and those two
are what keeps the engine answering.
Because a thread cap is read when the process imports the library, a warm
worker cannot be told a different one -- so an environment gets a pool of its
own and nodes deriving the same one share it, rather than paying a cold start
per call on exactly the nodes whose imports are slowest. XLA_FLAGS is never
derived: it is a composed, version-dependent string, so it travels in
resources.env where it is visible.
A node that declares nothing is not accounted for and behaves as it always
did -- it just gets FLOW_CPUS/FLOW_MAX_WORKERS as a thread cap, which is the
half of this that fixes the reported incident without anybody declaring
anything. An operator who set OMP_NUM_THREADS themselves still wins.
Resources are claimed strictly before a worker slot, so the two blocking
waits cannot deadlock. A node queued for them publishes node_queued and shows
on GET /workers/resources, because waiting and hanging looked identical.
Accounted, not enforced: no cgroups, no rlimits. Scheduling across machines,
flavours and enforcement are the next steps.
Three faults with one root: the stored body of a code-defined node is an
import shim, and nothing that mattered was ever read from the code itself.
- The run stamp could not identify what ran. The shim imports whatever is on
disk when the worker starts, and an uncommitted tree stamps <commit>-dirty
for every run it ever produces. Run.code_digest hashes the repository's .py
files, memoized on their stat state, and it is read again when the run is
actually claimed -- so a sweep queued for hours records the code each of its
runs executed, not the code that was there when it was submitted.
- The stage cache adopted code that was too new. The fingerprint hashed the
shim, which is invariant under any edit to the imported function or anything
it calls into, so a re-run was served from cache and answered without the
outputs the edit added. It now carries the repo digest and the node's
declared ports. Every fingerprint changes once, which invalidates the
existing cache; a canvas flow has no repository and keys as before.
- An interrupted sync looked like a hand-edited canvas. The engine answers a
new-node template for a node with no stored body, and the template carries
no marker, so the drift check read "somebody edited this" and demanded
--force -- for the one state that re-running the sync is the fix for.
NodeSource.missing states the fact, and sync skips those and reuses the
bodies it read instead of asking for each one twice.
`ADVISORY_ISSUES` moves next to `ValidationIssue` in pipeline.py, and the
model derives an `advisory` flag from its own code, so the distinction the
engine already made ships to the client instead of being re-guessed there.
The dock keeps its summary in `--destructive` only when a real fault is
among the issues and paints an advisory row `--muted-foreground`; the
canvas leaves advisories off a node's dot and border entirely, since node
status has three colours and no warning tier.
biome checks the generated `openapi.json`, which nothing formats since the
SDK script dropped its format pass — ignore it like the other generated
files.
Four things the python SDK turned up, each fixed where every client sees it.
A key no port declares is now an error rather than a silent drop, on the
return, the yield and the emit alike — the contract the docs already stated.
The SDK reads literal yields at sync time, so a typo fails before anything
runs, and an emission of one fails the call rather than being logged where
nobody looks.
NaN and infinity are refused at the port. JSON cannot spell either, so one
that travelled came back as a 500, a socket frame that stopped the canvas, or
a metric batch the database dropped whole.
An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the
engine — so the CLI, the run dialog and a python caller mean the same thing,
and a sweep can pass one at all.
Node timeouts are off by default. The clock measured silence, which a training
node is full of, and remote workers had already stopped enforcing it — their
heartbeat reset it. Now a heartbeat proves the agent rather than the node,
ninety seconds of nothing fails the call either way, and the engine touches
work it is still running so a long node is not redelivered at sixty seconds.
A code node in a batch run is now fingerprinted by its source, its raw
settings and the values it reads — an artifact input counting as its digest,
which is what the content addressing was always for. A run that finds the key
restores what the earlier one returned and skips the node, recorded as
`cached`. The run history is the cache: `run_node.outputs` beside the
`cache_key` the schema already had, no second store. On for code nodes, never
for the built-in and connector types that have side effects; off per node with
`@node(cache=False)` and per run with `--no-cache`.
Emissions are not replayed on a hit, so a cached training node returns its
result without redrawing its curve. Recorded in NOTEPAD.md with the two other
deliberate limits.
`fluksio run --local` boots the real app in the command's own process and
drives it through its ASGI interface behind the ordinary client, so a run no
longer needs a `serve` terminal beside it — same data directory, same history,
and the cache carries between the two. It always waits, because the engine it
starts lives exactly as long as the command.
Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists,
`run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited
run rather than abandoning it, coloured statuses on a terminal, and `name`
made optional on the metrics endpoint so a follower can ask for every series.
The gates have never gone green on the new runners. Three separate reasons:
- backend/Dockerfile shipped Python 3.10 while the code imports typing.Self
and datetime.UTC, so the container exited on import and the suite could not
even load its conftest. The image moves to 3.13 and the packages declare
>=3.12, which is the floor the tests actually pass on; ruff's target follows
and rewrites timezone.utc and asyncio.TimeoutError accordingly. Relocking
drops the 3.10 branch, which bumps FastAPI and so regenerates the SDK.
- frontend/README.md had no trailing newline and two dashboard widgets used
arbitrary text-[…] sizes. Both are em-relative on purpose, so they move to
the inline style the neighbouring ramp already uses.
- Every commit left its own run queued: without a concurrency group a runner
that was offline for a while works through a backlog nobody reads. A stack
that fails to come up now prints its logs before the teardown removes it.
The module docstrings and the notepad still described one rebuild that
touches everything. Closes the toggle cost, the seeding cost, the
per-save rebuild, the modules/apply rebuild and the Playwright spec that
could not fit a rebuild into its five seconds; files the follow-ups the
refactor leaves behind.
A stopped flow's nodes are built like any other flow's — being stopped
means having no subscriptions, schedules or webhooks, not being absent —
so a toggle only ever needed the lifecycle call and the gate that goes
with it. It was doing a whole-pipeline rebuild instead, which on a
populated installation is every node in every flow reconnecting.
A node's timeout now covers its body only: the pool loads the source into the
worker it picked, off the node's budget, so imports that outlast the timeout no
longer make a node impossible to run. Draft checks compile without caching, so
saving does not evict what a busy node is serving calls from. Requests carry an
id the worker echoes and the pool checks, a reply is encoded once, and the
remote-exception cache is bounded.
DELETE /flows/{name} answers 409 while the flow has a running or queued run,
which is what was letting run_node rows outlive their run.
A node's stop() and a supervised task's cancellation are both waited on
inside the rebuild lock, and neither had a deadline: an MQTT client whose
broker never acknowledges the disconnect leaves aiomqtt's __aexit__
waiting forever, so reload() never returned and every start, stop or
publish behind it hung until the container was restarted.
Each node now gets five seconds to close and is abandoned after that, and
cancel_all reports what is still running rather than waiting on it — it
also no longer swallows a cancellation aimed at the caller, which used to
make the lock holder unkillable. A rebuild asked for by a request gives up
on the lock after fifteen seconds with RebuildBusy, answered as a 503.
- A node's height follows the ports on its busiest side. It is a function of
the document, so `layoutGraph` reserves exactly what is drawn and nothing
measured is fed back into the layout.
- The three status controls now sit in slots that are there whether the
control is or not. A node running many times a second mounted and unmounted
the stop button on every execution, resizing the card each time.
- A port bound to another flow's message is drawn as a label, naming the node
at the far end and its type. Only the opposite direction was answered
before. The scan behind both is now cached on the store's commit counter
rather than reading every flow per request.
A cluster or GPU host installs `pip install fluksio-worker` and gets the
agent and the runner, not psycopg, numpy and the MCP SDK. The engine
depends on it as a workspace member, so the file it launches node code
with is the same file a remote worker runs — which is what keeps a node
unable to tell the difference. Copying the two files by hand still works.
A wheel whose top-level module is `app` collides with anything else in a
user's venv, so the package that is about to be published takes the name
it is published under. Only the Python package moves; the repo, the
Docker WORKDIR and the compose project keep theirs.
Melvin Strobl
stroblme