The poll loop remembered what it read rather than what it published, so a
value the node could not publish counted as said: the next poll skipped it,
succeeded, and health went back to ok with the port still dark. Remember it
only after inject returns, and report ok last.
A node reporting itself down is now derived into its flow's issues on read
and counted on the health summary, so the canvas marks it and Home says so.
Being down does not stop the flow, and the issue clears by itself when the
node reports well again. The repeating poll warning is logged once per
outage rather than once per tick.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1moruzue2kTJd3uVisgNk
Concurrent runs sat at 4 whatever FLOW_MAX_CASCADES said: that setting bounds
cascades, and the run drivers read a hardcoded MAX_PARALLEL nobody could reach.
FLOW_MAX_RUNS is the knob they read now, --max-runs/--max-cascades/--max-workers
are the same three as flags on serve, and the engine says which numbers it
started with — which is the only way to tell that a settings file was read.
Events keep the run they happened in. The payload always carried it and the
persist path dropped it, so reading one run's failures meant filtering the
engine-wide list; a batch run's id reaches those events now too, since a run
has no journaled item to name itself by.
Also: a provisioner's 0 means "no deadline" rather than "cancel on the next
reconcile", and a command that reaches no engine says how to start one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015sbYeYaVgYQqm1sbx7wPdL
The timer thread promoted due work on a fixed one-second tick, so every
delayed item was 0-1000ms late whatever the load — measured on the house
at 705ms mean on a rollershutter stop, which is 2-4% of a 26-second
travel and accumulates in the position the motor node believes it is at.
It now sleeps to the soonest deadline and is woken when a nearer one is
scheduled, which measures 0.9ms end to end through Redis.
A promoted timer also went to the back of the queue. It goes into a due
lane of its own that `claim` reads first, so work that has waited out a
deadline is not held up by work that is merely queued.
Beside it, in the same code: seeding a message now bumps its version, so
a re-put flow's synchronous nodes no longer wait forever on a value that
is sitting in state; the consumer group drops the consumers of engines
that are gone (138 had accumulated on this installation); and the cast
that closes the long-standing `xclaim` mypy error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A port may now declare `image`, `audio` or `video`. Each is the artifact
reference the engine already had, narrowed by the `media_type` on it, so a
speech recogniser declares what it eats rather than taking any bytes at all and
finding out. Bytes still never travel as a message and nothing on the wire
stops being JSON: a camera publishes one reference per frame, a microphone one
per chunk, and a reference may carry a `meta` dict nothing here interprets.
Streaming media is therefore an ordinary streaming port — with one change to
what that means. An emission used to journal an item with no payload, so
downstream read whatever was current when the item was claimed; a consumer
slower than its producer saw only the newest chunk and the ones between were
lost. That is right for a training curve and wrong for a second of speech, so
an emission now journals a `kind="emission"` item carrying its values, and the
executor hands them to the nodes reading that message instead of writing them
to state again. The value in state stays the latest, which is what everything
else reads, and the wave is filtered by what actually changed rather than
walking everything reachable. No queue serialization change — the existing
`outputs` field carries it.
Continuous media makes the store's missing GC a real problem, so this closes
it: `sweep_artifacts` runs hourly, keeps every digest a `run_artifact` row
records or a live message holds, spares anything written in the last hour, and
stands aside entirely while a run is in flight, since a node may store a
checkpoint long before it returns the reference to it. That also collects the
orphans a deleted flow has always left behind. `ARTIFACT_GC_INTERVAL_S=0` turns
it off.
Around the edges: `GET /artifacts/{digest}` serves the media type the caller
passes and answers ranged requests, so a browser plays a clip rather than
downloading it; `PUT` spools to disk instead of holding the whole body in
memory, as does `save_artifact` given a path; a Media widget draws whatever its
message points at, and a wall panel may fetch the bytes its own tiles are
showing and nothing else; and a connector gets `save_artifact`, for a device
whose readings are bytes.
What this cannot do is live video: a frame every second or two is a glance, and
the honest answer above that is the camera's own stream, which the widget takes
as a URL and the browser plays from source.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The store answers the new-node template when nothing was ever written for a
node, so such a node ran — returning {} on every call, reporting active and
ok, and saying nothing anywhere. Unreachable through `fluksio sync`, which
writes every body before it publishes; the editor end was open.
A run of a flow holding one is now refused, and the flow carries a
missing_source issue so it is visible before anybody runs it. A draft is
exempt: a node being written legitimately has no published body yet.
The generated client is regenerated for the new issue code, which also
catches up the drift left by earlier backend work (resources, code_digest,
idempotency_key).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A cache key held qualified input names, so the same node reading the same
values through two flows keyed differently and only a node with no inputs
could ever hit across one. The fingerprint beside the key already says what
the node is, and it has been flow-agnostic since it moved ahead of
assign_flow — the names were the last thing tying an entry to one flow.
Inputs now reduce by the node's own name for them; a name belonging to
another flow keeps its prefix, since reading it is part of what the
execution is. Every stored entry misses once and is re-run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The engine was I/O-bound on its own state backend. `RedisState.lock()` is one
key — `pipeline:_lock` — for the whole process, taken five times a message at
two round trips each, and every cascade and every node read queued behind it.
Inside it, reading a node's inputs was three round trips per input (an EXISTS
for `in`, then EXISTS and GET for the value), writing was two updates that a
single transaction already gives, and the version counters went one INCR at a
time.
Replaced with the atomic command that was always available: `get_present` is
one MGET and tells a missing key from one holding null, so the lock it used to
be read under bought nothing; value and timestamp land in one `update`, which
is a MULTI/EXEC; `increment_multi` pipelines the counters. `values()` — what
every websocket snapshot calls — is two reads whatever the message count
instead of two per message.
Beside that: every webhook did its blocking XADD on the asyncio event loop
(MQTT already used `to_thread`); the per-execution `NodeOutcome` was built and
validated even with no run watching; `_minute` built a tz-aware datetime per
event on the loop thread to key a dict, and now keys on an int; `move_due`
promoted delayed items one round trip each, every second; `FLOW_MAX_CASCADES`
makes the in-flight ceiling a setting rather than a constant.
`orjson` replaces stdlib json where a message pays for it — state, the
journal, the engine side of the worker pipe. `fluksio-worker` stays
dependency-free, and the run-cache digest stays on stdlib so no stored key is
invalidated. A non-finite number now stores as `null` rather than the bare
`NaN` that was never JSON.
Measured with `scripts/bench_engine.py` against a real Redis, 200 messages:
a five-node chain went from 43.9 to 103.1 msg/s with p50 latency 2110ms →
782ms and p95 3913ms → 1439ms; one source into twenty consumers went from 5.4
to 33.7 msg/s. In memory, twenty consumers went from 187 to 448 msg/s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
The limit was applied in `apply_outputs`, which the executor reaches after the
item is off the queue — so a subscriber told to publish every 15s still cost a
queue entry, a `cascade_started`, a run record and a walk of everything
reachable from it per inbound message. Seven relay nodes behind one inverter
ran 192 times a minute to publish six.
Two halves, matching the two shapes it takes:
`trigger()` now keeps a value whose every port is inside its window and
journals nothing at all. The window split came out of `_throttled` as a
read-only `_window_split`, so the question is asked the same way in both
places and the exact split is still made once, at claim time.
A cascade carries the names it actually published, and the wave runs only the
nodes something in that set feeds. A node whose triggering inputs were all
held back is completed without running, which frees its own consumers to be
judged the same way — the case where a node re-published 619 messages a minute
off inputs that changed six times. Redeliveries and emissions carry no such
set and still walk everything, since one has a half-finished wave to finish
and the other is the value already being in state.
Skipping a node can make one ready that the scheduling pass has already walked
past, so `submit_ready` runs to a fixpoint. That also closes the same latent
hole on the replay path, where a done-marker skip could strand a join with no
future outstanding to come back for it.
Measured with the new `scripts/bench_engine.py`, 500 messages through the
house's shape: a limited source went from 500 cascades / 3500 node runs /
5009 events to 1 / 7 / 19, publishing the same 8 values; an unlimited source
into limited relays took the node reading them from 500 runs to 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
`ADVISORY_ISSUES` moves next to `ValidationIssue` in pipeline.py, and the
model derives an `advisory` flag from its own code, so the distinction the
engine already made ships to the client instead of being re-guessed there.
The dock keeps its summary in `--destructive` only when a real fault is
among the issues and paints an advisory row `--muted-foreground`; the
canvas leaves advisories off a node's dot and border entirely, since node
status has three colours and no warning tier.
biome checks the generated `openapi.json`, which nothing formats since the
SDK script dropped its format pass — ignore it like the other generated
files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Gf7WaExcJ9bs3kfJXB3nK
A cache hit still replays no emissions — those values were the story of an
execution that is not happening — but the run they were recorded in is now
written on the row (`run_node.cached_from`), and the metrics endpoints read the
series back from there. So a reused run answers `run.metrics("train.loss")`
with the same points the run that trained did, rather than looking like a run
that produced no numbers at all. Pointed at rather than copied: a sweep of 500
reusing one frozen node would otherwise duplicate its curve 500 times.
That needed the cross-flow restore fixed first. The cache key has no flow in
it while the stored outputs are named for the flow that produced them, so
`quick.prepare` getting a hit from `train` wrote `train.dataset` into `quick`'s
state and the next node was called without its argument. One rule now covers
both halves: `requalify` reads a name owned by one flow as the same name in
another, applied to the restored outputs, to the node id behind the pointer,
and to the series names on the way out. Reuse across flows is kept.
Also: `@run:<id>.<output>` and a bare `sha256:` digest resolve on every input,
not only artifacts. Chaining a run's json config into the next one from a shell
meant pasting the whole object inline, and the CLI could not even send the
spelling — `_coerce` died in `json.loads` before the engine saw it. Both
spellings are reserved on every input now, `str` included, and `_from_run`
returns whatever the run's result holds rather than only a reference.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dp9L6gakMVro1K2C5zdtBE
A code node in a batch run is now fingerprinted by its source, its raw
settings and the values it reads — an artifact input counting as its digest,
which is what the content addressing was always for. A run that finds the key
restores what the earlier one returned and skips the node, recorded as
`cached`. The run history is the cache: `run_node.outputs` beside the
`cache_key` the schema already had, no second store. On for code nodes, never
for the built-in and connector types that have side effects; off per node with
`@node(cache=False)` and per run with `--no-cache`.
Emissions are not replayed on a hit, so a cached training node returns its
result without redrawing its curve. Recorded in NOTEPAD.md with the two other
deliberate limits.
`fluksio run --local` boots the real app in the command's own process and
drives it through its ASGI interface behind the ordinary client, so a run no
longer needs a `serve` terminal beside it — same data directory, same history,
and the cache carries between the two. It always waits, because the engine it
starts lives exactly as long as the command.
Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists,
`run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited
run rather than abandoning it, coloured statuses on a terminal, and `name`
made optional on the metrics endpoint so a follower can ask for every series.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The module docstrings and the notepad still described one rebuild that
touches everything. Closes the toggle cost, the seeding cost, the
per-save rebuild, the modules/apply rebuild and the Playwright spec that
could not fit a rebuild into its five seconds; files the follow-ups the
refactor leaves behind.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StpRc2C6au1WJ1EUU7fsfu
A wheel whose top-level module is `app` collides with anything else in a
user's venv, so the package that is about to be published takes the name
it is published under. Only the Python package moves; the repo, the
Docker WORKDIR and the compose project keep theirs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
stroblmeandClaude Opus 5