Three faults with one root: the stored body of a code-defined node is an
import shim, and nothing that mattered was ever read from the code itself.
- The run stamp could not identify what ran. The shim imports whatever is on
disk when the worker starts, and an uncommitted tree stamps <commit>-dirty
for every run it ever produces. Run.code_digest hashes the repository's .py
files, memoized on their stat state, and it is read again when the run is
actually claimed -- so a sweep queued for hours records the code each of its
runs executed, not the code that was there when it was submitted.
- The stage cache adopted code that was too new. The fingerprint hashed the
shim, which is invariant under any edit to the imported function or anything
it calls into, so a re-run was served from cache and answered without the
outputs the edit added. It now carries the repo digest and the node's
declared ports. Every fingerprint changes once, which invalidates the
existing cache; a canvas flow has no repository and keys as before.
- An interrupted sync looked like a hand-edited canvas. The engine answers a
new-node template for a node with no stored body, and the template carries
no marker, so the drift check read "somebody edited this" and demanded
--force -- for the one state that re-running the sync is the fix for.
NodeSource.missing states the fact, and sync skips those and reuses the
bodies it read instead of asking for each one twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The engine was I/O-bound on its own state backend. `RedisState.lock()` is one
key — `pipeline:_lock` — for the whole process, taken five times a message at
two round trips each, and every cascade and every node read queued behind it.
Inside it, reading a node's inputs was three round trips per input (an EXISTS
for `in`, then EXISTS and GET for the value), writing was two updates that a
single transaction already gives, and the version counters went one INCR at a
time.
Replaced with the atomic command that was always available: `get_present` is
one MGET and tells a missing key from one holding null, so the lock it used to
be read under bought nothing; value and timestamp land in one `update`, which
is a MULTI/EXEC; `increment_multi` pipelines the counters. `values()` — what
every websocket snapshot calls — is two reads whatever the message count
instead of two per message.
Beside that: every webhook did its blocking XADD on the asyncio event loop
(MQTT already used `to_thread`); the per-execution `NodeOutcome` was built and
validated even with no run watching; `_minute` built a tz-aware datetime per
event on the loop thread to key a dict, and now keys on an int; `move_due`
promoted delayed items one round trip each, every second; `FLOW_MAX_CASCADES`
makes the in-flight ceiling a setting rather than a constant.
`orjson` replaces stdlib json where a message pays for it — state, the
journal, the engine side of the worker pipe. `fluksio-worker` stays
dependency-free, and the run-cache digest stays on stdlib so no stored key is
invalidated. A non-finite number now stores as `null` rather than the bare
`NaN` that was never JSON.
Measured with `scripts/bench_engine.py` against a real Redis, 200 messages:
a five-node chain went from 43.9 to 103.1 msg/s with p50 latency 2110ms →
782ms and p95 3913ms → 1439ms; one source into twenty consumers went from 5.4
to 33.7 msg/s. In memory, twenty consumers went from 187 to 448 msg/s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
The limit was applied in `apply_outputs`, which the executor reaches after the
item is off the queue — so a subscriber told to publish every 15s still cost a
queue entry, a `cascade_started`, a run record and a walk of everything
reachable from it per inbound message. Seven relay nodes behind one inverter
ran 192 times a minute to publish six.
Two halves, matching the two shapes it takes:
`trigger()` now keeps a value whose every port is inside its window and
journals nothing at all. The window split came out of `_throttled` as a
read-only `_window_split`, so the question is asked the same way in both
places and the exact split is still made once, at claim time.
A cascade carries the names it actually published, and the wave runs only the
nodes something in that set feeds. A node whose triggering inputs were all
held back is completed without running, which frees its own consumers to be
judged the same way — the case where a node re-published 619 messages a minute
off inputs that changed six times. Redeliveries and emissions carry no such
set and still walk everything, since one has a half-finished wave to finish
and the other is the value already being in state.
Skipping a node can make one ready that the scheduling pass has already walked
past, so `submit_ready` runs to a fixpoint. That also closes the same latent
hole on the replay path, where a done-marker skip could strand a join with no
future outstanding to come back for it.
Measured with the new `scripts/bench_engine.py`, 500 messages through the
house's shape: a limited source went from 500 cascades / 3500 node runs /
5009 events to 1 / 7 / 19, publishing the same 8 values; an unlimited source
into limited relays took the node reading them from 500 runs to 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
`RedisWorkQueue.stats` read XPENDING, which counts entries delivered to a
consumer and not yet acknowledged — work in progress. Entries sitting in the
stream undelivered were counted nowhere, so an engine hours behind reported
itself idle: on the house, `pending: 4` while the group's lag was 1554.
The group's own `lag` is the missing number. `backlog` now carries it on both
queues (`len(_items)` in memory), leads the health tile, and a sustained one
publishes `engine_degraded` from the timer thread — named with the flow most
of the waiting work belongs to, sampled from the undelivered tail, since that
is the actionable half. It is a summary problem rather than a /utils/health
503: a backlog should not restart the container.
Also drops the keyspace `scan_iter` `stats()` did per poll to count parked
items — it walked every state and idempotency key twice per ten seconds — for
a set the park/unpark path maintains.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BpfSinyCBfjuieikyfMPbf
`ADVISORY_ISSUES` moves next to `ValidationIssue` in pipeline.py, and the
model derives an `advisory` flag from its own code, so the distinction the
engine already made ships to the client instead of being re-guessed there.
The dock keeps its summary in `--destructive` only when a real fault is
among the issues and paints an advisory row `--muted-foreground`; the
canvas leaves advisories off a node's dot and border entirely, since node
status has three colours and no warning tier.
biome checks the generated `openapi.json`, which nothing formats since the
SDK script dropped its format pass — ignore it like the other generated
files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Gf7WaExcJ9bs3kfJXB3nK
A cache hit still replays no emissions — those values were the story of an
execution that is not happening — but the run they were recorded in is now
written on the row (`run_node.cached_from`), and the metrics endpoints read the
series back from there. So a reused run answers `run.metrics("train.loss")`
with the same points the run that trained did, rather than looking like a run
that produced no numbers at all. Pointed at rather than copied: a sweep of 500
reusing one frozen node would otherwise duplicate its curve 500 times.
That needed the cross-flow restore fixed first. The cache key has no flow in
it while the stored outputs are named for the flow that produced them, so
`quick.prepare` getting a hit from `train` wrote `train.dataset` into `quick`'s
state and the next node was called without its argument. One rule now covers
both halves: `requalify` reads a name owned by one flow as the same name in
another, applied to the restored outputs, to the node id behind the pointer,
and to the series names on the way out. Reuse across flows is kept.
Also: `@run:<id>.<output>` and a bare `sha256:` digest resolve on every input,
not only artifacts. Chaining a run's json config into the next one from a shell
meant pasting the whole object inline, and the CLI could not even send the
spelling — `_coerce` died in `json.loads` before the engine saw it. Both
spellings are reserved on every input now, `str` included, and `_from_run`
returns whatever the run's result holds rather than only a reference.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dp9L6gakMVro1K2C5zdtBE
Two halves of the same gap: the CLI could start work but not show you any.
`fluksio status` draws the home screen's top half in a terminal — health and
what is wrong with it, every flow with its state and node count, and the
recent runs and failures under them. `--watch` keeps it there. Rich does the
drawing; it was already installed under fastapi's own CLI, and is named now
because a command depends on it.
`fluksio run` with no parameters at a terminal asks for them, one line per
declared input with its declared value in brackets — so Enter through the lot
is what running the defaults looks like, and an artifact input takes the
`@run:` spelling the engine now resolves. A scripted run is untouched: passing
any parameter, or piping the command, skips the questions, as does --defaults.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
Four things the python SDK turned up, each fixed where every client sees it.
A key no port declares is now an error rather than a silent drop, on the
return, the yield and the emit alike — the contract the docs already stated.
The SDK reads literal yields at sync time, so a typo fails before anything
runs, and an emission of one fails the call rather than being logged where
nobody looks.
NaN and infinity are refused at the port. JSON cannot spell either, so one
that travelled came back as a 500, a socket frame that stopped the canvas, or
a metric batch the database dropped whole.
An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the
engine — so the CLI, the run dialog and a python caller mean the same thing,
and a sweep can pass one at all.
Node timeouts are off by default. The clock measured silence, which a training
node is full of, and remote workers had already stopped enforcing it — their
heartbeat reset it. Now a heartbeat proves the agent rather than the node,
ninety seconds of nothing fails the call either way, and the engine touches
work it is still running so a long node is not redelivered at sixty seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
A code node in a batch run is now fingerprinted by its source, its raw
settings and the values it reads — an artifact input counting as its digest,
which is what the content addressing was always for. A run that finds the key
restores what the earlier one returned and skips the node, recorded as
`cached`. The run history is the cache: `run_node.outputs` beside the
`cache_key` the schema already had, no second store. On for code nodes, never
for the built-in and connector types that have side effects; off per node with
`@node(cache=False)` and per run with `--no-cache`.
Emissions are not replayed on a hit, so a cached training node returns its
result without redrawing its curve. Recorded in NOTEPAD.md with the two other
deliberate limits.
`fluksio run --local` boots the real app in the command's own process and
drives it through its ASGI interface behind the ordinary client, so a run no
longer needs a `serve` terminal beside it — same data directory, same history,
and the cache carries between the two. It always waits, because the engine it
starts lives exactly as long as the command.
Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists,
`run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited
run rather than abandoning it, coloured statuses on a terminal, and `name`
made optional on the metrics endpoint so a follower can ask for every series.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow this serves: make a venv, install what you work with, then `pip
install fluksio` into the same one. Building a second environment beside it
was exactly wrong — the packages the nodes need are already here, and the
Modules screen was asking for them a second time.
`NODE_VENV=auto` (the default) adopts that venv. It declines in the three
cases where adopting would be wrong: `managed` says otherwise, a managed venv
already exists and may hold packages somebody installed on purpose, or the
engine is not running from a venv at all. The images set `managed`, since the
venv in them holds the app and nothing of anybody else's.
An adopted venv is never written to. `uv pip sync` makes a venv hold exactly
the manifest, so pointed at somebody's own environment it uninstalls their
work and the engine with it — `sync()` refuses outright and `reconcile()`
returns before it can be called at startup, which is where that would have
happened first. The Modules screen lists what is installed and drops its
editor; `pip` is how that environment changes.
`fluksio serve` now names the interpreter node code runs on, which is the
thing a data scientist most needs to know at that moment. `fluksio-worker`
already defaulted `--python` to its own interpreter, so a GPU box works the
same way — that was only ever undocumented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
A dashboard is a wall panel somebody hangs in their own hallway, so it
now wears what they choose: a look, and a palette of their own colours.
Two complete component sets live under `Dashboard/ui/` — `glass`
(translucent panes over a slowly moving ground) and `material` (Material
3 tonal cards) — behind one prop contract. Every control's state,
keyboard and `aria-` live in `ui/core` and are shared, so the two sets
are the same dashboard drawn twice rather than two products: a set only
decides what a control looks like while doing it.
Four settings join the channel, each drivable by a flow like any other:
`look`, `palette`, `background` and `touch`. A palette is an ordered list
of hex colours — background, surface, primary, accent, text, then more
chart colours — pasted from a coolors.co link or typed, written onto the
canvas as the token variables everything already reads. Trailing roles
are derived, so three colours are a whole dashboard, and derived text is
held to AA rather than trusted (`theme.check.ts` measures it). A palette
also decides light or dark, since its first colour is the ground.
Widgets are measured against their own tile with container queries rather
than against the viewport, animate through `motion`, and can be drawn
without their title. The three reworks:
- a bar draws a row per reading, up to eight, each in the dashboard's own
data colours and each able to carry its own scale — replacing readings
nested in one fill, which could only ever share one colour and stop at
three. Documents written the old way are read as rows.
- a chart's range picker moved to a column down its right-hand edge, which
gives the plot back a whole row of a short tile.
- the colour wheel became a disc: hue is the angle and saturation the
distance from the middle, so a colour is one gesture rather than three,
with brightness on a slider beside it.
`index.css` and `lib/motion.ts` are untouched — the dashboard overrides
token *values* on its canvas, never the blocks the two repos share.
A stopped flow's nodes are built like any other flow's — being stopped
means having no subscriptions, schedules or webhooks, not being absent —
so a toggle only ever needed the lifecycle call and the gate that goes
with it. It was doing a whole-pipeline rebuild instead, which on a
populated installation is every node in every flow reconnecting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StpRc2C6au1WJ1EUU7fsfu
A node's timeout now covers its body only: the pool loads the source into the
worker it picked, off the node's budget, so imports that outlast the timeout no
longer make a node impossible to run. Draft checks compile without caching, so
saving does not evict what a busy node is serving calls from. Requests carry an
id the worker echoes and the pool checks, a reply is encoded once, and the
remote-exception cache is bounded.
DELETE /flows/{name} answers 409 while the flow has a running or queued run,
which is what was letting run_node rows outlive their run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StpRc2C6au1WJ1EUU7fsfu
A node's stop() and a supervised task's cancellation are both waited on
inside the rebuild lock, and neither had a deadline: an MQTT client whose
broker never acknowledges the disconnect leaves aiomqtt's __aexit__
waiting forever, so reload() never returned and every start, stop or
publish behind it hung until the container was restarted.
Each node now gets five seconds to close and is abandoned after that, and
cancel_all reports what is still running rather than waiting on it — it
also no longer swallows a cancellation aimed at the caller, which used to
make the lock holder unkillable. A rebuild asked for by a request gives up
on the lock after fifteen seconds with RebuildBusy, answered as a 503.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01StpRc2C6au1WJ1EUU7fsfu
- A node's height follows the ports on its busiest side. It is a function of
the document, so `layoutGraph` reserves exactly what is drawn and nothing
measured is fed back into the layout.
- The three status controls now sit in slots that are there whether the
control is or not. A node running many times a second mounted and unmounted
the stop button on every execution, resizing the card each time.
- A port bound to another flow's message is drawn as a label, naming the node
at the far end and its type. Only the opposite direction was answered
before. The scan behind both is now cached on the store's commit counter
rather than reading every flow per request.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ZeGnqVsf5VHQqvz4HdUhN
Four small things, each with a device behind it.
An MQTT filter now routes what it subscribed to. `+` and `#` reached the
broker and were then looked up in an exact-match dict, so every message a
wildcard subscription received was dropped in silence.
`json_key` lifts a value out of the object a device wraps it in — Victron
publishes `{"value": 47}` on every path, which was otherwise a Python node
per port.
The trigger node learned `passthrough` and `wait_port`, because how long to
wait can be a value rather than a constant: a rollershutter takes 26 seconds
up and 28 down. A wait of zero sends nothing afterwards and still cancels
what the last message scheduled, which is how a stop is commanded once
instead of forever.
The HTTP sender takes fixed `query` parameters, so an API key is a secret
reference rather than a message on the canvas, and `send_inputs` off for a
request whose inputs are only a trigger.
Also: `delay` accepts fractional seconds, and `TZ` reaches the container, so
a cron expression means local time. Left unset it is UTC, as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dashboard could only ever receive as a set of tiles. This adds the dashboard
itself as a receiver: `settings` maps a name to a value plus an optional
binding. Unbound, the setting is simply its value — a wall panel that is always
dark costs no flow. Bound, a flow drives it live and the value is the fallback.
Two settings are wired: `theme` (system/light/dark) and `locked` (read-only).
There is no schedule field on purpose — a node publishing to the bound message
on a cron is what a schedule is here, which is the point of a channel.
- `messages_for()` now walks a dashboard's bound settings as well as its
widgets' bindings. Without this a paired screen is refused its own theme
message, on the one surface the setting exists for; it bounds the socket too.
- `locked` is gated in `usePublish`, so every control inherits it, and each
control also draws itself disabled — a dead button reads as broken otherwise.
The panel surface says Read-only in the corner.
- The theme is a class on the dashboard's own surface, never the root: inside
the app shell it must not flip the chrome. `.light` gains the tokens `.dark`
already had (mirrored in the index repo) so both directions work on a subtree.
- Settings bindings are type-checked from the document alone, the rule widget
bindings follow, and mirrored on the server.
- A bound setting is drawn on the flow canvas as a dashboard-level endpoint.
- The demo's house flow now publishes `home.panel_theme`, which the demo
dashboard's theme binds to: the panel goes dark after sunset, at no tile cost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018tULRZJUkZsw7rMJ3h4xvu
A custom hue ring — a conic gradient, not a canvas — with saturation and
brightness sliders beside or under it depending on the tile's shape, sized
for a wall panel and reachable from a keyboard. It publishes [h, s, v] by
default, which is what the reference installation's DMX encoders read, and
`format` switches that to [r, g, b] or "#rrggbb".
`usePublish` moves to its own module so a widget in a file of its own can
reach it without importing `widgets.tsx` back.
One process owns this database — the image has run a single uvicorn
worker for that reason since the four-engines bug — so a file beside the
flows is the honest shape for it, and it is what lets `fluksio serve`
need no infrastructure at all. Live values, node execution and the work
queue never came here anyway; what does is a rollup a minute at a time,
a row per cascade and the run history, and WAL keeps the readers going
while that one writer works.
DATA_DIR is now the one setting that moves everything an installation
keeps; the rest derive from it and the images still spell theirs out.
The schema is prepared in-process at startup, so the prestart service is
gone, and the ten Postgres-only revisions collapse into one portable
baseline.
Three things only worked because psycopg was casting for us: a token's
subject arriving as a string where the column is a UUID, `greatest`, and
`date_bin`. The timestamps needed a column type of their own — SQLite
stores no offset, and a naive datetime read back either raises against an
aware `now` or serialises as local time.
Postgres stays in the stack only for Umami, behind the analytics profile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A cluster or GPU host installs `pip install fluksio-worker` and gets the
agent and the runner, not psycopg, numpy and the MCP SDK. The engine
depends on it as a workspace member, so the file it launches node code
with is the same file a remote worker runs — which is what keeps a node
unable to tell the difference. Copying the two files by hand still works.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A wheel whose top-level module is `app` collides with anything else in a
user's venv, so the package that is about to be published takes the name
it is published under. Only the Python package moves; the repo, the
Docker WORKDIR and the compose project keep theirs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`make lint-frontend` was `biome check --write --unsafe ./` — a lint target that
reformatted the whole tree rather than checking it, which is why every parallel
change in this repo has had to work around it. `lint` checks now and a new
`format` writes. The pre-commit hook and CI needed no edit at all: both call
`bun run lint`, so they became checks the moment its meaning changed.
`app/Makefile` assigned DOMAIN from .env, and a plain assignment beats an
inherited environment variable and is not exported — so `cd app && make
dev-local` served localhost while the same checkout's tests targeted the
configured domain. `export DOMAIN ?=` gives the lattice that was intended:
command line, then environment, then .env.
Alongside: the backend's htmlcov bind mount created that directory as root, so
`make test-backend` died on the coverage step after every test had passed,
which reads like a test failure and is not one. The alerts screen's copy of
ALERTING_EVENTS is now checked by a test rather than trusted. And the shard
comment claimed two spec files where there are nine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uq8mtNb97A7praJLyeEYgs
A node's error cleared the moment it ran again, so a failure that genuinely
fired an alert could leave no trace on the canvas by the time anyone looked.
The engine records it now — on the node's status, so it survives a reload and
every client agrees — and reading the traceback is what clears it. The seam is
the event bus, which is where every failing path already meets: a queued live
run, an explicit run, a preview, and a single triggered node all publish
`node_error`, while the controller's own observer would have seen only one of
them.
That was half the confusion. The other half: clicking a failed neuron on Home
often landed on a flow where everything looked fine. Nodes merge into one
neuron by instance key — every InfluxDB node pointing at the same bucket is one
neuron — and the click went to whichever flow contributed a member first, not
the one that failed. It now goes to the failing member and selects it, and the
canvas marks a failing node rather than leaving it to the dot alone.
The inject node emitted one payload to every port it declared, whatever their
types, so an inject on a bool port carrying the text "true" raised at publish
time. Each port gets its own field now, typed and parsed by that port's dtype,
and remembers what it last sent. A port that is renamed carries its value with
it; one that is removed takes its value with it. An inject written before this
keeps emitting exactly what it did.
The derived-cron chip also appeared on the delay node, where `interval` is a
rate limit and a schedule derived from it means nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uq8mtNb97A7praJLyeEYgs
A dashboard went live the moment it was created — an empty document straight to
the panels — while a new flow starts as a draft. It now works the way flows do:
published means `dashboard.json` exists, so every dashboard on every running
installation is already published and nothing needs migrating. Only the ones
created from here on start as drafts.
Mirroring FlowStore turned up a latent 500: discarding the draft of a dashboard
that had never been published unlinked its only file, and the read that followed
raised out of a 200 handler. It answers 400 now, the way a flow does.
Publishing all of them was 2N requests, because a publish has to name the
version it expects and the summaries did not carry one. They do now — and so do
the flow summaries, which had the same defect nobody had written down.
A panel had no way to hear about any of this. A publish, or a change to which
dashboards a panel carries, now puts one event on the bus and the screen
refetches what changed: no reload, so a wall display never blanks or asks for
its credential again. The subtle half is that a socket's message allowlist was
computed once at handshake — a reassigned panel would have fetched its new
document and then shown tiles that never updated.
The panels dialog logged non-superusers out. Every write in it needs a
superuser, not only the checkboxes the report mentioned, so the dialog is
read-only for everyone else. The logout itself was `main.tsx` treating 403 as a
dead session, against the contract deps.py spells out: only a 401 ends a
session, and a 403 now says so rather than silently signing someone out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uq8mtNb97A7praJLyeEYgs
The queue shared the `pipeline:` prefix with flow state, so `RedisState.clear()`
could DEL the queue stream and `keys()` enumerated queue entries — only callers
filtering `__`-prefixed names kept it safe. It moves to `queue:` without a
migration: whatever is in flight at the upgrade is dropped once, documented in
DEPLOY.md rather than papered over.
Alongside it: `pool_pre_ping`, so a connection idle across a Postgres restart
costs a round trip instead of a failed request; the test suite pins
ENVIRONMENT=local and DOMAIN=localhost itself rather than inheriting a
deployment's .env; and `depth` leaves the queue stats, where it reported the
capped journal length as if it were a backlog.
ALERTS_FILE and PANELS_FILE now point at /data. They defaulted to a path on no
volume, so alert routing and every wall-panel pairing were living in the
container's writable layer and vanishing on each rebuild. Carrying the existing
files across is a manual step; DEPLOY.md has it.
development.md was still the upstream template — compose.override.yml,
localhost.tiangolo.com, `docker compose watch` as the dev flow — and said
nothing about the Playwright suite. Rewritten against what the Makefiles
actually do. deployment.md was template text too, duplicating the root
DEPLOY.md, and is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uq8mtNb97A7praJLyeEYgs
Two things stopped the engine commanding this house. ConnectorNode hardwired
its node function to a no-op, so an input message reaching a connector was
discarded and Art-Net's packet builder was unreachable; write() now carries
the input ports, which is additive so the contract version holds. And the MQTT
publisher JSON-encoded every payload, so "ON" went on the wire quoted and the
devices on a shared broker, which speak bare values, ignored it.
seed_house_control.py is the rig: a flow that drives the washing machine plug,
a dimmer and a colour fixture over MQTT, carries the same two as DMX on an
Art-Net node with transmit still off, and a dashboard to drive it by hand.
A python node's settings are constants of its own function, so they are passed
the way its ports are: by name. The controller binds them to the compiled
function, the `params` field is gone from the worker and remote protocols, and
a setting sharing a port's name is reported as a node error rather than
shadowing it. The panel's scaffold follows suit and keeps the header in step
with both ports and settings.
The demo's `pace` moves from a flow input to a setting of the training node,
which is what it always was.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUb8YpL2s3gmN9WTACTt4q
The brain graph counts node_executed events client-side and the websocket is
torn down on every shell change, so anything a flow published during the
navigation gap was lost. The bus now keeps a session tally per qualified node
and the snapshot hands it back, letting a reconnecting client catch up.
Both the route and the tunnel connector build that snapshot from one helper
so the portal cannot drift from the direct connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HTsT1isxUjw5gtkJk8WhuA
Plumbing only: the widget-type literal and its dtype table on both sides,
the regenerated client, a curated lucide map and four stubs the renderers
are wired to. Also a latching switch and a segmented dropdown, both a
`style` on the control that already publishes and reads back, plus the
option editor a dropdown never had.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HTsT1isxUjw5gtkJk8WhuA
make seed-demo builds demo_training — prepare on the engine, a GPU-bound
train, evaluate back here — and a panel that draws the loss curve while the
training is still going. It is the session's whole argument in one flow: batch
runs with parameters and a result, a generator yielding on a declared port
rather than logging, fluksio.emit from inside a callback, artifacts carrying
the dataset and the weights between machines, and a sweep whose configs are
isolated from each other. The train node prefers its label rather than
requiring it, so it runs before a GPU box exists and says which machine and
which numeric backend it actually used.
Building it turned up two real bugs. A run waited for a worker its flow only
*preferred*, because required_labels ignored device_policy — so the example
hung on a label it did not need. And a run's seed never reached the flow, so
sweeping over seeds ran the same experiment N times; it now fills an input of
that name when the flow declares one, which is what the field looked like it
did all along.
Pressing Run on a batch flow now submits a run rather than taking the old
non-durable path — that button is the first thing anyone evaluating will press,
and it was quietly doing something else.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
The first cut had node code call fluksio.log_metric, which was a second,
undeclared way for data to leave a node: invisible to validation, absent from
the canvas, and stored where the graph could not see it. That is precisely the
MLflow discrepancy this framework exists to avoid, so it is gone.
A node that produces values over time is a generator. Every yield is a dict
keyed by output port, published the instant it happens — same port, same type
check, same place on the canvas as any other value — and what it returns is
its result. A port doing this declares stream: true, and a run keeps every
number one takes, so experiment tracking is a consequence of the graph rather
than an API beside it: a chart binds to a training curve the way it binds to a
temperature. fluksio.emit writes the same ports imperatively, for where a
yield cannot reach — inside a training framework's callback.
In a live flow an emission also wakes what is downstream, as a subscriber
publishing does; in a run it does not, because a run's graph is scheduled once
and mid-node cascades would leave 'finished' with nothing to mean. The
enqueued item carries no payload: the value is already in state, and one
carrying it would re-apply an old emission after the node returned.
Verified on the stack: 30 loss values arrived live on the flow socket during a
run, attributed to the node that produced them, and the same node run on the
remote worker streamed its curve back across the socket.
Also caches remote compile results per worker, so attaching a GPU box does not
put a network round trip in every rebuild.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
The engine runs where the automations are and the GPU is somewhere else,
usually behind a different network — so the worker connects out and the engine
answers over the socket it was given. Nothing has to expose Redis, and the
same connection works through the tunnel the hosted access will use.
What travels is the protocol the local pool already speaks, so a node cannot
tell which kind of worker it is on. A node declares device: gpu and
device_policy, the label is resolved per call (a worker attaching later needs
no rebuild), and a run whose labels nothing carries waits in the queue saying
what it waits for rather than failing — submit from the couch, the GPU box
picks it up when it is switched on.
Two things had to move with it. Compiling now happens on the machine that will
run the node: a node importing torch is correct on the GPU box and a missing
module on the engine, so checking it here failed nodes that were fine. And the
artifact endpoint accepts a worker's own credential, because storing a
checkpoint is exactly what that credential is for — and only that.
Verified against the real split: the training ran on this host (its checkpoint
names the machine and a numpy the engine does not have), streamed 40 metric
points back mid-run, and the evaluate node read the checkpoint on the engine.
Cancel kills the remote training; pulling the worker fails the run in six
seconds instead of waiting out its ten-minute timeout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
A checkpoint is not a message. DType.ARTIFACT carries a reference — digest,
size, media type, name — so everything on the wire stays JSON and thirty
megabytes never sit in Redis, which answers the vision's open binary-payload
question by narrowing it: inline codecs would only serve payloads too small to
be worth a round trip, and nothing asks for that.
The store is content-addressed rather than per-run, for three reasons that all
pay later: a sweep whose fifty configs share one preprocessed input stores it
once, a reference stays valid however it is passed around because it names
content instead of a location, and the digest is what a stage cache will
compare — so building it in now is what keeps that from being a change to the
message contract.
Node code calls fluksio.save_artifact/load_artifact and cannot tell whether it
is writing the engine's own directory or putting bytes over HTTP, which is
what will let the same flow run on a remote worker unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
A training loop has numbers worth keeping thousands of steps before it has a
result. Node code now imports fluksio and calls log_metric/progress, which
sends a line back without ending the call; the engine writes those to
run_metric in batches from the run's own driver rather than folding them off
the event bus, which drops what it cannot keep up with.
Two things fall out. Each event resets the worker deadline, so a node's
timeout measures silence rather than duration — which is what lets a two-hour
training keep a liveness contract instead of racing it. And the worker pool's
_running is now keyed by (run, node), so cancelling one config of a sweep
kills that training and leaves the rest alone.
Fixes a latent framing bug: read_line returned whatever a read had taken,
which was fine while a worker only ever sent one line per request and
unparseable as soon as it sent several. It now keeps the remainder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
A cascade has no end worth recording; a run does. Parameters go in, the graph
executes until it drains, and the result is kept — which is what an ML
experiment is and what a CI-style job is, so both are one entity.
Each run gets a state backend namespaced to itself, so two runs of one flow
cannot overwrite each other's messages; that is a constructor argument rather
than a change to the pipeline, because every key the engine keeps already goes
through the state backend. Its record is written by the driver thread rather
than folded off the event bus, which drops what it cannot keep up with. Its
own Redis stream wakes an engine up, and from the claim onwards the database
row is the truth: redelivering hours of training because an acknowledgement
was late is not recovery, so a stale lease is what marks a run whose engine
died.
Flows gain mode: batch, which are built and validated but never activated, and
nodes gain a device label for the worker that must run them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AD8SfVhzXBG2nAfFcVh3iD
A series, record or list message declares its shape instead of riding
DType.JSON, so a widget binds a shape rather than some JSON and a wrong
binding is refused before anything runs. A list declares its item type,
which is what keeps list[float] expressible for a pipeline.
On top of that: an agenda over a list, a notification over a record, and
a dashboard alert channel that publishes engine faults as one — so a
panel can show what went wrong without a flow wiring it by hand.
Also: only None means a node published nothing, a falsy value of the
wrong shape is now the named error it always should have been; and the
gauge's readout says its size is viewBox geometry rather than type scale.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dependency loop is flagged on the canvas and was invisible everywhere else:
/observability/summary answered "ok" with an empty problems list while the
published flow could not run at all. It now reports the flows validation
blocks, and the brain graph carries the reason on each neuron the issue names
so the view built to find broken wiring can show it.
Node errors stay counted once, as the nodes that failed to load, and an
advisory like an unauthenticated webhook marks nothing — it is worth saying,
but the flow still runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XC2jX6Hdj7pxGGKzBTrbqB
Both overviews carried the same toolbar twice, left-aligned, with a search
field permanently taking a row of width. One `OverviewToolbar` now serves
them: the search folds into an icon and expands again on click (Escape puts
it away and hands focus back), create is a `+`, and everything sits right of
the page. Each page keeps its own create dialog — the toolbar only renders
the trigger — so the testids the runtime spec and the capture script drive
stayed where they were.
Dashboards get the flow store's draft/publish split. The editor autosaves
`dashboard.draft.json` beside `dashboard.json`; `/view/{name}`, `bindings_for`
and `history_requirements` keep reading the published file, so a wall panel
sees an edit only once someone publishes it. `POST /dashboards/{name}/publish`
and `/discard` mirror the flow routes down to the version precondition and the
409, `GET /dashboards/{name}?draft=true` is what the editor asks for, and the
dock grows the same Publish button — which flushes a queued save first, so an
autosave in flight is not published around. Creating a dashboard still writes
the published file directly: an empty document on a panel is harmless, and it
keeps the store free of a never-published case.
"Publish all" is a checkmark in the toolbar, live only when something actually
has `has_draft`. A summary carries no version and publish needs the one it is
based on, so each document's detail is read immediately before its publish —
honest against a stale list, and no version-less backend path to maintain.
Failures are counted rather than swallowed: three of five fails says so and
names the three.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XC2jX6Hdj7pxGGKzBTrbqB
`AlertManager.send` swallowed every delivery failure, so the alerts screen's
Test button answered 200 whatever happened — the one thing it exists for. It
takes `raise_on_error` now, which only the test route passes; the per-channel
loop keeps the swallow, because one dead channel must not stop the others
hearing about the same fault. A refused delivery answers 502 with whatever the
sender said.
Renaming a flow left its values under the old name for good: the delete path
already swept them, the rename path never did. It calls the same `forget_flow`,
which covers the messages and the `__ts__`/`__version__`/`__history__`
bookkeeping keyed by message name. Cleanup, not migration — they repopulate
under the new name on the next run.
Triggering a node by hand ran `Node.__call__` with nothing catching it, so a
node that raised produced a 500 and a stack trace in the server log, and
nothing at all on the canvas. `Pipeline.publish_error` is the reporting half of
`_execute_node` lifted out; both paths go through it, so a manual failure now
reads the same on the canvas and in the metrics as a queued one. The route
answers 400 with the node's error.
`MemoryWorkQueue.stats()` counts claimed-but-unacknowledged work rather than
reporting zero, so the health tile means something without Redis. The metrics
collector's held tracebacks are capped at `DETAIL_CAP` and swept on the same
`RUN_STALE_S` cutoff the open runs use, instead of one untruncated traceback
per node kept for the life of the process — a traceback still survives the
flush between the log and the failure it belongs to.
`GET /observability/events` takes `since`/`until`, the window `/runs` already
took, so a failures list can cover the span the charts beside it are drawn from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XC2jX6Hdj7pxGGKzBTrbqB
The worker script is handed to the interpreter by path, so app/flow was
sys.path[0] for every node: `import queue` got the engine's. It now drops
its own directory before anything else imports, and runs with the
deployment's credentials scrubbed out of its environment.
Also: reload builds off the event loop, the pool wakes what is blocked on
it when it stops, a refused metrics flush is kept for the next one rather
than dropped, and the cascade events are paired through failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MeiWk3Yq12n2pTvnQWYvt
stroblmeandClaude Opus 5