Migration ccb21eacf5d5 deletes rows whose run is gone, then adds foreign keys
with ON DELETE CASCADE from run_node, run_metric and run_artifact to run.id.
Deleting runs is now the one statement. A node or metric written after its
run was deleted is dropped with an info line. Migrations run with foreign keys
off, since a batch rebuild of run would otherwise cascade into every child row.
Run.declared (migration 2e7ca7d10044) snapshots the declarations on submit
and import. The run detail's Inputs panel reads them, and a run from before
falls back to the flow's current declarations.
A flush ran its cascade under no id, so its executions and failures belonged
to no run. It now records a run with source "flush", and the holding run's id
travels on the queue item into a new `flow_run.parent` column, also on the
`/observability/runs` rows.
`ConnectorNode.store` keeps short strings per connector package in the
engine's database (`connector_state`, migration f8c3e5a91d27), so the nodes of
one package share them and `fluksio serve` needs nothing new: get, set and
delete, at most 64 kB a value and 256 keys a package. A superuser lists the
keys (never the values) and clears them through `/connectors/{node_type}/state`,
or under Stored state in a connector node's panel. Documented in the connector
contract.
`Run.passed` (migration d5a3c8e1f246) keeps the names the caller chose, so
the run detail and the comparison table can mark a value as the flow's own
again; rows from before it fall back to the names their params lack. Retries
hand on their parent's list, imports record their entry's.
Also: a cancel of a queued run no longer leaves its id in `_cancelled`, a
linked comparison no longer crashes before the list loads, and the pre-commit
failures are fixed (stale client, `text-[10px]`, worker formatting).
A history recorded before the move (MLflow experiments) had nowhere to
go, since runs were only made by running a flow. Each entry is stored as
it ended: parameters completed and checked like a submit, cause
"import", empty commit and code digest, metrics and artifacts written
as given, nothing queued or announced. Run.external_id (new migration)
makes a second import a no-op and is an export column so a row joins
back. The request is all or nothing and capped at 200 entries;
Client.import_runs pages a longer history.
The site promises simulated inputs and mocked sensor values, and nothing in
the app was that. A run already is: the values are the caller's, the state is
the run's own namespace, and nothing it computes reaches the live flow. What
was missing was a screen to do it from, and the draft flag being honoured.
`/runs/new` is a flow, a field per declared input, a seed and Run; `/runs`
stays the log. A comma-separated list in a number field expands into the grid
`fluksio sweep --param` builds and goes to the sweep route, so launching one
no longer needs a terminal. Only numbers split: a comma in a string is
content, and one in JSON is syntax.
`RunCreate.draft` was validated at submit and dropped before the run
executed, so "try the working copy" ran the published one. `Run.draft` is a
column now, the driver reads the same copy the submit checked, and a retry
carries it. `FlowSummary.mode` came with it so the rail can say which flows
are batch before one is picked.
Also here: a Retry button on a finished run, which the route has always had
and the UI never did, and parameter cells truncated to their column with the
full value on hover.
Follows the portal: the noun is "instance" everywhere the app says it —
UI strings, CLI output, error details, docs and comments. The wire keys
(`instance_id`, `instance_token`) and the hub route this calls move with it.
An existing cloud.json is adopted rather than refused: without the key
alias the dataclass fails to parse, which the caller swallows and reads as
"never enrolled" instead of "reconnect".
`instance_key` on a node type becomes `target_key`. It means the outside
thing a node points at, which is a different sense of the word, and keeping
both would put two meanings of "instance" in one codebase.
**SQLite is the database, and now says so.** `metric_minute` and every run
table are written with `sqlalchemy.dialects.sqlite.insert(...)
.on_conflict_do_update` and with `max(a, b)`, neither of which another
dialect has — so pointing `DATABASE_URL` at Postgres migrated cleanly,
served, logged in, and then lost every observability flush into the
collector's hold buffer and failed every run. It refuses at startup
instead. (The Postgres in the compose stack is Umami's; the engine's own
database has been a file beside the flows since 2026-08-21.)
**Every integer query parameter is bounded.** The caps were written as
`min(limit, 500)`, which a negative walks straight through — `?limit=-1`
compiles to `LIMIT -1` and SQLite returns the whole table. Ten signatures,
now `Query(ge=…, le=…)`. `hours=0` still means an hour, which
`_window_hours` was already deliberate about.
**Exports are capped at 10 000 runs** and say so with `X-Truncated`. The
filters bounded a sensible request and nothing bounded an unfiltered one,
which read every row into memory before a byte was streamed. `_series`
resolves cached curves in two queries rather than a `Run` lookup and a
`RunMetric` query per restored node — a comparison of twenty runs was
calling that twenty times over.
**`PUT /artifacts` has a size limit** (`MAX_ARTIFACT_BYTES`, 2 GiB, 0 to
disable), checked against `Content-Length` and again against the stream for
a chunked body, and its writes moved off the event loop.
**`/observability/timeseries` takes `since`/`until`**, the same window
`/runs` and `/events` take, capped at 2000 points — `hours=720&bucket_s=60`
was 43 200 of them in one array. It is also what a dragged chart needs to
re-fetch at its own resolution rather than magnifying buckets it has.
**Composite indexes** for the three list screens: `run(flow, created_at)`
and `(status, created_at)`, `flow_run(flow, started_at)`,
`engine_event(type, ts)`. Every index was single-column, so SQLite picked
one and sorted the rest by hand. Verified against a copy of a live database
(250k `flow_run` rows): the planner takes all four.
**Redis clients have socket timeouts.** A Redis that stops answering
without closing the connection hung the caller until the kernel gave up —
including `/utils/health/`, whose job is to notice.
**The panels file is written under one lock.** `save_panels` and
`unpair_panel` are both read-modify-write, and a save that read before an
unpair wrote put the old nonce back — silently un-revoking a screen that
had just been unpaired. The nonce carry-forward was written to make that
impossible; the gap between its read and its write is where it happened.
**Startup releases what it acquired.** Everything past `event_bus.bind`
registers how to close itself and the `finally` walks that list backwards;
a failure part-way through used to reach none of the shutdown steps and
leave the worker pool's subprocesses and every background task behind —
under `--reload`, once per bad edit. `modules.reconcile` moved into the
background: `uv` gets five minutes twice over, the healthcheck allows
eighty seconds, and the autoheal restarted the container before it could
finish installing.
`delete_run` takes SQLite's write lock up front (`core.db.writing`) rather
than upgrading a deferred transaction and losing to whichever flush
committed in between. `modules.sync` is serialised — two applies mutated
one venv at once. The proxied-call and stream dicts are bounded, and a
reused id cancels its predecessor instead of dropping the reference.
Security, found in passing and small enough to fix here:
- **`/secrets/` required only a signed-in user.** The names alone say what
this installation talks to, and `PUT /{name}` takes any name, so any
account could overwrite the credential a flow authenticates with.
Superuser now — which `/search` already assumed and said so.
- **`POST /login/access-token` had no rate limit.** Argon2 is deliberately
expensive and the route is unauthenticated and runs in the shared
threadpool. Ten *failed* attempts per address per five minutes; a
successful sign-in spends nothing.
- **a password reset link worked repeatedly for 48 hours.** The token now
carries a digest of the password hash it was minted against, so it stops
verifying once it has set one. No table of spent tokens needed.
- **enrolment accepted `http://`**, sending the claim code and then this
installation's credential in clear. https, or a local address.
- the rate limiter read `request.client.host`, which behind Traefik is the
proxy — so every per-address limit was one global bucket and one caller
could lock out everyone. It reads the forwarded address, and its
bucket table is capped rather than growing one key per address forever.
- SMTP has a timeout and sends after the response, so an unreachable mail
host cannot pin a threadpool worker, and a reply's timing no longer says
whether the address exists.
Test suite: engine-written rows are cleared between modules. A `FlowRun`
left `running` by one module turned up in another's query. Per-test
rollback is not available here — the module-scoped `client` runs the real
lifespan and its collector and run service write through sessions of their
own — so this bounds it where the writes come from. Three consecutive
green runs, orders randomised.
A batch run opens no cascade, and FlowRun was written only from
cascade_started — so `fluksio run` showed on /runs and in `fluksio status` and
was simply absent from Home. The collector now folds the run_started and
run_finished events RunService already published. Such a record is exempt from
the staleness sweep in both places: a training step of an hour is a normal one,
and only run_finished ends it.
Concurrent runs sat at 4 whatever FLOW_MAX_CASCADES said: that setting bounds
cascades, and the run drivers read a hardcoded MAX_PARALLEL nobody could reach.
FLOW_MAX_RUNS is the knob they read now, --max-runs/--max-cascades/--max-workers
are the same three as flags on serve, and the engine says which numbers it
started with — which is the only way to tell that a settings file was read.
Events keep the run they happened in. The payload always carried it and the
persist path dropped it, so reading one run's failures meant filtering the
engine-wide list; a batch run's id reaches those events now too, since a run
has no journaled item to name itself by.
Also: a provisioner's 0 means "no deadline" rather than "cancel on the next
reconcile", and a command that reaches no engine says how to start one.
Raw cpus and gpus are a property of the machines an installation has, so a node
written against a cluster quietly stops meaning anything when the cluster is
replaced. A node says "gpu-small" instead, and what that is stored here —
editable, and read again every time the node is built, so changing the flavor
changes what the next run gets.
Memory joins the schema properly (`ram`, in MB, accepting "2G"), along with
`duration_s` for how long a node is expected to take. That one is recorded and
shown and nothing else yet: a statement for whoever is planning around the node,
not a limit — the limit is still `timeout`.
A flavor and a number for the same thing is refused, compared by value so an
editor writing the whole object back with its defaults still round-trips. A name
nothing stores is refused at the save, which covers the canvas and `fluksio
sync` at once, and deleting one a node still asks for says which node.
Four sizes are seeded on an installation that has none, and never re-seeded:
re-adding one somebody deliberately removed is an argument nobody wins.
The engine answered "where does this node run" twice, in two ways that could
not see each other: a device sent it to a worker carrying that label, and
resources were counted against the engine's own cores. Declaring both meant the
second answer won and nothing was counted at all — which the data-science
getting-started page and the worked example both do.
One question now, in flow/placement.py: of every machine attached, which could
grant what this node asked for, and which of those has it free. The books move
onto each machine — one accountant per worker, built from the inventory it
reported — and the waiting moves above them, where one condition variable can
be woken by a release anywhere or by a worker attaching. Locks go one way:
placer, then a machine's books, never back.
So a node asking for a card now finds the box that has one, rather than being
clamped down to none and run here. When nothing can grant the ask at all it is
still cut down and run — a flow written on a cluster has to work on a laptop —
but the ceiling is one real machine now, since taking the largest of each
dimension separately can describe a machine nobody has.
Two things fixed on the way. A device on a connector node held every batch run
of its flow forever, waiting for a worker that could never run an entry point.
And `prefer` falling back to the engine skipped the books, so the fallback held
nothing.
The bench flow's node has taken a `params` argument that with_settings has not
forwarded for some time, so the benchmark could not run at all: 62 ms median
submit-to-result with this, against the 61 ms on record.
A run_artifact row is keyed by the message the bytes left on, and that was
also the only name it could answer with — so an `@run:` reference resolved
through the row was the same bytes under a name its producer never chose.
The row now records the file name beside the message name; rows written
before the column answer as they always did.
The fallback also checks the bytes are still in the store, which the bare
digest spelling beside it has always done. A missing blob now fails at
submit rather than in the middle of the run that wanted it.
Three faults with one root: the stored body of a code-defined node is an
import shim, and nothing that mattered was ever read from the code itself.
- The run stamp could not identify what ran. The shim imports whatever is on
disk when the worker starts, and an uncommitted tree stamps <commit>-dirty
for every run it ever produces. Run.code_digest hashes the repository's .py
files, memoized on their stat state, and it is read again when the run is
actually claimed -- so a sweep queued for hours records the code each of its
runs executed, not the code that was there when it was submitted.
- The stage cache adopted code that was too new. The fingerprint hashed the
shim, which is invariant under any edit to the imported function or anything
it calls into, so a re-run was served from cache and answered without the
outputs the edit added. It now carries the repo digest and the node's
declared ports. Every fingerprint changes once, which invalidates the
existing cache; a canvas flow has no repository and keys as before.
- An interrupted sync looked like a hand-edited canvas. The engine answers a
new-node template for a node with no stored body, and the template carries
no marker, so the drift check read "somebody edited this" and demanded
--force -- for the one state that re-running the sync is the fix for.
NodeSource.missing states the fact, and sync skips those and reuses the
bodies it read instead of asking for each one twice.
A driver script died of one slow answer: httpx.ReadTimeout out of
RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which
cost a sweep 78 of its 84 runs.
- Split the timeout (5 s connect, 120 s read): a wrong URL fails at once,
and a busy engine gets longer than the slowest thing it does on purpose
(a 60 s compile, a 15 s rebuild wait).
- Retry idempotent calls three times on a transport error or 502/503/504.
503 is the engine's own "ask again" — it is what RebuildBusy answers.
- Submit carries a key the engine stores with the run, so a retry after a
timeout returns that run instead of starting a second. A sweep keys every
entry, so a half-created one recreates only what is missing.
- wait() and --follow tolerate five failed polls in a row; a 404 still stops
at once, because that is an answer rather than a gap.
- CLI says "engine not answering" and names the run still on the engine,
instead of printing a traceback.
- runs: clamp the params column to 80 characters; events() takes the
flow/since/until the endpoint already had; RunHandle.failures answers
"what killed this run" from the run's own node rows.
`POST /runs/flows/{name}` hardcoded `cause: "api"`, so every row in the
history claimed the same origin. The body now carries an optional `cause`,
closed to the values the column knows — the dashboard sends nothing and stays
"api", `fluksio run` says "cli", and the SDK client says "sdk".
A cache hit still replays no emissions — those values were the story of an
execution that is not happening — but the run they were recorded in is now
written on the row (`run_node.cached_from`), and the metrics endpoints read the
series back from there. So a reused run answers `run.metrics("train.loss")`
with the same points the run that trained did, rather than looking like a run
that produced no numbers at all. Pointed at rather than copied: a sweep of 500
reusing one frozen node would otherwise duplicate its curve 500 times.
That needed the cross-flow restore fixed first. The cache key has no flow in
it while the stored outputs are named for the flow that produced them, so
`quick.prepare` getting a hit from `train` wrote `train.dataset` into `quick`'s
state and the next node was called without its argument. One rule now covers
both halves: `requalify` reads a name owned by one flow as the same name in
another, applied to the restored outputs, to the node id behind the pointer,
and to the series names on the way out. Reuse across flows is kept.
Also: `@run:<id>.<output>` and a bare `sha256:` digest resolve on every input,
not only artifacts. Chaining a run's json config into the next one from a shell
meant pasting the whole object inline, and the CLI could not even send the
spelling — `_coerce` died in `json.loads` before the engine saw it. Both
spellings are reserved on every input now, `str` included, and `_from_run`
returns whatever the run's result holds rather than only a reference.
A code node in a batch run is now fingerprinted by its source, its raw
settings and the values it reads — an artifact input counting as its digest,
which is what the content addressing was always for. A run that finds the key
restores what the earlier one returned and skips the node, recorded as
`cached`. The run history is the cache: `run_node.outputs` beside the
`cache_key` the schema already had, no second store. On for code nodes, never
for the built-in and connector types that have side effects; off per node with
`@node(cache=False)` and per run with `--no-cache`.
Emissions are not replayed on a hit, so a cached training node returns its
result without redrawing its curve. Recorded in NOTEPAD.md with the two other
deliberate limits.
`fluksio run --local` boots the real app in the command's own process and
drives it through its ASGI interface behind the ordinary client, so a run no
longer needs a `serve` terminal beside it — same data directory, same history,
and the cache carries between the two. It always waits, because the engine it
starts lives exactly as long as the command.
Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists,
`run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited
run rather than abandoning it, coloured statuses on a terminal, and `name`
made optional on the metrics endpoint so a follower can ask for every series.
The gates have never gone green on the new runners. Three separate reasons:
- backend/Dockerfile shipped Python 3.10 while the code imports typing.Self
and datetime.UTC, so the container exited on import and the suite could not
even load its conftest. The image moves to 3.13 and the packages declare
>=3.12, which is the floor the tests actually pass on; ruff's target follows
and rewrites timezone.utc and asyncio.TimeoutError accordingly. Relocking
drops the 3.10 branch, which bumps FastAPI and so regenerates the SDK.
- frontend/README.md had no trailing newline and two dashboard widgets used
arbitrary text-[…] sizes. Both are em-relative on purpose, so they move to
the inline style the neighbouring ramp already uses.
- Every commit left its own run queued: without a concurrency group a runner
that was offline for a while works through a backlog nobody reads. A stack
that fails to come up now prints its logs before the teardown removes it.
A data scientist keeps their code where it is and decorates it: `@node`
declares a function's ports beside the function, `Flow(name, nodes=[...])`
says which of them make a flow, and `use(fn, wire=..., **settings)` rebinds
one for a single flow. `fluksio sync` uploads the document plus a generated
import shim per node, so the store still holds a complete, runnable,
git-versioned definition while the code it imports stays theirs.
`fluksio login|run|runs` and `flow.submit().wait()` are the client half, over
the run endpoints that already existed. Runs record the user repository's
commit beside the store's, so "what code produced this number" is answerable
on the side that now holds the code.
- `fluksio/sdk/`: ports, decorators, the flow builder and its checks, the shim
generator, an HTTP client and sync. Standard library only at import, so
`from fluksio import node` in a training script pulls in no engine.
- `FlowDef.origin` marks a flow code-defined; `Run.origin_commit` carries the
repository's commit; `POST /modules/refresh` retires the workers without an
install, which every sync calls — a worker holds the imported package in
memory, so an edit to it is invisible until the process goes.
- The canvas shows a generated body read-only and names the repository to edit
instead; a body edited there stops the next sync rather than being discarded.
- The worker's reporter carries inert `Port`, `node`, `use` and `Flow`, since
the shim imports a module whose first line declares them.
- `examples/myresearch` is the worked example, `make sync-example` uploads it.
One process owns this database — the image has run a single uvicorn
worker for that reason since the four-engines bug — so a file beside the
flows is the honest shape for it, and it is what lets `fluksio serve`
need no infrastructure at all. Live values, node execution and the work
queue never came here anyway; what does is a rollup a minute at a time,
a row per cascade and the run history, and WAL keeps the readers going
while that one writer works.
DATA_DIR is now the one setting that moves everything an installation
keeps; the rest derive from it and the images still spell theirs out.
The schema is prepared in-process at startup, so the prestart service is
gone, and the ten Postgres-only revisions collapse into one portable
baseline.
Three things only worked because psycopg was casting for us: a token's
subject arriving as a string where the column is a UUID, `greatest`, and
`date_bin`. The timestamps needed a column type of their own — SQLite
stores no offset, and a naive datetime read back either raises against an
aware `now` or serialises as local time.
Postgres stays in the stack only for Umami, behind the analytics profile.
A wheel whose top-level module is `app` collides with anything else in a
user's venv, so the package that is about to be published takes the name
it is published under. Only the Python package moves; the repo, the
Docker WORKDIR and the compose project keep theirs.
stroblme