`--format jsonl` existed, but `_cell` ran json.dumps at row-build time,
before a format was chosen — so a nested value was a string by then and
jsonl only re-escaped it, leaving a consumer against csv's 128KB field
limit either way. Stringifying moved to the csv writer, so csv is
byte-identical and jsonl carries json. The runs TUI followed, or a record
would draw as a Python repr.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KYM38KSb4V4v2T71eifnZv
**SQLite is the database, and now says so.** `metric_minute` and every run
table are written with `sqlalchemy.dialects.sqlite.insert(...)
.on_conflict_do_update` and with `max(a, b)`, neither of which another
dialect has — so pointing `DATABASE_URL` at Postgres migrated cleanly,
served, logged in, and then lost every observability flush into the
collector's hold buffer and failed every run. It refuses at startup
instead. (The Postgres in the compose stack is Umami's; the engine's own
database has been a file beside the flows since 2026-08-21.)
**Every integer query parameter is bounded.** The caps were written as
`min(limit, 500)`, which a negative walks straight through — `?limit=-1`
compiles to `LIMIT -1` and SQLite returns the whole table. Ten signatures,
now `Query(ge=…, le=…)`. `hours=0` still means an hour, which
`_window_hours` was already deliberate about.
**Exports are capped at 10 000 runs** and say so with `X-Truncated`. The
filters bounded a sensible request and nothing bounded an unfiltered one,
which read every row into memory before a byte was streamed. `_series`
resolves cached curves in two queries rather than a `Run` lookup and a
`RunMetric` query per restored node — a comparison of twenty runs was
calling that twenty times over.
**`PUT /artifacts` has a size limit** (`MAX_ARTIFACT_BYTES`, 2 GiB, 0 to
disable), checked against `Content-Length` and again against the stream for
a chunked body, and its writes moved off the event loop.
**`/observability/timeseries` takes `since`/`until`**, the same window
`/runs` and `/events` take, capped at 2000 points — `hours=720&bucket_s=60`
was 43 200 of them in one array. It is also what a dragged chart needs to
re-fetch at its own resolution rather than magnifying buckets it has.
**Composite indexes** for the three list screens: `run(flow, created_at)`
and `(status, created_at)`, `flow_run(flow, started_at)`,
`engine_event(type, ts)`. Every index was single-column, so SQLite picked
one and sorted the rest by hand. Verified against a copy of a live database
(250k `flow_run` rows): the planner takes all four.
**Redis clients have socket timeouts.** A Redis that stops answering
without closing the connection hung the caller until the kernel gave up —
including `/utils/health/`, whose job is to notice.
**The panels file is written under one lock.** `save_panels` and
`unpair_panel` are both read-modify-write, and a save that read before an
unpair wrote put the old nonce back — silently un-revoking a screen that
had just been unpaired. The nonce carry-forward was written to make that
impossible; the gap between its read and its write is where it happened.
**Startup releases what it acquired.** Everything past `event_bus.bind`
registers how to close itself and the `finally` walks that list backwards;
a failure part-way through used to reach none of the shutdown steps and
leave the worker pool's subprocesses and every background task behind —
under `--reload`, once per bad edit. `modules.reconcile` moved into the
background: `uv` gets five minutes twice over, the healthcheck allows
eighty seconds, and the autoheal restarted the container before it could
finish installing.
`delete_run` takes SQLite's write lock up front (`core.db.writing`) rather
than upgrading a deferred transaction and losing to whichever flush
committed in between. `modules.sync` is serialised — two applies mutated
one venv at once. The proxied-call and stream dicts are bounded, and a
reused id cancels its predecessor instead of dropping the reference.
Security, found in passing and small enough to fix here:
- **`/secrets/` required only a signed-in user.** The names alone say what
this installation talks to, and `PUT /{name}` takes any name, so any
account could overwrite the credential a flow authenticates with.
Superuser now — which `/search` already assumed and said so.
- **`POST /login/access-token` had no rate limit.** Argon2 is deliberately
expensive and the route is unauthenticated and runs in the shared
threadpool. Ten *failed* attempts per address per five minutes; a
successful sign-in spends nothing.
- **a password reset link worked repeatedly for 48 hours.** The token now
carries a digest of the password hash it was minted against, so it stops
verifying once it has set one. No table of spent tokens needed.
- **enrolment accepted `http://`**, sending the claim code and then this
installation's credential in clear. https, or a local address.
- the rate limiter read `request.client.host`, which behind Traefik is the
proxy — so every per-address limit was one global bucket and one caller
could lock out everyone. It reads the forwarded address, and its
bucket table is capped rather than growing one key per address forever.
- SMTP has a timeout and sends after the response, so an unreachable mail
host cannot pin a threadpool worker, and a reply's timing no longer says
whether the address exists.
Test suite: engine-written rows are cleared between modules. A `FlowRun`
left `running` by one module turned up in another's query. Per-test
rollback is not available here — the module-scoped `client` runs the real
lifespan and its collector and run service write through sessions of their
own — so this bounds it where the writes come from. Three consecutive
green runs, orders randomised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6hPWS6YEbT1P8LxhhFb2T
`save_artifact` had no download counterpart: the run detail listed a run's
files and nothing in `fluksio --help` fetched one. `fluksio artifacts RUN`
lists them, `fluksio artifacts RUN NAME` writes one — under the name the
node saved it as, since the message name is chosen for the graph. The run
detail now carries that filename, which it held in the table and did not
report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc
`export runs` dropped a `param.*` column whose value was constant across the
exported runs, so a downstream filter broke depending on which runs the
selection happened to hold. Every input the selection recorded is a column
now; `--params` still narrows it to a sweep's axis. The metrics default is
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Hra4ndWMCLU5F3KjUuVAc
Runs: a run can now be deleted (DELETE /runs/{id}, cancelling a live one
first), exported as csv from the screen's own filters, and its flow label
opens the flow. Its "Parameters" panel became "Inputs" and lists every
input the flow declares, marking the ones that took the flow's own value
rather than the run's — the comparison table resolves the same defaults
instead of printing "unset".
Home reads brain, dashboards, health, flows: the mosaic is one full-width
scrolling strip, and the flows list and the flow-activity rollups merged
into a single left-joined table so a flow's state and its numbers sit on
one row.
Charts take a drag to narrow the x window and a double click or tap to
come back out. UplotChart holds the scale and passes resetScales:false
while a window is held, which is what the old comment said made this
impossible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019LrWVRguqbk33YzfEeUx5W
Three things the first export pass got wrong for a real study.
**Dotted paths.** A node returns a record, not a scalar — the numbers arrive
inside `final_metrics` — so `--metrics final_metrics.train_loss` yielded an
empty column and `--metrics final_metrics` yielded the whole record in one
cell. Both sides of the wide table now take dotted paths, and the defaults
reach the same depth: every number a result carries is a column named by its
path, and inputs are compared leaf by leaf, so two configurations differing in
one field give that field as the axis rather than two blobs that are merely
not equal. Lists stay whole — a curve belongs in the long table.
**`--list`.** Metric names are flow-qualified, so `--name train_loss` matched
nothing and said only that. `fluksio export metrics --list` prints the names
the selection carries, and an empty export made with `--name` points at it.
**A version to compare.** The CLI ships ahead of the engine and a stale one
answered a flat 404 with nothing anywhere in the API to tell how old it was.
The engine reports `version` on `/observability/summary`, `fluksio status`
prints it, and a 404 from export now names both versions — or says "older"
when the field itself predates the engine. Bumped to 0.1.5, which is what
makes the number worth reading.
Also formats `flow/metrics.py`, which had been committed unformatted and was
the last `ruff format --check` failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9Hdrmf2cwNABCnE5x9UJa
`fluksio export metrics` is the long table — a row per run, metric and step —
and `fluksio export runs` the wide one, a row per run with the inputs that
*vary* across the selection as columns beside its final numbers, status,
duration and the commit and digest of the code it ran. Both carry the run id
on every row, which is the join back to the run page and what makes an
exported file auditable. `Client.export_metrics`/`export_runs` answer the same
rows to a notebook.
The engine streams csv or jsonl from two routes declared above `/{run_id}`;
parquet is a client-side conversion behind the new `fluksio[parquet]` extra,
so nobody pays for pyarrow who does not want dtypes kept. The long export
reads each run through `_series`, so a cached node's curve comes with it, and
`--stride` thins each series rather than the concatenation of all of them.
Two things they needed on the way: `GET /runs` takes `?since=` and `?before=`,
so a long history pages by the last row's own timestamp instead of an offset
that shifts under it; and a read that reaches no engine now says so in half a
second rather than seven, because `runs`, `flavors`, `export` and an unwatched
`status` pass `retries=0`. Everything that submits keeps them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9Hdrmf2cwNABCnE5x9UJa
Three faults with one root: the stored body of a code-defined node is an
import shim, and nothing that mattered was ever read from the code itself.
- The run stamp could not identify what ran. The shim imports whatever is on
disk when the worker starts, and an uncommitted tree stamps <commit>-dirty
for every run it ever produces. Run.code_digest hashes the repository's .py
files, memoized on their stat state, and it is read again when the run is
actually claimed -- so a sweep queued for hours records the code each of its
runs executed, not the code that was there when it was submitted.
- The stage cache adopted code that was too new. The fingerprint hashed the
shim, which is invariant under any edit to the imported function or anything
it calls into, so a re-run was served from cache and answered without the
outputs the edit added. It now carries the repo digest and the node's
declared ports. Every fingerprint changes once, which invalidates the
existing cache; a canvas flow has no repository and keys as before.
- An interrupted sync looked like a hand-edited canvas. The engine answers a
new-node template for a node with no stored body, and the template carries
no marker, so the drift check read "somebody edited this" and demanded
--force -- for the one state that re-running the sync is the fix for.
NodeSource.missing states the fact, and sync skips those and reuses the
bodies it read instead of asking for each one twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A driver script died of one slow answer: httpx.ReadTimeout out of
RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which
cost a sweep 78 of its 84 runs.
- Split the timeout (5 s connect, 120 s read): a wrong URL fails at once,
and a busy engine gets longer than the slowest thing it does on purpose
(a 60 s compile, a 15 s rebuild wait).
- Retry idempotent calls three times on a transport error or 502/503/504.
503 is the engine's own "ask again" — it is what RebuildBusy answers.
- Submit carries a key the engine stores with the run, so a retry after a
timeout returns that run instead of starting a second. A sweep keys every
entry, so a half-created one recreates only what is missing.
- wait() and --follow tolerate five failed polls in a row; a 404 still stops
at once, because that is an answer rather than a gap.
- CLI says "engine not answering" and names the run still on the engine,
instead of printing a traceback.
- runs: clamp the params column to 80 characters; events() takes the
flow/since/until the endpoint already had; RunHandle.failures answers
"what killed this run" from the run's own node rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`POST /runs/flows/{name}` hardcoded `cause: "api"`, so every row in the
history claimed the same origin. The body now carries an optional `cause`,
closed to the values the column knows — the dashboard sends nothing and stays
"api", `fluksio run` says "cli", and the SDK client says "sdk".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Gf7WaExcJ9bs3kfJXB3nK
A cache hit still replays no emissions — those values were the story of an
execution that is not happening — but the run they were recorded in is now
written on the row (`run_node.cached_from`), and the metrics endpoints read the
series back from there. So a reused run answers `run.metrics("train.loss")`
with the same points the run that trained did, rather than looking like a run
that produced no numbers at all. Pointed at rather than copied: a sweep of 500
reusing one frozen node would otherwise duplicate its curve 500 times.
That needed the cross-flow restore fixed first. The cache key has no flow in
it while the stored outputs are named for the flow that produced them, so
`quick.prepare` getting a hit from `train` wrote `train.dataset` into `quick`'s
state and the next node was called without its argument. One rule now covers
both halves: `requalify` reads a name owned by one flow as the same name in
another, applied to the restored outputs, to the node id behind the pointer,
and to the series names on the way out. Reuse across flows is kept.
Also: `@run:<id>.<output>` and a bare `sha256:` digest resolve on every input,
not only artifacts. Chaining a run's json config into the next one from a shell
meant pasting the whole object inline, and the CLI could not even send the
spelling — `_coerce` died in `json.loads` before the engine saw it. Both
spellings are reserved on every input now, `str` included, and `_from_run`
returns whatever the run's result holds rather than only a reference.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dp9L6gakMVro1K2C5zdtBE
A code node in a batch run is now fingerprinted by its source, its raw
settings and the values it reads — an artifact input counting as its digest,
which is what the content addressing was always for. A run that finds the key
restores what the earlier one returned and skips the node, recorded as
`cached`. The run history is the cache: `run_node.outputs` beside the
`cache_key` the schema already had, no second store. On for code nodes, never
for the built-in and connector types that have side effects; off per node with
`@node(cache=False)` and per run with `--no-cache`.
Emissions are not replayed on a hit, so a cached training node returns its
result without redrawing its curve. Recorded in NOTEPAD.md with the two other
deliberate limits.
`fluksio run --local` boots the real app in the command's own process and
drives it through its ASGI interface behind the ordinary client, so a run no
longer needs a `serve` terminal beside it — same data directory, same history,
and the cache carries between the two. It always waits, because the engine it
starts lives exactly as long as the command.
Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists,
`run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited
run rather than abandoning it, coloured statuses on a terminal, and `name`
made optional on the metrics endpoint so a follower can ask for every series.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A data scientist keeps their code where it is and decorates it: `@node`
declares a function's ports beside the function, `Flow(name, nodes=[...])`
says which of them make a flow, and `use(fn, wire=..., **settings)` rebinds
one for a single flow. `fluksio sync` uploads the document plus a generated
import shim per node, so the store still holds a complete, runnable,
git-versioned definition while the code it imports stays theirs.
`fluksio login|run|runs` and `flow.submit().wait()` are the client half, over
the run endpoints that already existed. Runs record the user repository's
commit beside the store's, so "what code produced this number" is answerable
on the side that now holds the code.
- `fluksio/sdk/`: ports, decorators, the flow builder and its checks, the shim
generator, an HTTP client and sync. Standard library only at import, so
`from fluksio import node` in a training script pulls in no engine.
- `FlowDef.origin` marks a flow code-defined; `Run.origin_commit` carries the
repository's commit; `POST /modules/refresh` retires the workers without an
install, which every sync calls — a worker holds the imported package in
memory, so an edit to it is invisible until the process goes.
- The canvas shows a generated body read-only and names the repository to edit
instead; a body edited there stops the next sync rather than being discarded.
- The worker's reporter carries inert `Port`, `node`, `use` and `Flow`, since
the shim imports a module whose first line declares them.
- `examples/myresearch` is the worked example, `make sync-example` uploads it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ue1tkFWB1bcGy3aWhCKpU
A wheel whose top-level module is `app` collides with anything else in a
user's venv, so the package that is about to be published takes the name
it is published under. Only the Python package moves; the repo, the
Docker WORKDIR and the compose project keep theirs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
stroblmeandClaude Opus 5