Commit Graph
9 Commits
Author SHA1 Message Date
stroblmeandClaude Opus 5 1f7c6646f1 Survive a busy engine: retry, idempotent submit, resilient waiting
A driver script died of one slow answer: httpx.ReadTimeout out of
RunHandle.refresh() with a 30 s read timeout and no retry anywhere, which
cost a sweep 78 of its 84 runs.

- Split the timeout (5 s connect, 120 s read): a wrong URL fails at once,
  and a busy engine gets longer than the slowest thing it does on purpose
  (a 60 s compile, a 15 s rebuild wait).
- Retry idempotent calls three times on a transport error or 502/503/504.
  503 is the engine's own "ask again" — it is what RebuildBusy answers.
- Submit carries a key the engine stores with the run, so a retry after a
  timeout returns that run instead of starting a second. A sweep keys every
  entry, so a half-created one recreates only what is missing.
- wait() and --follow tolerate five failed polls in a row; a 404 still stops
  at once, because that is an answer rather than a gap.
- CLI says "engine not answering" and names the run still on the engine,
  instead of printing a traceback.
- runs: clamp the params column to 80 characters; events() takes the
  flow/since/until the endpoint already had; RunHandle.failures answers
  "what killed this run" from the run's own node rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 21:19:09 +02:00
stroblmeandClaude Opus 5 1148e54c9e A run says whether the dashboard, the CLI or a script asked for it
`POST /runs/flows/{name}` hardcoded `cause: "api"`, so every row in the
history claimed the same origin. The body now carries an optional `cause`,
closed to the values the column knows — the dashboard sends nothing and stays
"api", `fluksio run` says "cli", and the SDK client says "sdk".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Gf7WaExcJ9bs3kfJXB3nK
2026-08-25 22:06:29 +02:00
stroblmeandClaude Opus 5 3503512d05 Report how long a run has been going, not just how long it took
Docs / docs (push) Successful in 20s
Playwright Tests / test-playwright (1, 2) (push) Failing after 2m39s
Playwright Tests / test-playwright (2, 2) (push) Failing after 1m39s
pre-commit / pre-commit (push) Failing after 2m51s
Playwright Tests / merge-reports (push) Canceled after 0s
Test Backend / test-backend (push) Canceled after 1m4s
Compose Smoke Test / test-compose (push) Canceled after 0s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UytviPMJbXzD8P84nLvXcq
2026-08-25 18:33:28 +02:00
stroblme 4350916bd8 Scroll the runs table, pick a range, and choose what a comparison plots against
Docs / docs (push) Successful in 35s
Playwright Tests / test-playwright (1, 2) (push) Failing after 3m14s
Playwright Tests / test-playwright (2, 2) (push) Failing after 1m44s
pre-commit / pre-commit (push) Failing after 3m54s
Test Backend / test-backend (push) Successful in 3m12s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Failing after 1m28s
2026-08-25 16:32:34 +02:00
stroblmeandClaude Opus 5 3c15964364 A cached node keeps its curve, and any input can name a run's output
Docs / docs (push) Successful in 22s
Playwright Tests / test-playwright (1, 2) (push) Failing after 2m42s
Playwright Tests / test-playwright (2, 2) (push) Failing after 1m41s
pre-commit / pre-commit (push) Failing after 2m53s
Test Backend / test-backend (push) Successful in 2m21s
Compose Smoke Test / test-compose (push) Successful in 31s
Playwright Tests / merge-reports (push) Failing after 1m6s
A cache hit still replays no emissions — those values were the story of an
execution that is not happening — but the run they were recorded in is now
written on the row (`run_node.cached_from`), and the metrics endpoints read the
series back from there. So a reused run answers `run.metrics("train.loss")`
with the same points the run that trained did, rather than looking like a run
that produced no numbers at all. Pointed at rather than copied: a sweep of 500
reusing one frozen node would otherwise duplicate its curve 500 times.

That needed the cross-flow restore fixed first. The cache key has no flow in
it while the stored outputs are named for the flow that produced them, so
`quick.prepare` getting a hit from `train` wrote `train.dataset` into `quick`'s
state and the next node was called without its argument. One rule now covers
both halves: `requalify` reads a name owned by one flow as the same name in
another, applied to the restored outputs, to the node id behind the pointer,
and to the series names on the way out. Reuse across flows is kept.

Also: `@run:<id>.<output>` and a bare `sha256:` digest resolve on every input,
not only artifacts. Chaining a run's json config into the next one from a shell
meant pasting the whole object inline, and the CLI could not even send the
spelling — `_coerce` died in `json.loads` before the engine saw it. Both
spellings are reserved on every input now, `str` included, and `_from_run`
returns whatever the run's result holds rather than only a reference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dp9L6gakMVro1K2C5zdtBE
2026-08-25 15:12:28 +02:00
stroblme 7e422c0047 Count runs per flow, and page the run list by offset 2026-08-25 11:34:07 +02:00
stroblmeandClaude Opus 5 4941c2787c Add fluksio status, and ask for a run's parameters at a terminal
Two halves of the same gap: the CLI could start work but not show you any.

`fluksio status` draws the home screen's top half in a terminal — health and
what is wrong with it, every flow with its state and node count, and the
recent runs and failures under them. `--watch` keeps it there. Rich does the
drawing; it was already installed under fastapi's own CLI, and is named now
because a command depends on it.

`fluksio run` with no parameters at a terminal asks for them, one line per
declared input with its declared value in brackets — so Enter through the lot
is what running the defaults looks like, and an artifact input takes the
`@run:` spelling the engine now resolves. A scripted run is untouched: passing
any parameter, or piping the command, skips the questions, as does --defaults.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
2026-08-25 07:42:09 +02:00
stroblmeandClaude Opus 5 93374a310e Refuse what a node cannot publish, and stop timing out work that is fine
Four things the python SDK turned up, each fixed where every client sees it.

A key no port declares is now an error rather than a silent drop, on the
return, the yield and the emit alike — the contract the docs already stated.
The SDK reads literal yields at sync time, so a typo fails before anything
runs, and an emission of one fails the call rather than being logged where
nobody looks.

NaN and infinity are refused at the port. JSON cannot spell either, so one
that travelled came back as a 500, a socket frame that stopped the canvas, or
a metric batch the database dropped whole.

An artifact input takes `@run:<id>.<output>` or a bare digest, resolved on the
engine — so the CLI, the run dialog and a python caller mean the same thing,
and a sweep can pass one at all.

Node timeouts are off by default. The clock measured silence, which a training
node is full of, and remote workers had already stopped enforcing it — their
heartbeat reset it. Now a heartbeat proves the agent rather than the node,
ninety seconds of nothing fails the call either way, and the engine touches
work it is still running so a long node is not redelivered at sixty seconds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019V5bsYGNxcgPs4xXmTPx69
2026-08-25 07:30:14 +02:00
stroblmeandClaude Opus 5 400d7d9c5c Stage caching for batch runs, and an engine that lives in the command
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m5s
Playwright Tests / test-playwright (2, 2) (push) Failing after 20s
pre-commit / pre-commit (push) Failing after 2m33s
Test Backend / test-backend (push) Successful in 2m7s
Compose Smoke Test / test-compose (push) Failing after 20s
Playwright Tests / merge-reports (push) Failing after 1m3s
Publish / publish (push) Failing after 12s
A code node in a batch run is now fingerprinted by its source, its raw
settings and the values it reads — an artifact input counting as its digest,
which is what the content addressing was always for. A run that finds the key
restores what the earlier one returned and skips the node, recorded as
`cached`. The run history is the cache: `run_node.outputs` beside the
`cache_key` the schema already had, no second store. On for code nodes, never
for the built-in and connector types that have side effects; off per node with
`@node(cache=False)` and per run with `--no-cache`.

Emissions are not replayed on a hit, so a cached training node returns its
result without redrawing its curve. Recorded in NOTEPAD.md with the two other
deliberate limits.

`fluksio run --local` boots the real app in the command's own process and
drives it through its ASGI interface behind the ordinary client, so a run no
longer needs a `serve` terminal beside it — same data directory, same history,
and the cache carries between the two. It always waits, because the engine it
starts lives exactly as long as the command.

Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists,
`run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited
run rather than abandoning it, coloured statuses on a terminal, and `name`
made optional on the metrics endpoint so a follower can ask for every series.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 20:31:31 +02:00