Name the code a run ran, and let an interrupted sync finish

Three faults with one root: the stored body of a code-defined node is an
import shim, and nothing that mattered was ever read from the code itself.

- The run stamp could not identify what ran. The shim imports whatever is on
  disk when the worker starts, and an uncommitted tree stamps <commit>-dirty
  for every run it ever produces. Run.code_digest hashes the repository's .py
  files, memoized on their stat state, and it is read again when the run is
  actually claimed -- so a sweep queued for hours records the code each of its
  runs executed, not the code that was there when it was submitted.
- The stage cache adopted code that was too new. The fingerprint hashed the
  shim, which is invariant under any edit to the imported function or anything
  it calls into, so a re-run was served from cache and answered without the
  outputs the edit added. It now carries the repo digest and the node's
  declared ports. Every fingerprint changes once, which invalidates the
  existing cache; a canvas flow has no repository and keys as before.
- An interrupted sync looked like a hand-edited canvas. The engine answers a
  new-node template for a node with no stored body, and the template carries
  no marker, so the drift check read "somebody edited this" and demanded
  --force -- for the one state that re-running the sync is the fix for.
  NodeSource.missing states the fact, and sync skips those and reuses the
  bodies it read instead of asking for each one twice.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-26 21:27:10 +02:00
co-authored by Claude Opus 5
parent 1f7c6646f1
commit 4a38c6ed31
15 changed files with 422 additions and 28 deletions
+16 -6
View File
@@ -177,18 +177,28 @@ downloadable at `GET /api/v1/artifacts/{digest}`.
## Stage caching
A run mostly does not redo what an earlier one already did. Before a node
executes it is fingerprinted — a sha256 over its source, its settings and the
values it is about to read — and if some earlier run of that same fingerprint
finished, what that one returned is restored into this run's state and the node
is skipped. It is recorded with the status `cached` and a duration of zero, and
its artifacts are listed on the new run as well, so they stay downloadable from
either.
executes it is fingerprinted — a sha256 over its source, its settings, the ports
it declares and the values it is about to read — and if some earlier run of that
same fingerprint finished, what that one returned is restored into this run's
state and the node is skipped. It is recorded with the status `cached` and a
duration of zero, and its artifacts are listed on the new run as well, so they
stay downloadable from either.
The settings go into the key raw, so a secret contributes its `{"$secret":
name}` reference and never its value. An artifact input counts as its content
digest: the same bytes under a different filename are the same input. Each node
on a run carries the `cache_key` it was looked up by.
For a [code-defined flow](../getting-started/data-science.md), "its source" is
the generated shim, which imports the real function and does not change when
that function does. So the key carries one thing more: a digest of every
`.py` file under the repository the flow was declared in, read when the run
starts. Editing a helper three calls down from the node invalidates it, which
is the point — the alternative is a re-run answering with the previous code's
numbers. It is deliberately blunt: any edit anywhere in the repository re-runs
every node of its flows. An engine that cannot see the repository — a worker on
another machine — records no digest and keys as it did before.
The run history *is* the cache; there is no second store. A node's returned
outputs are kept on its run record as canonical JSON, up to 256000 characters —
a node returning more than that is simply not cacheable that run. An entry
+17 -6
View File
@@ -416,12 +416,15 @@ already up, so it is for the run you want now rather than the loop you are
iterating in.
A second run of a flow you did not change mostly does not execute. Each of
your nodes is fingerprinted by its source, its settings and the values it
reads, and one an earlier run already answered is restored from that run
rather than run again — reported as `cached`, so a flow with nothing left to
do finishes as `(3/3 cached)`. Change `--lr` and only the nodes downstream of
it run. `--no-cache` turns that off for one run, `@node(..., cache=False)` for
one node; the caveat and the details are in
your nodes is fingerprinted by its settings, the ports it declares, the values
it reads, and a digest of your repository's Python files — so editing anything
your node calls into invalidates it, not only the decorated function itself.
One an earlier run already answered is restored from that run rather than run
again — reported as `cached`, so a flow with nothing left to do finishes as
`(3/3 cached)`. Change `--lr` and only the nodes downstream of it run. The
digest is blunt on purpose: an edit anywhere in the repository re-runs
everything. `--no-cache` turns caching off for one run, `@node(...,
cache=False)` for one node; the caveat and the details are in
[Stage caching](../concepts/runs.md#stage-caching).
From Python, the flow you declared is also the handle to its runs:
@@ -452,6 +455,14 @@ commits — the flow store's and your repository's. That is the answer to "what
was the learning rate on the run that got 94%?". When a run failed,
`run.failures` is the node that did it, with its traceback and its logs.
Beside your commit is `code_digest`, and `fluksio runs` prints the pair as
`a1b2c3d-dirty+9f0e1a2`. The commit alone cannot identify what ran: your node
bodies are imports, so the engine executes whatever is on disk when the worker
starts, and an uncommitted tree stamps `-dirty` for every run it ever produces.
The digest is read at the moment the run starts — so in a sweep whose runs
queue for hours, each one records the code that actually executed it, not the
code that was there when you submitted.
### When the engine is busy
A driver script outlives the engine being slow, because a sweep is hours long