Name the code a run ran, and let an interrupted sync finish
Three faults with one root: the stored body of a code-defined node is an import shim, and nothing that mattered was ever read from the code itself. - The run stamp could not identify what ran. The shim imports whatever is on disk when the worker starts, and an uncommitted tree stamps <commit>-dirty for every run it ever produces. Run.code_digest hashes the repository's .py files, memoized on their stat state, and it is read again when the run is actually claimed -- so a sweep queued for hours records the code each of its runs executed, not the code that was there when it was submitted. - The stage cache adopted code that was too new. The fingerprint hashed the shim, which is invariant under any edit to the imported function or anything it calls into, so a re-run was served from cache and answered without the outputs the edit added. It now carries the repo digest and the node's declared ports. Every fingerprint changes once, which invalidates the existing cache; a canvas flow has no repository and keys as before. - An interrupted sync looked like a hand-edited canvas. The engine answers a new-node template for a node with no stored body, and the template carries no marker, so the drift check read "somebody edited this" and demanded --force -- for the one state that re-running the sync is the fix for. NodeSource.missing states the fact, and sync skips those and reuses the bodies it read instead of asking for each one twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -416,12 +416,15 @@ already up, so it is for the run you want now rather than the loop you are
|
||||
iterating in.
|
||||
|
||||
A second run of a flow you did not change mostly does not execute. Each of
|
||||
your nodes is fingerprinted by its source, its settings and the values it
|
||||
reads, and one an earlier run already answered is restored from that run
|
||||
rather than run again — reported as `cached`, so a flow with nothing left to
|
||||
do finishes as `(3/3 cached)`. Change `--lr` and only the nodes downstream of
|
||||
it run. `--no-cache` turns that off for one run, `@node(..., cache=False)` for
|
||||
one node; the caveat and the details are in
|
||||
your nodes is fingerprinted by its settings, the ports it declares, the values
|
||||
it reads, and a digest of your repository's Python files — so editing anything
|
||||
your node calls into invalidates it, not only the decorated function itself.
|
||||
One an earlier run already answered is restored from that run rather than run
|
||||
again — reported as `cached`, so a flow with nothing left to do finishes as
|
||||
`(3/3 cached)`. Change `--lr` and only the nodes downstream of it run. The
|
||||
digest is blunt on purpose: an edit anywhere in the repository re-runs
|
||||
everything. `--no-cache` turns caching off for one run, `@node(...,
|
||||
cache=False)` for one node; the caveat and the details are in
|
||||
[Stage caching](../concepts/runs.md#stage-caching).
|
||||
|
||||
From Python, the flow you declared is also the handle to its runs:
|
||||
@@ -452,6 +455,14 @@ commits — the flow store's and your repository's. That is the answer to "what
|
||||
was the learning rate on the run that got 94%?". When a run failed,
|
||||
`run.failures` is the node that did it, with its traceback and its logs.
|
||||
|
||||
Beside your commit is `code_digest`, and `fluksio runs` prints the pair as
|
||||
`a1b2c3d-dirty+9f0e1a2`. The commit alone cannot identify what ran: your node
|
||||
bodies are imports, so the engine executes whatever is on disk when the worker
|
||||
starts, and an uncommitted tree stamps `-dirty` for every run it ever produces.
|
||||
The digest is read at the moment the run starts — so in a sweep whose runs
|
||||
queue for hours, each one records the code that actually executed it, not the
|
||||
code that was there when you submitted.
|
||||
|
||||
### When the engine is busy
|
||||
|
||||
A driver script outlives the engine being slow, because a sweep is hours long
|
||||
|
||||
Reference in New Issue
Block a user