Key a node on the code it reaches, not on the repository around it
Docs / docs (push) Successful in 21s
Playwright Tests / test-playwright (1, 2) (push) Successful in 2m13s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m52s
pre-commit / pre-commit (push) Failing after 2m6s
Test Backend / test-backend (push) Successful in 2m30s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m53s
Docs / docs (push) Successful in 21s
Playwright Tests / test-playwright (1, 2) (push) Successful in 2m13s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m52s
pre-commit / pre-commit (push) Failing after 2m6s
Test Backend / test-backend (push) Successful in 2m30s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m53s
`sync` follows each node function's imports through the project's own modules — stopping at the standard library, at anything installed, and at Fluksio itself, whose checkout would otherwise be most of every digest — and records the file list with what it hashed to. The engine hashes those files again when the run is claimed, so the fingerprint is live rather than a snapshot, and falls back to what sync recorded when it cannot see them: a remote worker's runs used to share one empty digest, and therefore one key. Three things follow. Editing a helper a node calls into re-runs that node, as before. Editing something the node never reaches no longer re-runs anything — a notebook two directories away was invalidating every arm. And `Run.code_digest` is now the hash of its nodes' digests, so it is neither looser nor tighter than "the code behind these numbers", which is what makes it worth joining an exported table on. `sync` says so too: it compares the per-node digest against the stored one, so a helper edit prints `train: updated (flow, fit)` instead of `unchanged`. The digest is read when the document is built rather than when the flow is declared, so a second `sync()` in one process sees an edit between them. Also: `fluksio runs` shows only the inputs that differ from what the flow declares, fitted to the terminal, so a flow taking a few kB of json no longer wraps every line. Every existing cache entry misses once — the fingerprint changed shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9Hdrmf2cwNABCnE5x9UJa
This commit is contained in:
+14
-7
@@ -191,13 +191,20 @@ on a run carries the `cache_key` it was looked up by.
|
||||
|
||||
For a [code-defined flow](../getting-started/data-science.md), "its source" is
|
||||
the generated shim, which imports the real function and does not change when
|
||||
that function does. So the key carries one thing more: a digest of every
|
||||
`.py` file under the repository the flow was declared in, read when the run
|
||||
starts. Editing a helper three calls down from the node invalidates it, which
|
||||
is the point — the alternative is a re-run answering with the previous code's
|
||||
numbers. It is deliberately blunt: any edit anywhere in the repository re-runs
|
||||
every node of its flows. An engine that cannot see the repository — a worker on
|
||||
another machine — records no digest and keys as it did before.
|
||||
that function does. So the key carries one thing more: a digest of the project
|
||||
modules that node's function reaches, worked out by `fluksio sync` — which is
|
||||
the only side that imports your code and can see what it imports — and read
|
||||
again from those files when the run starts.
|
||||
|
||||
Editing a helper three calls down from the node invalidates it, which is the
|
||||
point: the alternative is a re-run answering with the previous code's numbers.
|
||||
Editing something the node does not reach leaves the hit standing, which is
|
||||
the other half — a notebook two directories away is not a reason to retrain.
|
||||
The walk follows imports statically and stops at the standard library, at
|
||||
anything installed, and at Fluksio itself; a module imported under a name the
|
||||
code computes is not followed. An engine that cannot see the files keeps what
|
||||
sync recorded instead of nothing, so a worker on another machine no longer
|
||||
keys every run the same.
|
||||
|
||||
The run history *is* the cache; there is no second store. A node's returned
|
||||
outputs are kept on its run record as canonical JSON, up to 256000 characters —
|
||||
|
||||
Reference in New Issue
Block a user