Key a node on the code it reaches, not on the repository around it
Docs / docs (push) Successful in 21s
Playwright Tests / test-playwright (1, 2) (push) Successful in 2m13s
Playwright Tests / test-playwright (2, 2) (push) Successful in 1m52s
pre-commit / pre-commit (push) Failing after 2m6s
Test Backend / test-backend (push) Successful in 2m30s
Compose Smoke Test / test-compose (push) Successful in 32s
Playwright Tests / merge-reports (push) Successful in 1m53s

`sync` follows each node function's imports through the project's own modules
— stopping at the standard library, at anything installed, and at Fluksio
itself, whose checkout would otherwise be most of every digest — and records
the file list with what it hashed to. The engine hashes those files again when
the run is claimed, so the fingerprint is live rather than a snapshot, and
falls back to what sync recorded when it cannot see them: a remote worker's
runs used to share one empty digest, and therefore one key.

Three things follow. Editing a helper a node calls into re-runs that node, as
before. Editing something the node never reaches no longer re-runs anything —
a notebook two directories away was invalidating every arm. And
`Run.code_digest` is now the hash of its nodes' digests, so it is neither
looser nor tighter than "the code behind these numbers", which is what makes
it worth joining an exported table on.

`sync` says so too: it compares the per-node digest against the stored one, so
a helper edit prints `train: updated (flow, fit)` instead of `unchanged`. The
digest is read when the document is built rather than when the flow is
declared, so a second `sync()` in one process sees an edit between them.

Also: `fluksio runs` shows only the inputs that differ from what the flow
declares, fitted to the terminal, so a flow taking a few kB of json no longer
wraps every line.

Every existing cache entry misses once — the fingerprint changed shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9Hdrmf2cwNABCnE5x9UJa
This commit is contained in:
2026-08-27 22:35:23 +02:00
co-authored by Claude Opus 5
parent 4479eeb726
commit 91ef2bbe9a
12 changed files with 530 additions and 36 deletions
+19 -7
View File
@@ -160,7 +160,14 @@ file path, because the shim has to import the same way.
Every sync retires the engine's workers, including one that had nothing to
upload — a worker holds your package in memory, so an edit to it is invisible
until the process goes. See
until the process goes.
It also records, per node, which of your modules that node's function imports
its way to, and what they hash to. That is what the
[stage cache](../concepts/runs.md#stage-caching) keys on, so editing a
helper a node calls into is reported as that node changing — `train: updated
(flow, fit)` — and re-runs it, while editing something the node never reaches
is left alone. See
[Getting started: data science](../getting-started/data-science.md).
### `fluksio run`
@@ -275,12 +282,17 @@ fluksio runs [--flow train] [--limit 20]
```
The runs an engine has recorded, newest first: id, status, flow, duration, the
commit of the repository it came from, and its parameters. Statuses are
coloured when a terminal is reading the output — `ok` green, `error` red,
`cached` cyan. Parameters are clamped to 80 characters so a flow taking a few
kB of JSON still lists as a table; `Client.runs()` is where the whole value is
read. `--local` reads the same history from an in-process engine, without one
having to be served.
commit of the repository it came from, and the inputs it was given. Statuses
are coloured when a terminal is reading the output — `ok` green, `error` red,
`cached` cyan.
Only the inputs that *differ from what the flow declares* are shown, and they
are clamped to what is left of the terminal's width — a run that took the
defaults lists none at all, and a flow taking a few kB of JSON does not push
everything else off the line. `Client.runs()` and
[`fluksio export runs`](#fluksio-export) are where the whole value is read.
`--local` reads the same history from an in-process engine, without one having
to be served.
### `fluksio flavors`
+14 -7
View File
@@ -191,13 +191,20 @@ on a run carries the `cache_key` it was looked up by.
For a [code-defined flow](../getting-started/data-science.md), "its source" is
the generated shim, which imports the real function and does not change when
that function does. So the key carries one thing more: a digest of every
`.py` file under the repository the flow was declared in, read when the run
starts. Editing a helper three calls down from the node invalidates it, which
is the point — the alternative is a re-run answering with the previous code's
numbers. It is deliberately blunt: any edit anywhere in the repository re-runs
every node of its flows. An engine that cannot see the repository — a worker on
another machine — records no digest and keys as it did before.
that function does. So the key carries one thing more: a digest of the project
modules that node's function reaches, worked out by `fluksio sync` — which is
the only side that imports your code and can see what it imports — and read
again from those files when the run starts.
Editing a helper three calls down from the node invalidates it, which is the
point: the alternative is a re-run answering with the previous code's numbers.
Editing something the node does not reach leaves the hit standing, which is
the other half — a notebook two directories away is not a reason to retrain.
The walk follows imports statically and stops at the standard library, at
anything installed, and at Fluksio itself; a module imported under a name the
code computes is not followed. An engine that cannot see the files keeps what
sync recorded instead of nothing, so a worker on another machine no longer
keys every run the same.
The run history *is* the cache; there is no second store. A node's returned
outputs are kept on its run record as canonical JSON, up to 256000 characters —
+7 -3
View File
@@ -542,9 +542,13 @@ Beside your commit is `code_digest`, and `fluksio runs` prints the pair as
`a1b2c3d-dirty+9f0e1a2`. The commit alone cannot identify what ran: your node
bodies are imports, so the engine executes whatever is on disk when the worker
starts, and an uncommitted tree stamps `-dirty` for every run it ever produces.
The digest is read at the moment the run starts — so in a sweep whose runs
queue for hours, each one records the code that actually executed it, not the
code that was there when you submitted.
The digest is over the modules this flow's nodes actually reach — not the
whole repository, so it moves when the code behind a number moves and stays
put when a notebook beside it changes — and it is read at the moment the run
starts, so in a sweep whose runs queue for hours each one records the code
that actually executed it rather than the code that was there when you
submitted. It is what makes the column worth joining an
[exported table](../concepts/runs.md#taking-it-into-a-dataframe) on.
### When the engine is busy