A sync that changed nothing says so again
Docs / docs (push) Successful in 32s
Playwright Tests / test-playwright (1, 2) (push) Successful in 2m58s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m9s
pre-commit / pre-commit (push) Failing after 2m23s
Test Backend / test-backend (push) Successful in 3m17s
Playwright Tests / merge-reports (push) Canceled after 0s
Compose Smoke Test / test-compose (push) Canceled after 28s
Docs / docs (push) Successful in 32s
Playwright Tests / test-playwright (1, 2) (push) Successful in 2m58s
Playwright Tests / test-playwright (2, 2) (push) Successful in 2m9s
pre-commit / pre-commit (push) Failing after 2m23s
Test Backend / test-backend (push) Successful in 3m17s
Playwright Tests / merge-reports (push) Canceled after 0s
Compose Smoke Test / test-compose (push) Canceled after 28s
Comparing the per-node digest against an engine that does not record one is comparing against nothing, and reporting every node as changed on every sync for ever — which is what a client newer than its engine did, since `NodeDef` drops fields it has never heard of. A node is named now only when both sides carry a digest, so a no-op sync is `unchanged` again and the signal one syncs for is back. That silence had also been the only sign of the mismatch, so sync now names it: one line saying the engine stored no record of what a node's code reaches, with both versions in it and what to run. Bumped to 0.1.6 — the digest changed the stored document's shape, and a version that does not move makes two different engines indistinguishable, which is the thing it was made load-bearing for a day ago. `— draft` was printed whenever there was simply nothing to publish, which reads as work left unfinished. It is said only when a draft is genuinely there, and `— published` when one was. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9Hdrmf2cwNABCnE5x9UJa
This commit is contained in:
@@ -301,8 +301,18 @@ def cmd_sync(args: argparse.Namespace) -> int:
|
|||||||
continue
|
continue
|
||||||
what = "created" if report.created else "updated"
|
what = "created" if report.created else "updated"
|
||||||
detail = ", ".join(report.changed)
|
detail = ", ".join(report.changed)
|
||||||
state = "published" if report.published else "draft"
|
# Nothing to publish is not the same as left as a draft, and saying
|
||||||
_say(f" {report.flow}: {what} ({detail}) — {state}")
|
# the second when it was the first reads as work not finished.
|
||||||
|
state = "published" if report.published else "draft" if report.drafted else ""
|
||||||
|
_say(f" {report.flow}: {what} ({detail})" + (f" — {state}" if state else ""))
|
||||||
|
if any(report.forgot_code for report in reports):
|
||||||
|
engine = _engine_version(client)
|
||||||
|
_say(
|
||||||
|
" note: this engine did not store what each node's code reaches, "
|
||||||
|
"so its cache is still keyed on the whole repository. It is "
|
||||||
|
f"{engine or 'older'} and this client is {__version__} — "
|
||||||
|
"`pip install -U fluksio` there."
|
||||||
|
)
|
||||||
stamp = origin["commit"][:7] + ("-dirty" if origin["dirty"] else "")
|
stamp = origin["commit"][:7] + ("-dirty" if origin["dirty"] else "")
|
||||||
_say(f"Stamped with {stamp or 'no commit'} from {repo}.")
|
_say(f"Stamped with {stamp or 'no commit'} from {repo}.")
|
||||||
return 0
|
return 0
|
||||||
@@ -984,12 +994,17 @@ def _list_names(client: Client, args: argparse.Namespace) -> int:
|
|||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _engine_version(client: Client) -> str:
|
||||||
|
"""What the engine says it is, or empty if it is older than saying so."""
|
||||||
|
try:
|
||||||
|
return str((client.summary() or {}).get("version") or "")
|
||||||
|
except (SyncError, ApiError, httpx.HTTPError):
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
def _too_old(client: Client) -> str:
|
def _too_old(client: Client) -> str:
|
||||||
"""A route this client knows and the engine does not."""
|
"""A route this client knows and the engine does not."""
|
||||||
try:
|
version = _engine_version(client)
|
||||||
version = (client.summary() or {}).get("version") or ""
|
|
||||||
except (ApiError, httpx.HTTPError):
|
|
||||||
version = ""
|
|
||||||
engine = f"the engine is {version}" if version else "the engine is older"
|
engine = f"the engine is {version}" if version else "the engine is older"
|
||||||
return (
|
return (
|
||||||
f"this engine has no export endpoints — {engine} and this client is "
|
f"this engine has no export endpoints — {engine} and this client is "
|
||||||
|
|||||||
@@ -636,6 +636,13 @@ class SyncReport:
|
|||||||
self.created = False
|
self.created = False
|
||||||
self.changed: list[str] = []
|
self.changed: list[str] = []
|
||||||
self.published = False
|
self.published = False
|
||||||
|
#: A draft exists and was left alone, because publishing was not asked
|
||||||
|
#: for. False when there was simply nothing to publish, which is not
|
||||||
|
#: the same thing and used to read as if it were.
|
||||||
|
self.drafted = False
|
||||||
|
#: The engine stored the node document without what each node's code
|
||||||
|
#: reaches, which means it is older than this client.
|
||||||
|
self.forgot_code = False
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def unchanged(self) -> bool:
|
def unchanged(self) -> bool:
|
||||||
@@ -729,6 +736,14 @@ def _sync_one(
|
|||||||
document = target.document(origin)
|
document = target.document(origin)
|
||||||
moved = _moved_code(stored, document)
|
moved = _moved_code(stored, document)
|
||||||
saved = client.put_flow(document | {"version": version})
|
saved = client.put_flow(document | {"version": version})
|
||||||
|
# An engine older than the per-node digest parses the document without
|
||||||
|
# those fields and stores it without them, silently: no error, no 404, and
|
||||||
|
# a cache still keyed on the whole repository.
|
||||||
|
kept = _digests(saved.get("definition"))
|
||||||
|
report.forgot_code = any(
|
||||||
|
digest and not kept.get(node_id)
|
||||||
|
for node_id, digest in _digests(document).items()
|
||||||
|
)
|
||||||
stored_version = int((saved.get("definition") or {}).get("version") or version)
|
stored_version = int((saved.get("definition") or {}).get("version") or version)
|
||||||
if report.created or stored_version != version:
|
if report.created or stored_version != version:
|
||||||
# The store bumps the version only when the content actually changed,
|
# The store bumps the version only when the content actually changed,
|
||||||
@@ -751,12 +766,23 @@ def _sync_one(
|
|||||||
if node_id not in report.changed:
|
if node_id not in report.changed:
|
||||||
report.changed.append(node_id)
|
report.changed.append(node_id)
|
||||||
|
|
||||||
if publish and (client.get_flow(target.name) or {}).get("has_draft"):
|
has_draft = bool((client.get_flow(target.name) or {}).get("has_draft"))
|
||||||
|
if publish and has_draft:
|
||||||
client.publish(target.name, version)
|
client.publish(target.name, version)
|
||||||
report.published = True
|
report.published = True
|
||||||
|
else:
|
||||||
|
# Only when one is actually there. "Nothing needed publishing" and
|
||||||
|
# "left as a draft" are different answers to the same question.
|
||||||
|
report.drafted = has_draft
|
||||||
return report
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def _digests(document: dict[str, Any] | None) -> dict[str, str]:
|
||||||
|
"""Each node's recorded code digest, by node id."""
|
||||||
|
nodes = (document or {}).get("nodes") or []
|
||||||
|
return {str(node.get("id")): str(node.get("code_digest") or "") for node in nodes}
|
||||||
|
|
||||||
|
|
||||||
def _moved_code(stored: dict[str, Any] | None, document: dict[str, Any]) -> list[str]:
|
def _moved_code(stored: dict[str, Any] | None, document: dict[str, Any]) -> list[str]:
|
||||||
"""The nodes whose reachable code changed since the last sync.
|
"""The nodes whose reachable code changed since the last sync.
|
||||||
|
|
||||||
@@ -764,18 +790,19 @@ def _moved_code(stored: dict[str, Any] | None, document: dict[str, Any]) -> list
|
|||||||
the shim reflects the declared ports rather than what the function calls
|
the shim reflects the declared ports rather than what the function calls
|
||||||
into — so editing a helper printed as nothing at all, which reads as
|
into — so editing a helper printed as nothing at all, which reads as
|
||||||
"there was nothing to do".
|
"there was nothing to do".
|
||||||
|
|
||||||
|
A node is named only when *both* sides carry a digest. An engine that does
|
||||||
|
not record one drops the field on the way in, so comparing against nothing
|
||||||
|
would report every node as changed on every sync, for ever — which is a
|
||||||
|
worse answer than the silence this replaced.
|
||||||
"""
|
"""
|
||||||
if stored is None:
|
if stored is None:
|
||||||
return []
|
return []
|
||||||
before = {
|
before = _digests(stored.get("definition"))
|
||||||
node.get("id"): node.get("code_digest") or ""
|
|
||||||
for node in (stored.get("definition") or {}).get("nodes") or []
|
|
||||||
}
|
|
||||||
return [
|
return [
|
||||||
node_id
|
node_id
|
||||||
for node in document.get("nodes") or []
|
for node_id, digest in _digests(document).items()
|
||||||
if (node_id := node.get("id")) in before
|
if digest and before.get(node_id) and before[node_id] != digest
|
||||||
and before[node_id] != (node.get("code_digest") or "")
|
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "fluksio"
|
name = "fluksio"
|
||||||
version = "0.1.5"
|
version = "0.1.6"
|
||||||
description = "Node-based automation engine: flows, dashboards, batch runs"
|
description = "Node-based automation engine: flows, dashboards, batch runs"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
license = "AGPL-3.0-or-later"
|
license = "AGPL-3.0-or-later"
|
||||||
|
|||||||
@@ -104,6 +104,25 @@ def test_a_node_whose_helper_moved_is_named_by_sync():
|
|||||||
assert _moved_code(None, document) == []
|
assert _moved_code(None, document) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_engine_that_records_no_digest_is_not_a_change_every_time():
|
||||||
|
"""It drops the field on parse, so comparing against it reports for ever.
|
||||||
|
|
||||||
|
Which is what an engine older than this client does, and what a node
|
||||||
|
drawn on the canvas looks like — neither is a helper somebody edited.
|
||||||
|
"""
|
||||||
|
from fluksio.sdk.client import _moved_code
|
||||||
|
|
||||||
|
older = {"definition": {"nodes": [{"id": "fit"}, {"id": "prepare"}]}}
|
||||||
|
document = {
|
||||||
|
"nodes": [
|
||||||
|
{"id": "fit", "code_digest": "ccc"},
|
||||||
|
{"id": "prepare", "code_digest": ""},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
assert _moved_code(older, document) == []
|
||||||
|
|
||||||
|
|
||||||
def test_a_second_sync_changes_nothing_and_commits_nothing(api, flows):
|
def test_a_second_sync_changes_nothing_and_commits_nothing(api, flows):
|
||||||
sync([flows["train"]], api, origin=ORIGIN)
|
sync([flows["train"]], api, origin=ORIGIN)
|
||||||
before = commits()
|
before = commits()
|
||||||
|
|||||||
+7
-1
@@ -167,7 +167,13 @@ its way to, and what they hash to. That is what the
|
|||||||
[stage cache](../concepts/runs.md#stage-caching) keys on, so editing a
|
[stage cache](../concepts/runs.md#stage-caching) keys on, so editing a
|
||||||
helper a node calls into is reported as that node changing — `train: updated
|
helper a node calls into is reported as that node changing — `train: updated
|
||||||
(flow, fit)` — and re-runs it, while editing something the node never reaches
|
(flow, fit)` — and re-runs it, while editing something the node never reaches
|
||||||
is left alone. See
|
is left alone.
|
||||||
|
|
||||||
|
A sync that changed nothing says `unchanged`, which is the answer worth
|
||||||
|
having. `— published` and `— draft` are said only when there was something to
|
||||||
|
publish or a draft was genuinely left behind. An engine too old to store what
|
||||||
|
a node's code reaches says so in a line naming both versions; until it is
|
||||||
|
upgraded its cache is keyed on the whole repository, as it was before. See
|
||||||
[Getting started: data science](../getting-started/data-science.md).
|
[Getting started: data science](../getting-started/data-science.md).
|
||||||
|
|
||||||
### `fluksio run`
|
### `fluksio run`
|
||||||
|
|||||||
@@ -869,7 +869,7 @@ wheels = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "fluksio"
|
name = "fluksio"
|
||||||
version = "0.1.5"
|
version = "0.1.6"
|
||||||
source = { editable = "backend" }
|
source = { editable = "backend" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "aiomqtt" },
|
{ name = "aiomqtt" },
|
||||||
|
|||||||
Reference in New Issue
Block a user