Stage caching for batch runs, and an engine that lives in the command
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m5s
Playwright Tests / test-playwright (2, 2) (push) Failing after 20s
pre-commit / pre-commit (push) Failing after 2m33s
Test Backend / test-backend (push) Successful in 2m7s
Compose Smoke Test / test-compose (push) Failing after 20s
Playwright Tests / merge-reports (push) Failing after 1m3s
Publish / publish (push) Failing after 12s
Docs / docs (push) Successful in 19s
Playwright Tests / test-playwright (1, 2) (push) Failing after 1m5s
Playwright Tests / test-playwright (2, 2) (push) Failing after 20s
pre-commit / pre-commit (push) Failing after 2m33s
Test Backend / test-backend (push) Successful in 2m7s
Compose Smoke Test / test-compose (push) Failing after 20s
Playwright Tests / merge-reports (push) Failing after 1m3s
Publish / publish (push) Failing after 12s
A code node in a batch run is now fingerprinted by its source, its raw settings and the values it reads — an artifact input counting as its digest, which is what the content addressing was always for. A run that finds the key restores what the earlier one returned and skips the node, recorded as `cached`. The run history is the cache: `run_node.outputs` beside the `cache_key` the schema already had, no second store. On for code nodes, never for the built-in and connector types that have side effects; off per node with `@node(cache=False)` and per run with `--no-cache`. Emissions are not replayed on a hit, so a cached training node returns its result without redrawing its curve. Recorded in NOTEPAD.md with the two other deliberate limits. `fluksio run --local` boots the real app in the command's own process and drives it through its ASGI interface behind the ordinary client, so a run no longer needs a `serve` terminal beside it — same data directory, same history, and the cache carries between the two. It always waits, because the engine it starts lives exactly as long as the command. Also: `fluksio sweep --param lr=0.1,0.01` for the product of the lists, `run --follow` for a run's numbers as they arrive, Ctrl-C cancelling a waited run rather than abandoning it, coloured statuses on a terminal, and `name` made optional on the metrics endpoint so a follower can ask for every series. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -15,6 +15,8 @@ nodes are built like any other's, so `set_enabled` only starts or stops them.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
import traceback
|
||||
@@ -51,7 +53,13 @@ from fluksio.flow.nodes import (
|
||||
SwitchNode,
|
||||
TriggerNode,
|
||||
)
|
||||
from fluksio.flow.pipeline import NodeOutcome, Pipeline, ValidationIssue, ValueSource
|
||||
from fluksio.flow.pipeline import (
|
||||
NodeOutcome,
|
||||
Pipeline,
|
||||
RunCacheLookup,
|
||||
ValidationIssue,
|
||||
ValueSource,
|
||||
)
|
||||
from fluksio.flow.remote import RemoteWorkerHub
|
||||
from fluksio.flow.schemas import (
|
||||
BrainEdge,
|
||||
@@ -904,6 +912,17 @@ class FlowController:
|
||||
params=params,
|
||||
name=node_def.id,
|
||||
)
|
||||
if run is not None and node_def.cache:
|
||||
# The raw params, not the resolved ones: a secret's value
|
||||
# must not end up in a key, and its name is what changes
|
||||
# when the node is reconfigured anyway.
|
||||
node.fingerprint = hashlib.sha256(
|
||||
json.dumps(
|
||||
{"source": code, "params": node_def.params},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode()
|
||||
).hexdigest()
|
||||
emissions.node = node
|
||||
else:
|
||||
node = node_type.cls(
|
||||
@@ -1339,6 +1358,7 @@ class FlowController:
|
||||
observer: Callable[[NodeOutcome], None] | None = None,
|
||||
emission_observer: Callable[[str, dict[str, Any]], None] | None = None,
|
||||
run: RunContext | None = None,
|
||||
run_cache: RunCacheLookup | None = None,
|
||||
) -> Pipeline:
|
||||
"""Build one flow as a pipeline of its own, for a single run.
|
||||
|
||||
@@ -1359,6 +1379,7 @@ class FlowController:
|
||||
initial_values=initial_values,
|
||||
observer=observer,
|
||||
emission_observer=emission_observer,
|
||||
run_cache=run_cache,
|
||||
)
|
||||
pipeline.history_limits = self.history_limits
|
||||
return pipeline
|
||||
|
||||
Reference in New Issue
Block a user