Check a generator's yield at the yield that produced it
The worker held each yield one behind, because the last one is the node's result when the generator returns nothing of its own. Only the engine knows what ports a node declared, so the check happened when the *next* yield arrived — a pass late, which for a training loop is however long one epoch takes. The worker now sends every yield as it happens and returns whatever its generator returned; EmitSink holds the last one back and decides at the end of the call what it was. Old "emit" frames are still handled, so a remote agent that has not been restarted keeps working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -262,6 +262,25 @@ def test_an_emission_that_nothing_declares_names_what_was_emitted():
|
||||
assert "study.undeclared" not in state
|
||||
|
||||
|
||||
def test_a_mistyped_port_fails_at_the_yield_that_produced_it():
|
||||
"""Not at the one after it: a training loop's second pass can be an hour."""
|
||||
passes = []
|
||||
|
||||
def stray(params):
|
||||
passes.append(1)
|
||||
yield {"undeclared": 1.0}
|
||||
passes.append(2)
|
||||
yield {"loss": 0.5}
|
||||
|
||||
node = make_node("train", "study", stray, provides=[spec("loss", stream=True)])
|
||||
seen: list[NodeOutcome] = []
|
||||
Pipeline(nodes=[node], state=MemoryState(), observer=seen.append).run()
|
||||
|
||||
assert not seen[0].ok
|
||||
assert "undeclared" in seen[0].error
|
||||
assert passes == [1]
|
||||
|
||||
|
||||
def test_emissions_reach_the_run_as_a_series_with_a_step_each():
|
||||
sink = MetricSink("run-1", batch=1)
|
||||
sink.handle("study.train", {"study.loss": 1.0, "study.tag": "ignored"})
|
||||
|
||||
Reference in New Issue
Block a user