feat(experimentalist): stop a run once the objective is reached - #1197
Merged
Conversation
The loop had two ways to stop: the round budget, and a judgement that progress had stalled. Neither notices success. A round that goes 0.333 -> 1.000 is the clearest possible case of *not* stagnating, so a solved run kept buying rounds that could only match what it already had -- and a baseline that already met the goal paid for the whole budget before returning itself. Adds an optional `target` to MetricTarget: the value at which that objective is satisfied, in the metric's own units. When every targeted objective is met, the run stops. Unset, nothing changes -- which is the only safe default, because metrics are not required to be normalized and there is no value that means "as good as possible" for an arbitrary one. Three properties worth stating, because each was a decision: Not gated by `disable_convergence_check`. That flag turns off an *inference* about stagnation, which is worth disabling when the signal is unreliable. This is a measurement against a threshold the caller stated. Anyone who does not want the run to stop here simply does not set a target. Checked at the top of every round including the first. The baseline is scored before the loop begins, so a baseline that already meets the targets ends the run without a single round being paid for. Only candidates that could win are considered -- survivors and the round-0 baseline, mirroring finalization. A killed candidate meeting the target would otherwise end the run in favour of a winner that never reaches it. Insight-authored objectives carry no target, so Mode 1 is unaffected: an LLM-invented metric has no known satisfied value. test_metric_contract now pins that, alongside the `target` a demoted objective keeps when it becomes a guardrail. Signed-off-by: Christian Schüller <cschueller@nvidia.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughChangesObjective-based termination
Sequence Diagram(s)sequenceDiagram
participant Terminator
participant EvolutionTree
participant MetricTarget
Terminator->>EvolutionTree: inspect scored survivors and baseline nodes
EvolutionTree-->>Terminator: return eligible candidates and metrics
Terminator->>MetricTarget: evaluate configured targets
MetricTarget-->>Terminator: return satisfaction results
Terminator-->>Terminator: stop before budget or convergence checks
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Twenty lines restating the commit message. Cut to the four things a caller needs: how it differs from convergence, why `disable_convergence_check` does not gate it, that it runs before the first round, and which candidates are eligible. The reasoning behind each stays in the commit that introduced them. Signed-off-by: Christian Schüller <cschueller@nvidia.com>
Contributor
|
Signed-off-by: Gaia Di Lorenzo <gdilorenzo@ethz.ch>
gaiadilorenzo
approved these changes
Aug 11, 2026
Merged
15 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The loop had two ways to stop: the round budget, and a judgement that progress had stalled. Neither notices success. A round that goes
0.333 -> 1.000is the clearest possible case of not stagnating, so a solved run kept buying rounds that could only match what it already had — and a baseline that already met the goal paid for the whole budget before returning itself.This adds an optional
targettoMetricTarget: the value at which that objective is satisfied, in the metric's own units. When every targeted objective is met, the run stops. Unset, nothing changes.Before and after, on a run whose first round solves the problem:
target: 1.0)objective reached by agent-1 (reward=1.0)targetconfiguredChanges
MetricTarget.target: float | None, withis_satisfied_by()handling direction and treating a missing measurement as not satisfied.Terminator.assess_objective_reached, consulted first inTerminator.run().test_objective_reached.py— 13 tests.test_metric_contract.py— updated for the new field, plus an assertion that insight-authored objectives carry no target.Three properties, each a deliberate decision:
Not gated by
disable_convergence_check. That flag turns off an inference about stagnation, which is worth disabling when the signal is unreliable. This is a measurement against a threshold the caller stated. Anyone who does not want the run to stop here does not set a target.Checked at the top of every round, including the first. The baseline is scored before the loop begins, so a baseline already meeting the targets ends the run without a single round being paid for.
Only candidates that could win are considered — survivors and the round-0 baseline, mirroring finalization. A killed candidate meeting the target would otherwise end the run in favour of a winner that never reaches it.
Unset is the only safe default: metrics are not required to be normalized, so there is no value that means "as good as possible" for an arbitrary one. Insight-authored objectives get no target for the same reason, leaving Mode 1 unaffected — an LLM-invented metric has no known satisfied value.
Type of Change
Quality Gates
Field(description=...), which is where the other optimizer settings are described; there is no separate settings reference to update.Verification
Signed-off-by:traileruv run pre-commit run -apasses, or any blocked checks are identified belowTargeted validation:
That hook is environmental and pre-existing — it needs uv 0.9.14 locally and this machine has 0.9.30. It was reproduced on pristine
mainwith these changes stashed while preparing #1195, and CI's ownCheck uv lockandCheck latest uv compatibilityboth passed there. This PR modifies nopyproject.tomland nouv.lock. Left unchecked rather than claiming a gate that did not pass.Behaviour was also confirmed by mutation rather than by green tests alone: with no target the run continues exactly as before; with
target: 1.0the same tree stops and reportsobjective reached by agent-1 (reward=1.0).Note
Found while running a fixture whose expected score is known in advance. Three groups each reached
1.000and then spent a further round that could not improve on it, and one reached it in round 1 of 2. Nothing was wrong with those runs — there was simply no way to say "this is finished".Summary by CodeRabbit
New Features
Bug Fixes