test(run_agent): align switch_model test with new clear-on-switch semantics - #24776
Closed
briandevans wants to merge 1 commit into
Closed
test(run_agent): align switch_model test with new clear-on-switch semantics#24776briandevans wants to merge 1 commit into
briandevans wants to merge 1 commit into
Conversation
…antics Commit 8ac3514 ("fix(agent): clear stale config context_length on model switch", NousResearch#24724 closing NousResearch#21509) flipped switch_model's behavior: the per-agent _config_context_length override is now cleared to None before the runtime field swap so the new model's actual context window is resolved via the full chain (custom_providers, endpoint probe, models.dev, etc.) instead of inheriting the previous model's stale value. That commit broke tests/run_agent/test_switch_model_context.py:: test_switch_model_preserves_config_context_length, which was added by fd3e855 under the older "preserve across switches" interpretation and still asserts the stale value is passed through to get_model_context_length. The test contradicts the now-intended invariant and fails on every CI run on origin/main with `AssertionError: assert None == 32768`. Update the test (and the module docstring) to verify the new invariant: get_model_context_length receives `config_context_length=None` after the switch, and the agent attribute itself is cleared. The sibling test_switch_model_without_config_context_length keeps passing unchanged because the unconditional-None case is the same in both worlds. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates a failing run_agent regression test to match the updated AIAgent.switch_model() semantics where _config_context_length is cleared on model switch so the new model’s context window is auto-resolved.
Changes:
- Update module docstring to describe the new “clear-on-switch” invariant.
- Rename the test to reflect the new behavior and adjust assertions to expect
config_context_length=Noneand a clearedagent._config_context_length.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
59
to
+62
| mock_ctx_len.assert_called_once() | ||
| call_kwargs = mock_ctx_len.call_args.kwargs | ||
| assert call_kwargs.get("config_context_length") == 32_768 | ||
| assert call_kwargs.get("config_context_length") is None | ||
|
|
alt-glitch
added
type/test
Contributor
Author
|
CI audit — all 20
This PR is a pure test-only change to |
Contributor
Author
|
Closing to keep the queue clean — happy to reopen if this is still useful. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
8ac351407(#24724, closing fix: clear stale config context_length override on model switch #21509) flippedswitch_model's semantics from "preserve_config_context_lengthacross switches" to "clear it so the new model is auto-resolved"tests/run_agent/test_switch_model_context.py::test_switch_model_preserves_config_context_length(added byfd3e855d5under the old interpretation) still asserts the stale value is passed through and is now a baseline CI failure onmainThe bug
On clean
origin/main:The test asserts
get_model_context_lengthis called with the oldconfig_context_length=32_768, but after8ac351407the override is cleared toNonebefore the runtime field swap. The test name "preserves" no longer describes the intended behavior — it was a direct casualty of the policy flip.The fix
Rename the test to
test_switch_model_clears_stale_config_context_lengthand assert the new invariant:get_model_context_lengthis called withconfig_context_length=Noneagent._config_context_lengthitself is cleared on the instance so any subsequent resolution stays cleanThe sibling
test_switch_model_without_config_context_lengthkeeps passing because the unconditional-Nonecase is identical under both policies — left it untouched.Test plan
uv run --with pytest --with pytest-xdist --with pytest-asyncio python3 -m pytest tests/run_agent/test_switch_model_context.py -v→ 2 passeduv run ... python3 -m pytest tests/run_agent/→ 1354 passed, 9 skipped, 0 failedorigin/mainthe renamed test passes; before the rename (i.e. on currentmain)test_switch_model_preserves_config_context_lengthfails withassert None == 32768Related