Skip to content

fix(tests): align run_agent expectations with current routing - #4554

Closed
kshitijk4poor wants to merge 1 commit into
NousResearch:mainfrom
kshitijk4poor:fix/run-agent-test-drift
Closed

fix(tests): align run_agent expectations with current routing#4554
kshitijk4poor wants to merge 1 commit into
NousResearch:mainfrom
kshitijk4poor:fix/run-agent-test-drift

Conversation

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Summary

  • make TestBuildApiKwargs explicitly target the current OpenRouter route/model combinations required for extra_body assertions
  • make the Claude prompt-caching test explicitly use the OpenRouter base URL instead of relying on implicit defaults
  • patch the streaming fallback test to mock the request-scoped OpenAI client factory introduced in the current run_agent.py path

Problem

While validating #4552, the broader tests/test_run_agent.py tests/test_run_agent_codex_responses.py -q run hit unrelated failures on current main in tests/test_run_agent.py:

  • TestBuildApiKwargs::* expected extra_body on a fixture that no longer explicitly targeted an OpenRouter-compatible route/model
  • TestInit.test_prompt_caching_claude_openrouter expected Claude-via-OpenRouter prompt caching without explicitly setting the OpenRouter base URL
  • TestStreamingApiCall.test_api_exception_falls_back_to_non_streaming only patched agent.client, but the current implementation now creates request-scoped clients through _create_request_openai_client

These were test-drift issues, not production regressions in run_agent.py.

Fix

1. Make routing assumptions explicit in TestBuildApiKwargs

The extra_body assertions now set:

  • agent.base_url = "https://openrouter.ai/api/v1"
  • a reasoning-capable OpenRouter model where needed

This matches the current _build_api_kwargs() gates instead of relying on fixture defaults.

2. Make the Claude prompt-caching test explicitly OpenRouter

test_prompt_caching_claude_openrouter now passes the OpenRouter base URL directly, so it tests the intended path instead of an implicit default that has drifted.

3. Patch request-scoped client creation in the streaming fallback test

test_api_exception_falls_back_to_non_streaming now mocks _create_request_openai_client so both the streaming path and the non-streaming fallback use the same mocked client and raise the expected ConnectionError.

Test Results

Passed:

  • python -m pytest tests/test_run_agent.py -q -k "TestBuildApiKwargs or test_prompt_caching_claude_openrouter or test_api_exception_falls_back_to_non_streaming"
  • python -m pytest tests/test_run_agent.py tests/test_run_agent_codex_responses.py -q

This follow-up is intentionally test-only and separate from #4552.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the contribution, @kshitijk4poor!

Closing this as stale. The test tree was rearchitected in #5946 — tests are now under tests/run_agent/ as individual test files rather than a single tests/test_run_agent.py. The file this PR modifies no longer exists on main.

If you want to revisit this for the refactored code on current main, please open a new PR rebased on main targeting the current file layout. Your authorship will be preserved through cherry-pick if we salvage any of your work.

@teknium1 teknium1 closed this Apr 21, 2026
@kshitijk4poor
kshitijk4poor deleted the fix/run-agent-test-drift branch August 5, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants