Skip to content

fix(tests): add missing mocks for vision fallback and codex gateway tests - #4297

Closed
crazywriter1 wants to merge 1 commit into
NousResearch:mainfrom
crazywriter1:fix/flaky-test-missing-mocks
Closed

fix(tests): add missing mocks for vision fallback and codex gateway tests#4297
crazywriter1 wants to merge 1 commit into
NousResearch:mainfrom
crazywriter1:fix/flaky-test-missing-mocks

Conversation

@crazywriter1

Copy link
Copy Markdown
Contributor

Summary

Fixes 2 pre-existing test failures that affect main and all open PRs (e.g. #4245, #4246).

These tests pass locally in some environments but consistently fail in CI because they depend on runtime config resolution that behaves differently without a real config.yaml or hermes_cli setup.

Failing tests

1. test_vision_auto_falls_back_to_custom_endpoint

  • File: tests/agent/test_auxiliary_client.py
  • Error: assert None is not None
  • Root cause: The test sets OPENAI_BASE_URL and OPENAI_API_KEY env vars expecting the vision auto-fallback to pick up the custom endpoint. However, _try_custom_endpoint() delegates to _resolve_custom_runtime() which imports hermes_cli.runtime_provider.resolve_runtime_provider — in CI this resolution returns (None, None) because there is no config file, so the custom endpoint is never constructed.
  • Fix: Mock _resolve_custom_runtime to return the test endpoint values directly, matching what the test already sets via env vars.

2. test_gateway_run_agent_codex_path_handles_internal_401_refresh

  • File: tests/test_codex_execution_paths.py
  • Error: assert '⚠️ Codex Responses request 'model' must be a non-empty string.' == 'Recovered via refresh'
  • Root cause: The test sets HERMES_MODEL env var but _run_agent() resolves the model via _resolve_gateway_model() which reads from config.yaml, not env vars. Without a config file, it returns an empty string — the codex API then rejects the request with a validation error before the 401-refresh logic is ever reached.
  • Fix: Mock _resolve_gateway_model to return the test model string directly.

Changes

  • tests/agent/test_auxiliary_client.py: Added _resolve_custom_runtime mock (+2 lines)
  • tests/test_codex_execution_paths.py: Added _resolve_gateway_model mock (+1 line)

Test plan

  • Both previously failing tests now pass locally
  • No other tests affected (3 insertions only, no behavioral changes)

@crazywriter1
crazywriter1 force-pushed the fix/flaky-test-missing-mocks branch from 1faa3e6 to 4044899 Compare March 31, 2026 17:54
…ests

- Mock _resolve_custom_runtime in vision auto fallback test so the
  custom endpoint is actually reachable in test environment
- Mock _resolve_gateway_model in codex gateway test so the model
  field is populated correctly

These tests were failing in CI due to missing runtime config.
@crazywriter1
crazywriter1 force-pushed the fix/flaky-test-missing-mocks branch from 4044899 to ab4a59c Compare April 6, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant