Skip to content

fix(tests): patch the catalog-request seam in gemini probe UA tests - #64200

Closed
Soju06 wants to merge 1 commit into
NousResearch:mainfrom
Soju06:upstream-pr/fix-gemini-probe-tests
Closed

fix(tests): patch the catalog-request seam in gemini probe UA tests#64200
Soju06 wants to merge 1 commit into
NousResearch:mainfrom
Soju06:upstream-pr/fix-gemini-probe-tests

Conversation

@Soju06

@Soju06 Soju06 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Problem

main CI is currently red: tests/hermes_cli/test_model_validation.py::TestProbeApiModelsUserAgent::test_probe_sends_client_context_to_gemini and ::test_probe_omits_gemini_client_context_for_other_providers (added in b8eb89f) fail on every run with:

TypeError: 'NoneType' object is not subscriptable

The tests patch hermes_cli.models.urllib.request.urlopen, but probe_api_models sends catalog requests through open_credentialed_url, which opens via an OpenerDirector (opener.open(...)) and never calls the module-level urlopen. The mock is never invoked, the request escapes to the real network (and fails in CI), and mock_urlopen.call_args is None.

This also fails every open PR's "All required checks pass" gate, since the merge ref inherits the broken tests.

Change

Patch hermes_cli.models._urlopen_model_catalog_request instead — the same seam every neighboring test in this class already uses. Assertions unchanged; the header behavior under test (X-goog-api-client present for the official Gemini endpoint, absent otherwise) passes once the mock actually intercepts.

Test-only change.

Tests

  • tests/hermes_cli/test_model_validation.py + tests/agent/test_gemini_native_adapter.py + tests/tools/test_tts_gemini.py: 139 passed.
  • The two fixed tests fail before this change and pass after, with no production code touched.

🤖 Generated with Claude Code

test_probe_sends_client_context_to_gemini and
test_probe_omits_gemini_client_context_for_other_providers patch
hermes_cli.models.urllib.request.urlopen, but probe_api_models routes
requests through open_credentialed_url's OpenerDirector, which never
calls the module-level urlopen. The mock is never hit, the request
escapes to the real network, and mock_urlopen.call_args is None —
'NoneType' object is not subscriptable on every CI run since the tests
landed. Patch _urlopen_model_catalog_request instead, the same seam
every neighboring test in this class uses.
@teknium1

Copy link
Copy Markdown
Contributor

Thanks @Soju06 — you submitted this fix first (~3h before ours), and your diagnosis was exactly right: the probe requests go through the _urlopen_model_catalog_request seam (open_credentialed_url's OpenerDirector), so patching module-level urllib.request.urlopen never intercepts anything and call_args comes back None.

Our duplicate-PR sweep unfortunately missed this before we shipped the identical repoint in #64318 (merged as 0d3ad19) to unblock CI, which was red on every run. Apologies that the fixes crossed — credit for the first correct diagnosis and fix is yours.

Closing since the test change is now redundant on main. The AUTHOR_MAP addition in scripts/release.py is best bundled with a salvage of one of your substantive PRs rather than standalone, so it isn't lost — it'll ride along there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists type/test Test coverage or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants