fix(models): use certifi CA bundle for live model-list probes - #51563
fix(models): use certifi CA bundle for live model-list probes#51563isaachuangGMICLOUD wants to merge 1 commit into
Conversation
The live /models probe is a hand-rolled urllib call that trusts Python's
default OpenSSL trust store. On python.org macOS installs that never ran
"Install Certificates.command" that store is empty, so the probe fails with
CERTIFICATE_VERIFY_FAILED even though the cert is valid — and the picker
silently degrades to the static fallback list (e.g. GLM-5.2 on GMI looked
"unsupported"). curl and the chat path both work because the OpenAI SDK /
httpx / requests already bundle certifi; only these hand-rolled probes were
left relying on the system store.
Union certifi (a pinned dependency) on top of the system trust store in
probe_api_models so these probes match what the rest of Hermes already does.
Union (not replace) keeps any corporate/internal CA that only lives in the
system store; the helper returns None on any error, so a probe that worked
before can never start failing.
Verified: with no SSL_CERT_FILE set, provider_model_ids("gmi") returns the
full live list (incl. zai-org/GLM-5.2-FP8) instead of the static fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
6bade10 to
83598a7
Compare
teknium1
left a comment
There was a problem hiding this comment.
Thanks for tracing the macOS certificate-store failure. The underlying gap is still present on current main, but this branch predates the catalog-request security refactor.
Problems
probe_api_models()now calls_urlopen_model_catalog_request()athermes_cli/models.py:3608, noturllib.request.urlopen()directly. The shared wrapper was introduced by6e75ba7fa066c1667cdc09db02c0c1dca8cd4671and its wire-level regression test ensuresprobe_api_models()does not leak arbitrary credential headers across redirects (tests/hermes_cli/test_urllib_security.py:409). A directurlopen(..., context=...)salvage would bypass that protection.- The same wrapper serves other live catalogs, including OpenRouter (
hermes_cli/models.py:1409) and GitHub Copilot (hermes_cli/models.py:2881), so the CA-context handling should be added at the shared wrapper layer. - The added
urllib.request.urlopenmock no longer covers the current probe path.
Suggested changes
- Thread a certifi-unioned SSL context through
open_credentialed_url()while preserving its installed-opener and redirect-sanitization behavior, then test through that current path.
Automated hermes-sweeper review.
|
|
||
| def fake_urlopen(req, timeout=None, context=None): | ||
| captured["context"] = context | ||
| raise OSError("stop after capturing") |
There was a problem hiding this comment.
Current main no longer invokes urllib.request.urlopen from probe_api_models; it calls _urlopen_model_catalog_request() (hermes_cli/models.py:3608) so the credential-safe redirect wrapper stays in force. Please retarget this test to the current wrapper path when salvaging the SSL-context change.
|
Closing as resolved on main — you diagnosed this correctly and first, and the fix that landed covers your scenario through a different seam. Two things happened since this PR was opened:
One deliberate difference from your approach: the landed fix uses certifi as a macOS fallback rather than unioning certifi on top of the system store everywhere — corporate/internal CAs in the system store stay authoritative on Linux. If you hit a concrete case the fallback misses (e.g. empty store on a non-macOS platform), please open an issue. Thanks for the clear root-cause writeup — sorry this one sat long enough to be overtaken. |
What does this PR do?
The live
/modelsprobe (probe_api_models) is a hand-rolledurllibcall that trusts Python's default OpenSSL trust store. On python.org macOS installs that never ran "Install Certificates.command" that store is empty, so the probe fails withCERTIFICATE_VERIFY_FAILEDeven though the server cert is valid - and the picker silently degrades to the static fallback list. That is exactly how a GMI partner concluded GLM-5.2 was "not supported" when GMI does serve it live (zai-org/GLM-5.2-FP8).curland Hermes's own chat path both work on the same machine, because the OpenAI SDK / httpx / requests already bundle and usecertifi. Only these hand-rolledurllibprobes were left relying on the (often empty) system store. This PR brings the straggler in line.Related Issue
Related to #51533 (the visibility/warning + GLM-5.2 catalog change). This PR is the root-cause fix; #51533 makes any residual fetch failure visible. Both touch
probe_api_models, so whichever merges second needs a one-line conflict resolution (the line just before the probe loop and theurlopen(...)call).Type of Change
Changes Made
hermes_cli/models.py_models_ssl_context(): buildsssl.create_default_context()(system trust store) and unionscertifi.where()on top of it. ReturnsNoneon any error so the probe falls back to urllib's exact previous behaviour.probe_api_models(): pass that context tourllib.request.urlopen(...).tests/hermes_cli/test_models.py: addTestModelsSSLContext(context has a non-empty trust store; the probe passes anSSLContexttourlopen).How to Test
pytest tests/hermes_cli/test_models.py -q-> passes.SSL_CERT_FILEset on a machine whose Python has an empty default trust store,provider_model_ids("gmi")now returns the full live list (incl.zai-org/GLM-5.2-FP8) instead of the static fallback. Before this change the same call returned only the static fallback.Why union (not replace)?
Using
create_default_context(cafile=certifi.where())would replace the trust store and drop any corporate/internal root that only lives in the system store. Starting fromcreate_default_context()and adding certifi viaload_verify_locationsmakes the trust set a superset of the old default - it can only add trust, never remove it - so no probe that verified before can start failing.Checklist
Code
fix(scope):)tests/hermes_cli/test_models.py(78 passed). Fullpytest tests/ -qhas pre-existing collection errors in unrelatedtests/acp/*modules, not caused by this PR.Documentation & Housekeeping
cli-config.yaml.example: N/A (no config keys changed)CONTRIBUTING.md/AGENTS.md: N/A (no architecture/workflow change)certifiis already a pinned dependency; helper returnsNoneon errorScreenshots / Logs
Before (empty default trust store):
provider_model_ids("gmi")-> 6 static models, no GLM-5.2.After:
provider_model_ids("gmi")-> 65 live models incl.zai-org/GLM-5.2-FP8, with noSSL_CERT_FILEset.