fix(models): surface live model-fetch failures instead of silent fallback - #51533
Open
isaachuangGMICLOUD wants to merge 1 commit into
Open
Conversation
…back A failed `/models` probe (e.g. an SSL certificate error on a machine whose Python has no CA bundle) was swallowed silently and the picker degraded to the static fallback list. To users this reads as "this provider only supports a handful of old models" — which is exactly how a GMI partner reported that GLM-5.2 was "not in the supporting list" (it is served live as `zai-org/GLM-5.2-FP8`; their box just couldn't verify GMI's TLS cert). - probe_api_models now records the failure reason and returns it as `error`. - When a credentialed live fetch comes back empty, re-probe once: use the result if the endpoint recovered, otherwise emit a one-time stderr warning that names the reason and gives the macOS SSL_CERT_FILE fix. HTTP 404/405 (no-catalog providers) are not treated as a degradation. - Refresh the GMI static fallback list + docs to lead with GLM-5.2-FP8 so the no-credentials/API-down path isn't stale either. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
19 tasks
teknium1
reviewed
Jul 15, 2026
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for addressing a real silent-degradation path. Current origin/main still swallows every /models exception in hermes_cli/models.py:3617-3628, then reaches profile fallbacks at hermes_cli/models.py:2527-2529.
Problems
- The added test patches
urllib.request.urlopen, but currentprobe_api_models()calls_urlopen_model_catalog_request()athermes_cli/models.py:3618after6e75ba7fa. Update the test to intercept the current catalog-request seam during salvage. - The proposed diagnostic retry should preserve the profile request context.
ProviderProfile.fetch_models()addsdefault_headersatproviders/base.py:209-210; a directprobe_api_models()retry can diagnose a different request. - The exact
GLM-5.2-FP8fallback assertion is a catalog snapshot rather than the behavior contract required byAGENTS.md.
Suggested changes
- Salvage the error propagation onto the current catalog helper and test it there.
- Pass the same profile headers into the diagnostic probe, or centralize the failure result in the profile fetch path.
- Test warning/fallback behavior rather than a particular catalog entry.
Automated hermes-sweeper review.
|
|
||
| def test_probe_surfaces_error_reason(self): | ||
| import ssl | ||
| err = ssl.SSLCertVerificationError( |
Contributor
There was a problem hiding this comment.
Current main routes this request through _urlopen_model_catalog_request() (hermes_cli/models.py:3618, via 6e75ba7fa), so patch that seam when salvaging; patching urllib.request.urlopen no longer directly exercises the catalog probe.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
A GMI Cloud partner reported that GLM-5.2 was "not in the supporting list". GMI does serve it live (
zai-org/GLM-5.2-FP8, confirmed againstGET /v1/models). The real cause: on a machine whose Python has no CA bundle, the live/modelsprobe throwsSSL: CERTIFICATE_VERIFY_FAILED, andprobe_api_modelsswallowed it with a bareexcept: continue. The picker then silently degraded to the 6-item static fallback list, which reads to users as "this provider only supports a handful of old models".So a trivial local cert issue (or any network/auth hiccup) masquerades as "Hermes doesn't support model X", and it will recur for every provider/user that hits a transport problem. This PR makes the failure visible instead of silent. It does not touch cert logic or the GMI live-fetch special-case behaviour.
Related Issue
N/A - reported via Slack by a GMI Cloud partner; no GitHub issue.
Type of Change
Changes Made
hermes_cli/models.pyprobe_api_models(): record the failure reason and return it as a newerrorfield (purely additive; existing.get("models")callers unaffected).provider_model_ids(): when a credentialed live fetch comes back empty, re-probe once. Use the result if the endpoint recovered, otherwise emit a one-time stderr warning naming the reason. TLS-cert failures additionally print theSSL_CERT_FILE/Install Certificates.commandfix. HTTP 404/405 (no-catalog providers) are not treated as a degradation._warn_live_model_fetch_failed()helper plus a process-lifetime dedup set._PROVIDER_MODELS["gmi"]: lead withzai-org/GLM-5.2-FP8.plugins/model-providers/gmi/__init__.py: addzai-org/GLM-5.2-FP8tofallback_models.website/docs/integrations/providers.mdandwebsite/i18n/zh-Hans/.../integrations/providers.md: update GMI examples tozai-org/GLM-5.2-FP8.tests/hermes_cli/test_models.py: addTestLiveModelFetchDiagnostics.How to Test
urllib.request.urlopento raisessl.SSLCertVerificationError, then callprobe_api_models("k", "https://api.gmi-serving.com/v1"). It now returns{'models': None, ..., 'error': 'SSLCertVerificationError: certificate verify failed'}instead of dropping the reason.GMI_API_KEYand a working CA bundle,provider_model_ids("gmi")returns the full live list includingzai-org/GLM-5.2-FP8; with a broken CA bundle it now prints a warning plus theSSL_CERT_FILEfix instead of silently showing 6 models.pytest tests/hermes_cli/test_models.py -q-> 81 passed.Checklist
Code
fix(scope):)tests/hermes_cli/test_models.py(81 passed) plus the provider/picker/merge suites (210 passed). Fullpytest tests/ -qhas pre-existing collection errors in unrelatedtests/acp/*modules, not caused by this PR.Documentation & Housekeeping
website/docs/..., docstrings)cli-config.yaml.example: N/A (no config keys changed)CONTRIBUTING.md/AGENTS.md: N/A (no architecture/workflow change)Screenshots / Logs
Warning emitted when the live fetch fails on a broken-CA machine: