Skip to content

fix(models): surface live model-fetch failures instead of silent fallback - #51533

Open
isaachuangGMICLOUD wants to merge 1 commit into
NousResearch:mainfrom
isaachuangGMICLOUD:fix/surface-live-model-fetch-failure
Open

fix(models): surface live model-fetch failures instead of silent fallback#51533
isaachuangGMICLOUD wants to merge 1 commit into
NousResearch:mainfrom
isaachuangGMICLOUD:fix/surface-live-model-fetch-failure

Conversation

@isaachuangGMICLOUD

Copy link
Copy Markdown
Contributor

What does this PR do?

A GMI Cloud partner reported that GLM-5.2 was "not in the supporting list". GMI does serve it live (zai-org/GLM-5.2-FP8, confirmed against GET /v1/models). The real cause: on a machine whose Python has no CA bundle, the live /models probe throws SSL: CERTIFICATE_VERIFY_FAILED, and probe_api_models swallowed it with a bare except: continue. The picker then silently degraded to the 6-item static fallback list, which reads to users as "this provider only supports a handful of old models".

So a trivial local cert issue (or any network/auth hiccup) masquerades as "Hermes doesn't support model X", and it will recur for every provider/user that hits a transport problem. This PR makes the failure visible instead of silent. It does not touch cert logic or the GMI live-fetch special-case behaviour.

Related Issue

N/A - reported via Slack by a GMI Cloud partner; no GitHub issue.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Security fix
  • Documentation update
  • Tests (adding or improving test coverage)
  • Refactor (no behavior change)
  • New skill (bundled or hub)

Changes Made

  • hermes_cli/models.py
    • probe_api_models(): record the failure reason and return it as a new error field (purely additive; existing .get("models") callers unaffected).
    • provider_model_ids(): when a credentialed live fetch comes back empty, re-probe once. Use the result if the endpoint recovered, otherwise emit a one-time stderr warning naming the reason. TLS-cert failures additionally print the SSL_CERT_FILE / Install Certificates.command fix. HTTP 404/405 (no-catalog providers) are not treated as a degradation.
    • Added _warn_live_model_fetch_failed() helper plus a process-lifetime dedup set.
    • _PROVIDER_MODELS["gmi"]: lead with zai-org/GLM-5.2-FP8.
  • plugins/model-providers/gmi/__init__.py: add zai-org/GLM-5.2-FP8 to fallback_models.
  • website/docs/integrations/providers.md and website/i18n/zh-Hans/.../integrations/providers.md: update GMI examples to zai-org/GLM-5.2-FP8.
  • tests/hermes_cli/test_models.py: add TestLiveModelFetchDiagnostics.

How to Test

  1. Reproduce the silent degradation by simulating a broken-CA box: patch urllib.request.urlopen to raise ssl.SSLCertVerificationError, then call probe_api_models("k", "https://api.gmi-serving.com/v1"). It now returns {'models': None, ..., 'error': 'SSLCertVerificationError: certificate verify failed'} instead of dropping the reason.
  2. With a real GMI_API_KEY and a working CA bundle, provider_model_ids("gmi") returns the full live list including zai-org/GLM-5.2-FP8; with a broken CA bundle it now prints a warning plus the SSL_CERT_FILE fix instead of silently showing 6 models.
  3. pytest tests/hermes_cli/test_models.py -q -> 81 passed.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix
  • I've run the relevant suites: tests/hermes_cli/test_models.py (81 passed) plus the provider/picker/merge suites (210 passed). Full pytest tests/ -q has pre-existing collection errors in unrelated tests/acp/* modules, not caused by this PR.
  • I've added tests for my changes
  • I've tested on my platform: macOS 15.5 (Darwin 24.5.0)

Documentation & Housekeeping

  • I've updated relevant documentation (website/docs/..., docstrings)
  • cli-config.yaml.example: N/A (no config keys changed)
  • CONTRIBUTING.md / AGENTS.md: N/A (no architecture/workflow change)
  • I've considered cross-platform impact: warning uses stderr plus a POSIX cert hint; behaviour is unchanged on Windows/Linux (the macOS-specific hint text is harmless elsewhere)
  • I've updated tool descriptions/schemas: N/A (no tool behavior changed)

Screenshots / Logs

Warning emitted when the live fetch fails on a broken-CA machine:

WARNING: gmi: could not fetch the live model list (SSLCertVerificationError: certificate verify failed: unable to get local issuer certificate); showing the built-in fallback list instead.
   Looks like a TLS certificate problem (Python has no CA bundle). Fix with:
     export SSL_CERT_FILE="$(python3 -m certifi)"
   (or run the 'Install Certificates.command' shipped with python.org Python).

…back

A failed `/models` probe (e.g. an SSL certificate error on a machine whose
Python has no CA bundle) was swallowed silently and the picker degraded to
the static fallback list. To users this reads as "this provider only
supports a handful of old models" — which is exactly how a GMI partner
reported that GLM-5.2 was "not in the supporting list" (it is served live
as `zai-org/GLM-5.2-FP8`; their box just couldn't verify GMI's TLS cert).

- probe_api_models now records the failure reason and returns it as `error`.
- When a credentialed live fetch comes back empty, re-probe once: use the
  result if the endpoint recovered, otherwise emit a one-time stderr warning
  that names the reason and gives the macOS SSL_CERT_FILE fix. HTTP 404/405
  (no-catalog providers) are not treated as a degradation.
- Refresh the GMI static fallback list + docs to lead with GLM-5.2-FP8 so
  the no-credentials/API-down path isn't stale either.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists labels Jun 23, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing a real silent-degradation path. Current origin/main still swallows every /models exception in hermes_cli/models.py:3617-3628, then reaches profile fallbacks at hermes_cli/models.py:2527-2529.

Problems

  • The added test patches urllib.request.urlopen, but current probe_api_models() calls _urlopen_model_catalog_request() at hermes_cli/models.py:3618 after 6e75ba7fa. Update the test to intercept the current catalog-request seam during salvage.
  • The proposed diagnostic retry should preserve the profile request context. ProviderProfile.fetch_models() adds default_headers at providers/base.py:209-210; a direct probe_api_models() retry can diagnose a different request.
  • The exact GLM-5.2-FP8 fallback assertion is a catalog snapshot rather than the behavior contract required by AGENTS.md.

Suggested changes

  • Salvage the error propagation onto the current catalog helper and test it there.
  • Pass the same profile headers into the diagnostic probe, or centralize the failure result in the profile fetch path.
  • Test warning/fallback behavior rather than a particular catalog entry.

Automated hermes-sweeper review.


def test_probe_surfaces_error_reason(self):
import ssl
err = ssl.SSLCertVerificationError(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current main routes this request through _urlopen_model_catalog_request() (hermes_cli/models.py:3618, via 6e75ba7fa), so patch that seam when salvaging; patching urllib.request.urlopen no longer directly exercises the catalog probe.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants