Skip to content

Config-driven override for the Gemma picker-visibility filter - #62746

Open
rrschott-ai wants to merge 2 commits into
NousResearch:mainfrom
rrschott-ai:fix/gemma-picker-config-override
Open

rrschott-ai wants to merge 2 commits into
NousResearch:mainfrom
rrschott-ai:fix/gemma-picker-config-override

Conversation

@rrschott-ai

Copy link
Copy Markdown

Summary

  • _GOOGLE_HIDDEN_MODELS (agent/models_dev.py) hides several Gemma IDs from the interactive model/fallback picker by default — their low TPM quotas trip under agent-style traffic for most users, which is a reasonable default.
  • The only way to see a hidden model in the picker was editing the frozenset directly in source. A hermes update resets that file to upstream, silently wiping the edit — a user who deliberately wants one of these models has to reapply a source patch after every release, with no warning it happened.
  • Adds models.show_hidden_google_models to config.yaml (default [], no behavior change unless set): a list of model IDs to force-show despite the built-in filter. Config lives outside the package, so it survives updates.
  • Raw config entries (fallback_providers, a manually-typed model id) already bypassed this filter entirely and worked fine — this only changes what the picker UI shows, matching the intent of the original filter (a default for typical users, not a hard block).

Test plan

  • Both touched files parse clean (ast.parse)
  • Verified directly: with no override, a hidden Gemma id stays hidden; with models.show_hidden_google_models containing it, it's shown; an unrelated hidden id (not in the override list) stays hidden; a non-Google provider is unaffected; a config-load failure fails safe (stays hidden, no exception)
  • Confirmed end-to-end against a real local config.yaml with the override set — picker filter correctly un-hides only the configured ids
  • Reviewer: confirm models.show_hidden_google_models is the right home/name for this vs. e.g. folding it into the existing providers config dict

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the existing picker filter configurable; the underlying limitation is present on current main (agent/models_dev.py:567-597).

Problems

  • The new override un-hides every _GOOGLE_HIDDEN_MODELS member. That set also contains stale/retired Gemini slugs that are explicitly documented as 404ing at agent/models_dev.py:557-563, not only low-TPM Gemma IDs. Preserve the stale-model filter while allowing the intended Gemma subset.
  • Commit d5b3da2 also adds agent.skill_auto_patch, which is outside the PR summary and has no implementation consumer on current main (rg -n 'skill_auto_patch' . returned no matches). Please remove this unrelated config addition from the change.
  • No tests are included. Extend tests/hermes_cli/test_gemini_provider.py:309-356 to cover the configured Gemma override on both catalog paths and verify stale Gemini IDs remain hidden.

Suggested changes

  • Separate the low-TPM Gemma IDs from stale Google IDs, and consult the opt-in list only for the former.
  • Keep the change scoped to the picker override and add targeted regression coverage.

Automated hermes-sweeper review.

Comment thread agent/models_dev.py
def _should_hide_from_provider_catalog(provider: str, model_id: str) -> bool:
provider_lower = (provider or "").strip().lower()
model_lower = (model_id or "").strip().lower()
if provider_lower in {"gemini", "google"} and model_lower in _GOOGLE_HIDDEN_MODELS:
return True
return model_lower not in _google_show_hidden_overrides()
return False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_GOOGLE_HIDDEN_MODELS also contains stale/retired Gemini IDs that main documents as 404ing (agent/models_dev.py:557-563). This permits users to re-add known-invalid models; restrict this override to the low-TPM Gemma subset and keep stale IDs unconditionally hidden.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 11, 2026
@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard provider/gemini Google Gemini (AI Studio, Cloud Code) area/config Config system, migrations, profiles P3 Low — cosmetic, nice to have labels Jul 11, 2026
… filter

_GOOGLE_HIDDEN_MODELS (agent/models_dev.py) hides several Gemma IDs from
the interactive model/fallback picker by default - their low TPM quotas
trip under agent-style traffic for most users, a reasonable default. The
only way to see a hidden model in the picker was editing the frozenset
directly, which a `hermes update` silently wipes on the next release
(source is reset to upstream), forcing a user who deliberately wants one
of these models to reapply a source patch after every update.

Add agent.models (config.yaml) -> models.show_hidden_google_models: a
list of model IDs to force-show despite the built-in filter. Config
lives outside the package and survives updates. Raw config entries
(fallback_providers, a manually-typed model id) already bypassed this
filter entirely and are unaffected - this only changes picker
visibility. Empty by default, no behavior change for users who don't
set it.
…, add tests

Per hermes-sweeper's automated review on PR NousResearch#62746:

- Split _GOOGLE_HIDDEN_MODELS into _GOOGLE_LOW_TPM_MODELS (the low-TPM
  Gemma ids the override is meant for) and _GOOGLE_STALE_MODELS (retired
  Google slugs that 404 on current endpoints - a fact, not a posture, so
  never eligible for the config override). _should_hide_from_provider_catalog
  now only consults show_hidden_google_models for the low-TPM set; a stale
  slug in that config list is silently ignored rather than un-hiding a
  broken model.

- Dropped the agent.skill_auto_patch addition from hermes_cli/config.py -
  it leaked into this PR from an unrelated, separately-committed change in
  the same working file during branch prep. Not part of this PR's scope.

- Added three tests to tests/hermes_cli/test_gemini_provider.py covering
  both catalog paths reviewer asked for: the override un-hiding a
  configured Gemma id while leaving other Gemma ids hidden, a stale slug
  in the override list staying hidden regardless, and default (no config)
  behavior unchanged. All pass, plus the existing suite in this file
  (4 pre-existing failures are an unrelated missing concurrent_log_handler
  dependency in this environment, not touched by this change).
@rrschott-ai
rrschott-ai force-pushed the fix/gemma-picker-config-override branch from d5b3da2 to b9ef402 Compare July 11, 2026 19:53
@rrschott-ai

Copy link
Copy Markdown
Author

Addressed all three points, pushed as a new commit (b9ef402):

  1. Scoped the override correctly. Split _GOOGLE_HIDDEN_MODELS into _GOOGLE_LOW_TPM_MODELS (what the config override is meant for) and _GOOGLE_STALE_MODELS (retired slugs that 404 — a fact, not a posture). _should_hide_from_provider_catalog now only consults show_hidden_google_models for the low-TPM set; a stale slug in that config list is silently ignored rather than un-hiding a model that doesn't work. Verified directly: a stale id stays hidden even when explicitly listed in the override, while a low-TPM Gemma id in the same override list is correctly un-hidden.

  2. Removed the stray agent.skill_auto_patch hunk. You're right that it's unrelated — it leaked in from a diff-extraction mistake while splitting several concurrent local changes into separate PRs (that config key belongs to Add agent.skill_auto_patch config gate for skill_manage patches #58353, not this one). Confirmed zero occurrences of skill_auto_patch in the current diff.

  3. Added test coverage to tests/hermes_cli/test_gemini_provider.py (TestGeminiModelsDev): the override un-hiding a configured Gemma id while other Gemma ids stay hidden, a stale slug in the override list staying hidden regardless, and default (no config) behavior unchanged. All pass, along with the pre-existing suite in that file (4 unrelated pre-existing failures in TestGeminiAgentInit are a missing concurrent_log_handler dependency in my local environment, not touched by this change).

Thanks for catching the cross-contamination — verified with git diff that this PR is now scoped to just the picker-override change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have provider/gemini Google Gemini (AI Studio, Cloud Code) sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants