Skip to content

feat(model-promotion): teach model_family a github-models publisher rule #2855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stranske merged 1 commit into from
Jul 31, 2026
Merged

feat(model-promotion): teach model_family a github-models publisher rule #2855

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion tests/tools/test_prepare_model_promotion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,23 @@ def test_model_family_rules():
assert pmp.model_family("anthropic", "claude-sonnet-5") == "claude-sonnet"
assert pmp.model_family("openai", "gpt-5.6-terra") == "gpt-5"
assert pmp.model_family("openai", "gpt-5.4") == "gpt-5"
# Unknown provider -> exact id, so nothing is ever "same family" by accident.
# github-models ids are publisher-namespaced: keep the publisher and apply the
# OpenAI-style rule to the remainder, so a future openai/gpt-5.x is a
# same-family successor to the reviewed openai/gpt-5 selection.
assert pmp.model_family("github-models", "openai/gpt-5") == "openai/gpt-5"
assert pmp.model_family("github-models", "openai/gpt-5-mini") == "openai/gpt-5"
assert pmp.model_family("github-models", "openai/gpt-5.1") == "openai/gpt-5"
# A different major line stays a different family.
assert pmp.model_family("github-models", "openai/gpt-4.1") == "openai/gpt-4"
# A bare (unpublished) github-models id keeps its exact id, so the superseded
# codex-mini-latest is never "same family" as openai/gpt-5 by accident.
assert pmp.model_family("github-models", "codex-mini-latest") == "codex-mini-latest"
assert pmp.model_family("github-models", "codex-mini-latest") != pmp.model_family(
"github-models", "openai/gpt-5"
)
# Unknown provider -> exact id, so nothing is ever "same family" by accident.
# (azure-openai is NOT unknown: it normalizes to openai and uses that rule.)
assert pmp.model_family("mistral", "some-model-7") == "some-model-7"


def test_same_family_cheaper_pass_is_prepared():
Expand Down
32 changes: 24 additions & 8 deletions tools/prepare_model_promotion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,25 +51,41 @@ def _normalize_provider(provider: str) -> str:
return normalized


def _openai_style_family(model_id: str) -> str:
"""Collapse an OpenAI-style id to ``<head>-<major>`` (``gpt-5.6-terra`` -> ``gpt-5``)."""
parts = model_id.split("-")
head = parts[0] if parts else model_id # "gpt", then version token in parts[1]
if len(parts) >= 2:
major = parts[1].split(".")[0]
return f"{head}-{major}"
return model_id


def model_family(provider: str, model_id: str) -> str:
"""Return a conservative model-family key used to gate same-family promotions.

anthropic ``claude-opus-4-8`` -> ``claude-opus``; openai ``gpt-5.6-terra`` ->
``gpt-5``. For any other provider the family is the exact model id, so an
unrecognised provider never yields a same-family candidate (auto-preparation
stays off until a human teaches it the family rule).
``gpt-5``. github-models ids are publisher-namespaced, so the publisher is
preserved and the OpenAI-style rule is applied to the remainder:
``openai/gpt-5-mini`` -> ``openai/gpt-5``. A bare github-models id with no
Comment on lines +68 to +70

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the new GitHub Models family rule

When maint-86-model-promotion-prepare.yml evaluates a GitHub Models benchmark, this newly treats namespaced tier/version variants such as openai/gpt-5-mini and openai/gpt-5.1 as same-family and therefore eligible for a prepared promotion, but docs/MODEL_SELECTION_POLICY.md still describes family handling only with direct OpenAI and Anthropic examples. Update that canonical policy so operators reviewing the generated human-approval PR understand the expanded eligibility contract.

AGENTS.md reference: AGENTS.md:L60-L65

Useful? React with 👍 / 👎.

publisher (e.g. ``codex-mini-latest``) keeps its exact id, and for any other
provider the family is the exact model id, so an unrecognised provider never
yields a same-family candidate (auto-preparation stays off until a human
teaches it the family rule).
"""
provider = _normalize_provider(provider)
model_id = (model_id or "").strip()
if provider == "anthropic":
parts = model_id.split("-")
return "-".join(parts[:2]) if len(parts) >= 2 else model_id
if provider == "openai":
parts = model_id.split("-")
head = parts[0] if parts else model_id # "gpt", then version token in parts[1]
if len(parts) >= 2:
major = parts[1].split(".")[0]
return f"{head}-{major}"
return _openai_style_family(model_id)
if provider == "github-models" and "/" in model_id:
publisher, _, remainder = model_id.partition("/")
publisher = publisher.strip()
remainder = remainder.strip()
if publisher and remainder:
return f"{publisher}/{_openai_style_family(remainder)}"
return model_id
return model_id

Expand Down
Loading