Skip to content

chore(model-registry): advance provisional verifier selections to current catalog #2852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stranske merged 2 commits into from
Jul 31, 2026
Merged

chore(model-registry): advance provisional verifier selections to current catalog #2852

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
125c35a
chore(model-registry): advance provisional verifier selections to cur…
stranske Jul 31, 2026
bf9490f
docs: name the advanced verifier models in README and WORKFLOWS.md
stranske Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Each stage completes, then dispatches auto-pilot again with the next step name.

### Verification Pipeline

After PR merge, applying a `verify:*` label (typically `verify:evaluate` via auto-pilot, or `verify:compare` for dual-model mode) triggers the verifier. In `compare` mode, two LLM providers (gpt-5.4 + claude-opus-4-6) independently evaluate the diff against acceptance criteria with unanimous PASS required. On CONCERNS or FAIL, maintainers or automation can apply the `verify:create-new-pr` label to trigger a 4-round LLM pipeline that generates a follow-up issue (analyze -> tasks -> acceptance criteria -> format).
After PR merge, applying a `verify:*` label (typically `verify:evaluate` via auto-pilot, or `verify:compare` for dual-model mode) triggers the verifier. In `compare` mode, two LLM providers (gpt-5.6-terra + claude-sonnet-5) independently evaluate the diff against acceptance criteria with unanimous PASS required. On CONCERNS or FAIL, maintainers or automation can apply the `verify:create-new-pr` label to trigger a 4-round LLM pipeline that generates a follow-up issue (analyze -> tasks -> acceptance criteria -> format).

**Live verifier and pipeline metrics** are surfaced through the weekly summary tracker — see [issue #2211](https://github.com/stranske/Workflows/issues/2211) (durable auto-bot tracker, posted Mondays at 06:00 UTC) and the LangSmith dashboard wired by [`maint-80-langsmith-metrics-dashboard.yml`](.github/workflows/maint-80-langsmith-metrics-dashboard.yml). The original Feb 2026 baseline (40-PR sample, first-fix 35%, avg chain depth 2.7) is preserved at [`docs/analysis/verify-compare-40pr-evaluation-feb-2026.md`](docs/analysis/verify-compare-40pr-evaluation-feb-2026.md) for historical comparison.

Expand Down
16 changes: 8 additions & 8 deletions config/model_eval_candidates.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"candidates": [
{
"provider": "anthropic",
"model_id": "claude-opus-4-6",
"model_id": "claude-sonnet-5",
"role": "incumbent"
},
{
Expand All @@ -12,37 +12,37 @@
},
{
"provider": "anthropic",
"model_id": "claude-opus-4-8",
"model_id": "claude-opus-4-6",
"role": "candidate"
},
{
"provider": "anthropic",
"model_id": "claude-sonnet-5",
"model_id": "claude-opus-4-8",
"role": "candidate"
},
{
"provider": "github-models",
"model_id": "codex-mini-latest",
"model_id": "openai/gpt-5",
"role": "incumbent"
},
{
"provider": "github-models",
"model_id": "openai/gpt-5",
"model_id": "codex-mini-latest",
"role": "candidate"
},
{
"provider": "openai",
"model_id": "gpt-5.4",
"model_id": "gpt-5.6-terra",
"role": "incumbent"
},
{
"provider": "openai",
"model_id": "gpt-5.6-sol",
"model_id": "gpt-5.4",
"role": "candidate"
},
{
"provider": "openai",
"model_id": "gpt-5.6-terra",
"model_id": "gpt-5.6-sol",
"role": "candidate"
}
]
Expand Down
49 changes: 45 additions & 4 deletions config/model_registry.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"schema_version": "2.0.0",
"as_of": "2026-07-10",
"review_by": "2026-07-24",
"review_by": "2026-08-30",
Comment on lines 3 to +4

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the catalog-facts freshness deadline

The catalog facts and baselines remain dated 2026-07-10, but advancing the top-level review_by to 2026-08-30 makes _review_date() prefer that deadline over as_of + 30 days. Consequently, maint-77 reports the registry as fresh through August 30 even though the catalog has not been queried for up to 51 days; for example, strict evaluation on 2026-08-15 returns no finding. Only the per-selection deadlines should move unless the catalog facts are actually refreshed.

Useful? React with 👍 / 👎.

"purpose": "Auditable model facts and auxiliary judge/evaluator selections. Coding-worker execution profiles remain in .github/agents/registry.yml::execution_profiles.",
"selection_policy": "config/model_selection_policy.json",
"catalog_baselines": {
Expand Down Expand Up @@ -235,6 +235,38 @@
}
],
"selections": [
{
"profile": "verifier-balanced",
"provider": "openai",
"model_id": "gpt-5.6-terra",
Comment on lines 237 to +241

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the consumer template with the new selections

This registry is consumer-managed through .github/sync-manifest.yml, but templates/consumer-repo/config/model_registry.json still selects gpt-5.4, claude-opus-4-6, and codex-mini-latest. A consumer initialized from the repository template therefore receives different verifier defaults until a later fleet sync overwrites the file; align the template in this change as required for consumer-affecting updates.

AGENTS.md reference: AGENTS.md:L63-L65

Useful? React with 👍 / 👎.

"status": "provisional",
"decided_at": "2026-07-31",
"review_by": "2026-08-30",
"evidence_ids": ["catalog-review-2026-07-10"],
"rationale": "Provisional selection advanced to the current catalog generation; positioning 'balanced' matches the verifier-balanced profile. This is NOT a benchmark-proven promotion: no passing workload-benchmark evidence exists for this model or for the superseded incumbent, so status stays provisional until the paired pilot runs. Human-approved by merging this PR."
Comment on lines +241 to +246

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep catalog-only candidates out of runtime selections

These replacements become the live verifier models immediately because select_model_for_profile() accepts provisional decisions, even though their only evidence is explicitly catalog-only. This contradicts docs/MODEL_SELECTION_POLICY.md:88-92 and 119-122, which require the provisional incumbents to remain the runtime baseline and prohibit changing a selection until paired workload evidence passes; human approval is an additional gate, not a substitute for that evidence. A regression in false-PASS behavior would therefore affect verifier decisions across every synced consumer before the pilot detects it.

Useful? React with 👍 / 👎.

},
{
"profile": "verifier-balanced",
"provider": "anthropic",
"model_id": "claude-sonnet-5",
"status": "provisional",
"decided_at": "2026-07-31",
"review_by": "2026-08-30",
"evidence_ids": ["catalog-review-2026-07-10"],
"rationale": "Provisional selection advanced to the current catalog generation; positioning 'balanced' matches the verifier-balanced profile. Cross-family relative to the superseded claude-opus incumbent, so it was human-initiated rather than auto-prepared. This is NOT a benchmark-proven promotion: no passing workload-benchmark evidence exists for this model or the incumbent, so status stays provisional until the paired pilot runs. Human-approved by merging this PR."
},
{
"profile": "verifier-balanced",
"provider": "github-models",
"model_id": "openai/gpt-5",
"status": "provisional",
"decided_at": "2026-07-31",
"review_by": "2026-08-30",
"evidence_ids": ["catalog-review-2026-07-10"],
"rationale": "Provisional catalog fallback advanced to a model confirmed present in the 2026-07-10 GitHub Models catalog baseline. This is NOT a benchmark-proven promotion: no passing workload-benchmark evidence exists for this model or the superseded codex-mini-latest, so status stays provisional until the paired pilot runs. Human-approved by merging this PR."
}
],
"selection_history": [
{
"profile": "verifier-balanced",
"provider": "openai",
Expand All @@ -243,7 +275,10 @@
"decided_at": "2026-07-10",
"review_by": "2026-07-24",
"evidence_ids": ["catalog-review-2026-07-10"],
"rationale": "Incumbent baseline retained until a paired repository benchmark proves a replacement meets every quality gate."
"rationale": "Incumbent baseline retained until a paired repository benchmark proves a replacement meets every quality gate.",
"superseded_at": "2026-07-31",
"superseded_by": "gpt-5.6-terra",
"supersede_reason": "Manual provisional refresh to the current catalog generation; not a benchmark-proven promotion."
},
{
"profile": "verifier-balanced",
Expand All @@ -253,7 +288,10 @@
"decided_at": "2026-07-10",
"review_by": "2026-07-24",
"evidence_ids": ["catalog-review-2026-07-10"],
"rationale": "Incumbent baseline retained until a paired repository benchmark proves a replacement meets every quality gate."
"rationale": "Incumbent baseline retained until a paired repository benchmark proves a replacement meets every quality gate.",
"superseded_at": "2026-07-31",
"superseded_by": "claude-sonnet-5",
"supersede_reason": "Manual provisional refresh to the current catalog generation; not a benchmark-proven promotion."
},
{
"profile": "verifier-balanced",
Expand All @@ -263,7 +301,10 @@
"decided_at": "2026-07-10",
"review_by": "2026-07-24",
"evidence_ids": ["catalog-review-2026-07-10"],
"rationale": "Incumbent fallback retained until catalog availability and paired repository evidence approve a replacement."
"rationale": "Incumbent fallback retained until catalog availability and paired repository evidence approve a replacement.",
"superseded_at": "2026-07-31",
"superseded_by": "openai/gpt-5",
"supersede_reason": "Manual provisional refresh to the current catalog generation; not a benchmark-proven promotion."
}
],
"evidence": [
Expand Down
2 changes: 1 addition & 1 deletion docs/ci/WORKFLOWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ Consumer default note: `agents-pr-meta-v4.yml` is a Workflows-repo service workf
* [`agents-verify-to-new-pr.yml`](../../.github/workflows/agents-verify-to-new-pr.yml) creates a follow-up issue from verification feedback, enforces the follow-up chain-depth limit, emits verifier follow-up ledger records, and kicks off a new PR when policy allows it.
* [`maint-auto-label-dep-prs.yml`](../../.github/workflows/maint-auto-label-dep-prs.yml) automatically applies the `agents:allow-change` label to dependency-bot PRs (Dependabot + Renovate) so dependency-only protected workflow version updates can use the guard's dependency-update lane.
* [`maint-auto-lock-deps.yml`](../../.github/workflows/maint-auto-lock-deps.yml) regenerates requirements.lock (re-running the command recorded in the lock header) when a dependency bot updates an input — a Workflows-local backstop to the fleet Renovate pip-compile manager.
* [`agents-verifier.yml`](../../.github/workflows/agents-verifier.yml) runs when `verify:*` labels are applied to a pull request (or via manual dispatch) to assemble acceptance/task context, execute LLM-based verifier modes, and post a verdict. `checkbox` mode uses the Codex CLI to drive checklist-style verification, while `evaluate` and `compare` run non-Codex verifier flows. In `compare` mode, two LLM providers (gpt-5.4 + claude-sonnet-4-6) evaluate independently with unanimous-PASS consensus. On CONCERNS or FAIL, maintainers or follow-up automation can apply `verify:create-issue` for an issue-only follow-up or `verify:create-new-pr` to trigger `agents-verify-to-new-pr.yml`, which uses a 4-round LLM pipeline to generate a follow-up issue when chain-depth policy allows it. The `verify:create-new-pr` path enforces the follow-up chain-depth cap, records policy/disposition metadata, and applies `needs-human` at the limit. See [`docs/analysis/verify-compare-40pr-evaluation-feb-2026.md`](../analysis/verify-compare-40pr-evaluation-feb-2026.md) for the Feb 2026 evaluation baseline.
* [`agents-verifier.yml`](../../.github/workflows/agents-verifier.yml) runs when `verify:*` labels are applied to a pull request (or via manual dispatch) to assemble acceptance/task context, execute LLM-based verifier modes, and post a verdict. `checkbox` mode uses the Codex CLI to drive checklist-style verification, while `evaluate` and `compare` run non-Codex verifier flows. In `compare` mode, two LLM providers (gpt-5.6-terra + claude-sonnet-5) evaluate independently with unanimous-PASS consensus. On CONCERNS or FAIL, maintainers or follow-up automation can apply `verify:create-issue` for an issue-only follow-up or `verify:create-new-pr` to trigger `agents-verify-to-new-pr.yml`, which uses a 4-round LLM pipeline to generate a follow-up issue when chain-depth policy allows it. The `verify:create-new-pr` path enforces the follow-up chain-depth cap, records policy/disposition metadata, and applies `needs-human` at the limit. See [`docs/analysis/verify-compare-40pr-evaluation-feb-2026.md`](../analysis/verify-compare-40pr-evaluation-feb-2026.md) for the Feb 2026 evaluation baseline.
* [`agents-weekly-metrics.yml`](../../.github/workflows/agents-weekly-metrics.yml) aggregates agent metrics (keepalive, autofix, verifier) on a weekly schedule and generates a markdown summary.
* [`agents-debug-issue-event.yml`](../../.github/workflows/agents-debug-issue-event.yml) dumps the GitHub event context for debugging issue triggers.
* [`autofix.yml`](../../.github/workflows/autofix.yml) detects formatting failures in agent PRs, applies automated fixes via ruff, and pushes autofix branches when the autofix label is present.
Expand Down
18 changes: 15 additions & 3 deletions tests/tools/test_langchain_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,18 @@
from tools import langchain_client, llm_registry


def _reviewed_model(provider: str) -> str:
"""Return the registry's reviewed selection for ``provider``.

Tests assert the resolver serves the *reviewed selection*, not a hardcoded
model id. Pinning literals here made every selection change — including an
auto-prepared maint-86 promotion PR — fail CI for no real defect.
"""
model = llm_registry.select_model_for_profile(provider=provider)
assert model, f"no reviewed selection for {provider}; registry/policy is misconfigured"
return model
Comment on lines +14 to +23

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Pin _reviewed_model to verifier-balanced.

The helper relies on select_model_for_profile’s default profile. If that default changes, these tests can validate a different profile while still passing. Pass profile="verifier-balanced" or use a named constant for this profile.

Proposed fix 
-    model = llm_registry.select_model_for_profile(provider=provider)
+    model = llm_registry.select_model_for_profile(
+        provider=provider,
+        profile="verifier-balanced",
+    )

As per path instructions, this Python test change must preserve correctness and test coverage.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def _reviewed_model(provider: str) -> str:
"""Return the registry's reviewed selection for ``provider``.
Tests assert the resolver serves the *reviewed selection*, not a hardcoded
model id. Pinning literals here made every selection change — including an
auto-prepared maint-86 promotion PR — fail CI for no real defect.
"""
model = llm_registry.select_model_for_profile(provider=provider)
assert model, f"no reviewed selection for {provider}; registry/policy is misconfigured"
return model
def _reviewed_model(provider: str) -> str:
"""Return the registry's reviewed selection for ``provider``.
Tests assert the resolver serves the *reviewed selection*, not a hardcoded
model id. Pinning literals here made every selection change — including an
auto-prepared maint-86 promotion PR — fail CI for no real defect.
"""
model = llm_registry.select_model_for_profile(
provider=provider,
profile="verifier-balanced",
)
assert model, f"no reviewed selection for {provider}; registry/policy is misconfigured"
return model
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/tools/test_langchain_client.py` around lines 14 - 23, Update the
_reviewed_model helper to explicitly request the verifier-balanced profile when
calling select_model_for_profile, using the existing named constant if one is
available. Keep the provider argument and assertion behavior unchanged so tests
remain tied to the intended reviewed profile.

Source: Path instructions



def _install_fake_langchain_openai(monkeypatch: pytest.MonkeyPatch):
fake_module = types.ModuleType("langchain_openai")

Expand Down Expand Up @@ -50,7 +62,7 @@ def test_build_chat_client_prefers_openai_slot(monkeypatch: pytest.MonkeyPatch)
assert isinstance(resolved.client, FakeChatOpenAI)
assert resolved.client.kwargs["api_key"] == "oa-token"
assert "base_url" not in resolved.client.kwargs
assert resolved.model == "gpt-5.4"
assert resolved.model == _reviewed_model(langchain_client.PROVIDER_OPENAI)


def test_build_chat_client_github_fallback(monkeypatch: pytest.MonkeyPatch) -> None:
Expand Down Expand Up @@ -85,7 +97,7 @@ def test_build_chat_client_anthropic_fallback(monkeypatch: pytest.MonkeyPatch) -
assert resolved.provider == langchain_client.PROVIDER_ANTHROPIC
assert isinstance(resolved.client, FakeChatAnthropic)
assert resolved.client.kwargs["anthropic_api_key"] == "claude-token"
assert resolved.model == "claude-opus-4-6"
assert resolved.model == _reviewed_model(langchain_client.PROVIDER_ANTHROPIC)


def test_build_chat_client_anthropic_without_openai_package(
Expand Down Expand Up @@ -592,7 +604,7 @@ def test_build_chat_clients_env_model_override(monkeypatch: pytest.MonkeyPatch)

assert [client.model for client in clients] == [
"gpt-4.1-mini",
"codex-mini-latest",
_reviewed_model(langchain_client.PROVIDER_GITHUB),
]
assert isinstance(clients[0].client, FakeChatOpenAI)

Expand Down
Loading