Repository navigation
feat(llm): model-registry freshness gate (maint-77) — stop old models ossifying as primary #2709
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
a1d6364
feat(llm): add model-registry freshness gate (maint-77)
stranske a824ea2
fix: harden model registry freshness gate
stranske 5a794ee
fix: satisfy model freshness gate
stranske 24aa403
fix: keep slot fallback when registry is partial
stranske 55750bd
docs: clarify model freshness PR trigger
stranske File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
133 changes: 133 additions & 0 deletions
133
.github/workflows/maint-77-model-registry-freshness.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| name: Maint 77 Model Registry Freshness | ||
|
|
||
| # Detects when the canonical LLM model configuration has gone stale so old models | ||
| # do not get stuck as the primary ones indefinitely. OFFLINE + deterministic: | ||
| # it only reads config/model_registry.json + config/llm_slots.json (no provider | ||
| # API calls, no secrets). On staleness it opens/refreshes a single tracking issue; | ||
| # a human reviews + refreshes the registry, which then propagates to consumers via | ||
| # the existing maint-68 sync (model_registry.json is copy-synced). | ||
| # | ||
| # This does NOT change model selection and does NOT touch the sync workflow. | ||
|
|
||
| on: | ||
| schedule: | ||
| # Weekly, Monday 05:20 UTC (after the 05:00 maint-68 sync window). | ||
| - cron: '20 5 * * 1' | ||
| workflow_dispatch: {} | ||
| pull_request: | ||
| paths: | ||
| - config/model_registry.json | ||
| - config/llm_slots.json | ||
| - tools/check_model_registry_freshness.py | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| freshness: | ||
| runs-on: ubuntu-latest | ||
| outputs: | ||
| rc: ${{ steps.gate.outputs.rc }} | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v7 | ||
|
github-advanced-security[bot] marked this conversation as resolved.
Fixed
|
||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.14' | ||
|
|
||
| # On PRs, fail the job on staleness so registry/slot changes are gated. | ||
| # On schedule/dispatch, never fail the run — open a tracking issue instead. | ||
| - name: Run freshness gate | ||
| id: gate | ||
| run: | | ||
| set +e | ||
| python3 tools/check_model_registry_freshness.py --json > freshness.json | ||
| rc=$? | ||
| echo "rc=$rc" >> "$GITHUB_OUTPUT" | ||
| cat freshness.json | ||
| { | ||
| echo '### Model registry freshness' | ||
| echo '```json' | ||
| cat freshness.json | ||
| echo '```' | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| if [ "$rc" = "2" ]; then | ||
| echo "::error::Freshness gate configuration error." | ||
| exit 2 | ||
| fi | ||
|
|
||
| - name: Fail PRs on staleness | ||
| if: github.event_name == 'pull_request' && steps.gate.outputs.rc == '1' | ||
| run: | | ||
| echo "::error::Model registry/slots are stale — see job summary." | ||
| exit 1 | ||
|
|
||
| tracking-issue: | ||
| if: github.event_name != 'pull_request' && needs.freshness.outputs.rc == '1' | ||
| needs: freshness | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| issues: write | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v7 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.14' | ||
|
|
||
| - name: Recreate freshness report | ||
| run: | | ||
| set +e | ||
| python3 tools/check_model_registry_freshness.py --json > freshness.json | ||
| rc=$? | ||
| cat freshness.json | ||
| if [ "$rc" = "2" ]; then | ||
| echo "::error::Freshness gate configuration error." | ||
| exit 2 | ||
| fi | ||
| if [ "$rc" != "1" ]; then | ||
| echo "::notice::Freshness finding cleared before issue update." | ||
| exit 0 | ||
| fi | ||
|
|
||
| - name: Open or refresh tracking issue (scheduled/dispatch only) | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| TITLE="🧭 LLM model registry needs review (freshness gate)" | ||
| BODY_FILE=issue_body.md | ||
| { | ||
| echo "The model-registry freshness gate (\`tools/check_model_registry_freshness.py\`, maint-77) flagged stale or dominated model configuration." | ||
| echo "" | ||
| echo "Refresh \`config/model_registry.json\` (and \`config/llm_slots.json\` if pins are dominated/blocked), bump \`review_by\`, and merge — the update propagates to consumers via maint-68 (model_registry.json is copy-synced)." | ||
| echo "" | ||
| echo "**Findings:**" | ||
| echo '```json' | ||
| cat freshness.json | ||
| echo '```' | ||
| echo "" | ||
| echo "_Run:_ ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" | ||
| } > "$BODY_FILE" | ||
| EXISTING=$(gh issue list --search "$TITLE in:title" --state open --json number --jq '.[0].number' 2>/dev/null) | ||
| if [ -n "$EXISTING" ]; then | ||
| gh issue comment "$EXISTING" --body-file "$BODY_FILE" | ||
| echo "::warning::Refreshed existing freshness issue #$EXISTING" | ||
| else | ||
| gh issue create --title "$TITLE" --body-file "$BODY_FILE" --label "automated" | ||
| echo "::warning::Opened model-registry freshness issue" | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| """Tests for the model-registry freshness gate (tools/check_model_registry_freshness.py).""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import datetime as dt | ||
| import json | ||
| from pathlib import Path | ||
|
|
||
| from tools import check_model_registry_freshness as gate | ||
|
|
||
| TODAY = dt.date(2026, 6, 28) | ||
|
|
||
|
|
||
| def _registry(**over): | ||
| base = { | ||
| "version": "1.0.0", | ||
| "review_by": "2026-12-31", | ||
| "models": [ | ||
| {"model_id": "gpt-5.4", "provider": "openai", "quality": {"T5": 0.97}}, | ||
| {"model_id": "gpt-5.1", "provider": "openai", "quality": {"T5": 0.93}}, | ||
| {"model_id": "claude-sonnet-4-6", "provider": "anthropic", "quality": {"T5": 0.95}}, | ||
| { | ||
| "model_id": "old-blocked", | ||
| "provider": "openai", | ||
| "quality": {"T5": 0.30}, | ||
| "blocked": True, | ||
| }, | ||
| ], | ||
| } | ||
| base.update(over) | ||
| return base | ||
|
|
||
|
|
||
| def _slots(*pairs): | ||
| return { | ||
| "slots": [ | ||
| {"name": f"slot{i+1}", "provider": p, "model": m} for i, (p, m) in enumerate(pairs) | ||
| ] | ||
| } | ||
|
|
||
|
|
||
| def _kinds(findings): | ||
| return sorted(f["kind"] for f in findings) | ||
|
|
||
|
|
||
| def test_fresh_registry_has_no_findings(): | ||
| findings = gate.evaluate(_registry(), _slots(("openai", "gpt-5.4")), today=TODAY) | ||
| assert findings == [] | ||
|
|
||
|
|
||
| def test_review_overdue_explicit_review_by(): | ||
| reg = _registry(review_by="2026-05-01") | ||
| findings = gate.evaluate(reg, _slots(("openai", "gpt-5.4")), today=TODAY) | ||
| assert _kinds(findings) == ["review_overdue"] | ||
| assert "overdue" in findings[0]["detail"] | ||
|
|
||
|
|
||
| def test_review_overdue_derived_from_last_updated(): | ||
| reg = _registry() | ||
| reg.pop("review_by") | ||
| reg["last_updated"] = "2026-04-14" # +60d -> 2026-06-13 < today | ||
| findings = gate.evaluate(reg, _slots(("openai", "gpt-5.4")), today=TODAY, max_age_days=60) | ||
| assert "review_overdue" in _kinds(findings) | ||
|
|
||
|
|
||
| def test_review_not_overdue_within_window(): | ||
| reg = _registry() | ||
| reg.pop("review_by") | ||
| reg["last_updated"] = "2026-06-20" # +60d in the future | ||
| findings = gate.evaluate(reg, _slots(("openai", "gpt-5.4")), today=TODAY, max_age_days=60) | ||
| assert "review_overdue" not in _kinds(findings) | ||
|
|
||
|
|
||
| def test_blocked_pin(): | ||
| findings = gate.evaluate(_registry(), _slots(("openai", "old-blocked")), today=TODAY) | ||
| assert _kinds(findings) == ["blocked_pin"] | ||
|
|
||
|
|
||
| def test_unknown_pin(): | ||
| findings = gate.evaluate(_registry(), _slots(("openai", "gpt-9-imaginary")), today=TODAY) | ||
| assert _kinds(findings) == ["unknown_pin"] | ||
|
|
||
|
|
||
| def test_dominated_pin_flags_better_same_provider_model(): | ||
| # Pin sonnet-4-6 (0.95) while registry also has a higher claude model. | ||
| reg = _registry() | ||
| reg["models"].append( | ||
| {"model_id": "claude-opus-4-6", "provider": "anthropic", "quality": {"T5": 0.98}} | ||
| ) | ||
| findings = gate.evaluate(reg, _slots(("anthropic", "claude-sonnet-4-6")), today=TODAY) | ||
| assert _kinds(findings) == ["dominated_pin"] | ||
| assert "claude-opus-4-6" in findings[0]["detail"] | ||
|
|
||
|
|
||
| def test_dominated_pin_uses_slot_quality_tier(): | ||
| reg = _registry() | ||
| reg["models"] = [ | ||
| { | ||
| "model_id": "primary-for-t4", | ||
| "provider": "openai", | ||
| "quality": {"T4": 0.95, "T5": 0.80}, | ||
| }, | ||
| { | ||
| "model_id": "better-t5-only", | ||
| "provider": "openai", | ||
| "quality": {"T4": 0.90, "T5": 0.99}, | ||
| }, | ||
| ] | ||
| slots = { | ||
| "slots": [ | ||
| { | ||
| "name": "slot1", | ||
| "provider": "openai", | ||
| "model": "primary-for-t4", | ||
| "quality_tier": "T4", | ||
| } | ||
| ] | ||
| } | ||
| findings = gate.evaluate(reg, slots, today=TODAY) | ||
| assert findings == [] | ||
|
|
||
|
|
||
| def test_tier_derived_slot_without_model_is_not_flagged(): | ||
| # A slot with no pinned model derives from the registry at runtime -> non-ossifying. | ||
| slots = {"slots": [{"name": "slot1", "provider": "anthropic", "quality_tier": "T5"}]} | ||
| findings = gate.evaluate(_registry(), slots, today=TODAY) | ||
| assert findings == [] | ||
|
|
||
|
|
||
| def test_real_repo_files_parse_and_run(tmp_path): | ||
| # The shipped config must at least load and evaluate without raising. | ||
| root = Path(__file__).resolve().parent.parent | ||
| reg = json.loads((root / "config" / "model_registry.json").read_text()) | ||
| slots = json.loads((root / "config" / "llm_slots.json").read_text()) | ||
| findings = gate.evaluate(reg, slots, today=TODAY) | ||
| assert isinstance(findings, list) | ||
|
|
||
|
|
||
| def test_main_exit_codes(tmp_path): | ||
| reg = tmp_path / "reg.json" | ||
| slots = tmp_path / "slots.json" | ||
| reg.write_text(json.dumps(_registry(review_by="2026-05-01"))) | ||
| slots.write_text(json.dumps(_slots(("openai", "gpt-5.4")))) | ||
| rc = gate.main( | ||
| ["--registry", str(reg), "--slots", str(slots), "--today", "2026-06-28", "--json"] | ||
| ) | ||
| assert rc == 1 # overdue | ||
| reg.write_text(json.dumps(_registry(review_by="2026-12-31"))) | ||
| rc = gate.main(["--registry", str(reg), "--slots", str(slots), "--today", "2026-06-28"]) | ||
| assert rc == 0 | ||
| rc = gate.main(["--registry", str(reg), "--slots", str(slots), "--today", "not-a-date"]) | ||
| assert rc == 2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.