maint-86: prepare (not auto-apply) verifier model promotions (#2819 move 3)

.github/workflows/maint-86-model-promotion-prepare.yml

@@ -0,0 +1,88 @@
+name: Maint 86 Model Promotion Prepare
+
+# Turns a passing benchmark into a *prepared* verifier-model promotion
+# (stranske/Workflows#2819, move 3). It evaluates a benchmark artifact, and if a
+# same-family, non-inferior, cost<= candidate qualifies (or an active model
+# breached its gates), it opens a PR that changes the registry selection. A human
+# merges the PR — that IS the approval, so `human_approval_required` stays true.
+#
+# Dispatch-only for now: it must be pointed at a real benchmark evidence file, and
+# a trustworthy one does not exist until the corpus reaches the approval minimum
+# (grown by maint-79) and the pilot runs on it. The schedule trigger + the
+# pilot->benchmark bridge are the final wiring step, deliberately deferred so this
+# cannot fabricate a promotion from thin data.
+
+on:
+  workflow_dispatch:
+    inputs:
+      benchmark_path:
+        description: "Path to an evaluate_model_benchmark.py output JSON in the repo."
+        required: true
+      mode:
+        description: "promote | rollback | auto"
+        type: choice
+        options: [auto, promote, rollback]
+        default: auto
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: false
+
+jobs:
+  prepare:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.14"
+      - name: Prepare promotion/rollback
+        id: prep
+        env:
+          BENCHMARK_PATH: ${{ inputs.benchmark_path }}
+          MODE: ${{ inputs.mode }}
+        run: |
+          set +e
+          python -m tools.prepare_model_promotion \
+            --benchmark "$BENCHMARK_PATH" \
+            --mode "$MODE" \
+            --write config/model_registry.json | tee prepare.log
+          rc=$?
+          echo "rc=$rc" >> "$GITHUB_OUTPUT"
+          {
+            echo '## Model promotion prepare'
+            echo '```'
+            cat prepare.log
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+          if [ "$rc" = "2" ]; then
+            echo "::error::prepare_model_promotion input error."
+            exit 2
+          fi
+          exit 0
+      - name: Open promotion PR (human merges to approve)
+        if: steps.prep.outputs.rc == '10'
+        uses: peter-evans/create-pull-request@v8
+        with:
+          token: ${{ secrets.OWNER_PR_PAT }}
+          branch: model-promotion/prepared
+          base: main
+          title: "model: prepared verifier selection change (review + merge to approve)"
+          commit-message: "model: apply prepared verifier selection change (#2819 move 3)"
+          labels: "model-selection,needs-human-approval"
+          delete-branch: true
+          body: |
+            Automated **preparation** of a verifier model selection change
+            (stranske/Workflows#2819 move 3). See the run summary for the exact
+            promotion/rollback and its evidence.
+
+            This PR is intentionally **not** auto-merged: merging it is the human
+            approval the policy requires (`human_approval_required=true`). Review the
+            attached benchmark evidence before merging. To reject, close the PR — the
+            live selection is unchanged until merge.

docs/MODEL_SELECTION_POLICY.md

@@ -91,6 +91,19 @@ assembled and run; catalog discovery can add candidates but cannot change a
 selection. A replacement requires paired workload evidence that passes every
 quality gate and an explicit approval update.
 
+### Prepared promotions and rollbacks
+
+`tools/prepare_model_promotion.py` (run by `maint-86`) can *prepare* a selection
+change from a passing benchmark, but never applies one on its own. It only
+prepares a candidate that is the **same family** as the incumbent (e.g. openai
+`gpt-5.x`, anthropic `claude-<line>`), **passed every quality gate** (including
+paired non-inferiority), and costs **≤** the incumbent per accepted review.
+Cross-family swaps are never auto-prepared. It writes the registry mutation
+(recording the prior selection in `selection_history`) and opens a PR; merging
+that PR is the human approval this policy requires — `human_approval_required`
+stays true. The inverse path prepares a rollback to the prior selection when the
+active model shows a failed workload-benchmark (a quality-gate breach).
+
 ## Catalog Discovery
 
 Run:

docs/ci/WORKFLOWS.md

@@ -208,6 +208,7 @@ Scheduled health jobs keep the automation ecosystem aligned:
 * [`maint-77-model-registry-freshness.yml`](../../.github/workflows/maint-77-model-registry-freshness.yml) checks the canonical LLM registry for overdue or unproved decisions, invalid lifecycle/evidence references, and profile/slot drift. Scheduled and manual runs also perform credential-gated provider catalog discovery; catalog additions become review candidates and never auto-promote (scheduled weekly, manual dispatch, PR gate for registry/slot/policy/checker changes).
 * [`maint-78-model-evaluation-pilot.yml`](../../.github/workflows/maint-78-model-evaluation-pilot.yml) runs the frozen 30-case verifier corpus against explicit incumbent and candidate models using repository credentials, then uploads artifact-only paired results. The pilot narrows candidates; it cannot approve or migrate a model.
 * [`maint-79-verifier-corpus-harvest.yml`](../../.github/workflows/maint-79-verifier-corpus-harvest.yml) grows the frozen verifier evaluation corpus from realized PR outcomes (stable merge, revert, resolved follow-up). High-confidence cases auto-promote via an auto-merging PR; ambiguous cases land in an auto-expiring staging file. It only narrows/expands the corpus; it cannot approve or migrate a model.
+* [`maint-86-model-promotion-prepare.yml`](../../.github/workflows/maint-86-model-promotion-prepare.yml) turns a passing benchmark into a *prepared* verifier-model selection change: a same-family, non-inferior, cost≤ promotion (or a gate-breach rollback) is written to the registry and opened as a PR. The PR is **not** auto-merged — a human merges it to approve, so `human_approval_required` stays true. Dispatch-only until a trustworthy approval benchmark exists.
 * [`maint-80-langsmith-metrics-dashboard.yml`](../../.github/workflows/maint-80-langsmith-metrics-dashboard.yml) generates weekly LangSmith trace coverage dashboard - downloads metrics from autopilot artifacts, computes coverage, creates issue report (scheduled Monday 9AM UTC, manual dispatch).
 * [`maint-81-langsmith-fleet-conformance.yml`](../../.github/workflows/maint-81-langsmith-fleet-conformance.yml) validates fleet artifact coverage against `config/langsmith_fleet_registry.json` and reports missing/stale/invalid records (scheduled Monday 9:30AM UTC, manual dispatch with optional enforcement).
 * [`maint-82-sync-dependency-campaign.yml`](../../.github/workflows/maint-82-sync-dependency-campaign.yml) refreshes a GitHub-visible sync/dependency campaign issue so local Codex only claims queued bot-review work when remote discovery finds active review threads.

docs/ci/WORKFLOW_SYSTEM.md

@@ -747,6 +747,7 @@ Keep this table handy when you are triaging automation: it confirms which workfl
 | **Maint 77 Model Registry Freshness** (`maint-77-model-registry-freshness.yml`, maintenance bucket) | `schedule` (Mondays 05:20 UTC), `workflow_dispatch`, `pull_request` (registry/slot/policy/checker paths) | Validates explicit model decisions, evidence, lifecycle, and profile slots offline. Scheduled/manual runs add credential-gated provider-catalog drift and refresh one review issue; catalog changes never auto-select a model. | ⚪ Scheduled/manual + PR gate | [Model registry freshness runs](https://github.com/stranske/Workflows/actions/workflows/maint-77-model-registry-freshness.yml) |
 | **Maint 78 Model Evaluation Pilot** (`maint-78-model-evaluation-pilot.yml`, maintenance bucket) | `workflow_dispatch` | Runs the frozen 30-case verifier corpus against explicit incumbent and candidate models with repository credentials. Uploads artifact-only paired verdict, schema, and latency evidence; never changes a selection. | ⚪ Manual evaluation | [Model evaluation pilot runs](https://github.com/stranske/Workflows/actions/workflows/maint-78-model-evaluation-pilot.yml) |
 | **Maint 79 Verifier Corpus Harvest** (`maint-79-verifier-corpus-harvest.yml`, maintenance bucket) | `schedule` (weekly), `workflow_dispatch` | Grows the frozen verifier evaluation corpus from realized PR outcomes. High-confidence cases auto-promote via an auto-merging PR; ambiguous cases stage FYI-only and auto-expire. Never changes a selection. | 🟢 Scheduled | [Verifier corpus harvest runs](https://github.com/stranske/Workflows/actions/workflows/maint-79-verifier-corpus-harvest.yml) |
+| **Maint 86 Model Promotion Prepare** (`maint-86-model-promotion-prepare.yml`, maintenance bucket) | `workflow_dispatch` | Evaluates a benchmark and, for a same-family non-inferior cost≤ candidate (or a gate-breach rollback), opens a registry-change PR. Not auto-merged — a human merges to approve. Dispatch-only until a trustworthy approval benchmark exists. | ⚪ Manual/prepared | [Model promotion prepare runs](https://github.com/stranske/Workflows/actions/workflows/maint-86-model-promotion-prepare.yml) |
 | **LangSmith Metrics Dashboard** (`maint-80-langsmith-metrics-dashboard.yml`, maintenance bucket) | `workflow_dispatch`, `schedule` (Mondays 09:00 UTC) | Generates weekly LangSmith trace coverage dashboard by downloading metrics from autopilot artifacts, computing coverage, and creating issue reports. | ⚪ Manual/scheduled | [LangSmith metrics runs](https://github.com/stranske/Workflows/actions/workflows/maint-80-langsmith-metrics-dashboard.yml) |
 | **LangSmith Fleet Conformance** (`maint-81-langsmith-fleet-conformance.yml`, maintenance bucket) | `workflow_dispatch`, `schedule` (Mondays 09:30 UTC) | Validates LangSmith fleet artifact coverage against `config/langsmith_fleet_registry.json`, emits markdown/JSON reports, and can optionally enforce non-valid rows. | ⚪ Manual/scheduled | [LangSmith fleet conformance runs](https://github.com/stranske/Workflows/actions/workflows/maint-81-langsmith-fleet-conformance.yml) |
 | **Sync/Dependency Campaign** (`maint-82-sync-dependency-campaign.yml`, maintenance bucket) | `schedule`, `workflow_dispatch`, `repository_dispatch` | Refreshes a GitHub-visible campaign issue for sync-generated and dependency-bot PRs with active bot review threads so local Codex only claims queued work when remote discovery finds it. | ⚪ Scheduled/manual | [Sync/Dependency campaign runs](https://github.com/stranske/Workflows/actions/workflows/maint-82-sync-dependency-campaign.yml) |

tests/tools/test_prepare_model_promotion.py

@@ -0,0 +1,222 @@
+"""Tests for tools/prepare_model_promotion.py (#2819 move 3)."""
+
+from __future__ import annotations
+
+import datetime as dt
+import json
+
+from tools import prepare_model_promotion as pmp
+
+TODAY = dt.date(2026, 8, 1)
+
+
+def _registry(model_id="claude-opus-4-6"):
+    return {
+        "selections": [
+            {
+                "profile": "verifier-balanced",
+                "provider": "anthropic",
+                "model_id": model_id,
+                "status": "provisional",
+                "decided_at": "2026-07-10",
+                "review_by": "2026-08-09",
+                "evidence_ids": ["catalog-1"],
+            }
+        ]
+    }
+
+
+def _result(model_id, provider, *, status, cost, latency=100.0):
+    return {
+        "provider": provider,
+        "model_id": model_id,
+        "status": status,
+        "gate_results": {"paired_success_noninferiority": status == "passed"},
+        "metrics": {"cost_per_accepted_review_usd": cost, "p95_latency_ms": latency},
+    }
+
+
+def _report(results, *, baseline="claude-opus-4-6"):
+    return {
+        "baseline_model_id": baseline,
+        "results": results,
+        "registry_evidence": [
+            {
+                "evidence_id": f"bench-2026-08:{r['provider']}:{r['model_id']}",
+                "provider": r["provider"],
+                "model_id": r["model_id"],
+                "kind": "workload-benchmark",
+                "status": r["status"],
+            }
+            for r in results
+        ],
+    }
+
+
+def test_model_family_rules():
+    assert pmp.model_family("anthropic", "claude-opus-4-8") == "claude-opus"
+    assert pmp.model_family("anthropic", "claude-opus-4-6") == "claude-opus"
+    assert pmp.model_family("anthropic", "claude-sonnet-5") == "claude-sonnet"
+    assert pmp.model_family("openai", "gpt-5.6-terra") == "gpt-5"
+    assert pmp.model_family("openai", "gpt-5.4") == "gpt-5"
+    # Unknown provider -> exact id, so nothing is ever "same family" by accident.
+    assert pmp.model_family("github-models", "codex-mini-latest") == "codex-mini-latest"
+
+
+def test_same_family_cheaper_pass_is_prepared():
+    report = _report(
+        [
+            _result("claude-opus-4-6", "anthropic", status="passed", cost=0.10),
+            _result("claude-opus-4-8", "anthropic", status="passed", cost=0.08),
+        ]
+    )
+    props = pmp.find_promotions(report, _registry())
+    assert len(props) == 1
+    assert props[0]["to_model_id"] == "claude-opus-4-8"
+    assert props[0]["from_model_id"] == "claude-opus-4-6"
+
+
+def test_cross_family_is_not_prepared():
+    report = _report(
+        [
+            _result("claude-opus-4-6", "anthropic", status="passed", cost=0.10),
+            _result("claude-sonnet-5", "anthropic", status="passed", cost=0.02),
+        ]
+    )
+    assert pmp.find_promotions(report, _registry()) == []  # different family -> human only
+
+
+def test_more_expensive_same_family_is_not_prepared():
+    report = _report(
+        [
+            _result("claude-opus-4-6", "anthropic", status="passed", cost=0.10),
+            _result("claude-opus-4-8", "anthropic", status="passed", cost=0.20),
+        ]
+    )
+    assert pmp.find_promotions(report, _registry()) == []
+
+
+def test_failed_candidate_is_not_prepared():
+    report = _report(
+        [
+            _result("claude-opus-4-6", "anthropic", status="passed", cost=0.10),
+            _result("claude-opus-4-8", "anthropic", status="failed", cost=0.05),
+        ]
+    )
+    assert pmp.find_promotions(report, _registry()) == []
+
+
+def test_candidate_ignored_if_baseline_is_not_the_registry_incumbent():
+    # registry incumbent is opus-4-6 but benchmark baseline is something else
+    report = _report(
+        [
+            _result("claude-opus-9-9", "anthropic", status="passed", cost=0.10),
+            _result("claude-opus-4-8", "anthropic", status="passed", cost=0.05),
+        ],
+        baseline="claude-opus-9-9",
+    )
+    assert pmp.find_promotions(report, _registry()) == []
+
+
+def test_cheapest_same_family_candidate_wins_per_provider():
+    report = _report(
+        [
+            _result("claude-opus-4-6", "anthropic", status="passed", cost=0.10),
+            _result("claude-opus-4-8", "anthropic", status="passed", cost=0.09),
+            _result("claude-opus-4-7", "anthropic", status="passed", cost=0.05),
+        ]
+    )
+    props = pmp.find_promotions(report, _registry())
+    assert len(props) == 1 and props[0]["to_model_id"] == "claude-opus-4-7"
+
+
+def test_apply_promotion_records_history_and_updates_selection():
+    report = _report(
+        [
+            _result("claude-opus-4-6", "anthropic", status="passed", cost=0.10),
+            _result("claude-opus-4-8", "anthropic", status="passed", cost=0.08),
+        ]
+    )
+    promotion = pmp.find_promotions(report, _registry())[0]
+    new = pmp.apply_promotion(_registry(), promotion, today=TODAY)
+    sel = new["selections"][0]
+    assert sel["model_id"] == "claude-opus-4-8"
+    assert "bench-2026-08:anthropic:claude-opus-4-8" in sel["evidence_ids"]
+    assert sel["decided_at"] == "2026-08-01"
+    assert sel["review_by"] == "2026-08-31"
+    assert new["selection_history"][0]["model_id"] == "claude-opus-4-6"
+    assert new["selection_history"][0]["superseded_by"] == "claude-opus-4-8"
+
+
+def test_promote_then_breach_rolls_back_to_prior():
+    # 1) promote 4-6 -> 4-8
+    up = _report(
+        [
+            _result("claude-opus-4-6", "anthropic", status="passed", cost=0.10),
+            _result("claude-opus-4-8", "anthropic", status="passed", cost=0.08),
+        ]
+    )
+    promoted = pmp.apply_promotion(
+        _registry(), pmp.find_promotions(up, _registry())[0], today=TODAY
+    )
+    assert promoted["selections"][0]["model_id"] == "claude-opus-4-8"
+
+    # 2) later benchmark shows the new active model failing -> rollback
+    breach = _report(
+        [_result("claude-opus-4-8", "anthropic", status="failed", cost=0.08)],
+        baseline="claude-opus-4-8",
+    )
+    rollbacks = pmp.find_rollbacks(breach, promoted)
+    assert len(rollbacks) == 1 and rollbacks[0]["to_model_id"] == "claude-opus-4-6"
+    reverted = pmp.apply_rollback(promoted, rollbacks[0], today=TODAY)
+    assert reverted["selections"][0]["model_id"] == "claude-opus-4-6"
+    assert reverted["selection_history"] == []  # history consumed by the rollback
+
+
+def test_rollback_needs_history():
+    breach = _report([_result("claude-opus-4-6", "anthropic", status="failed", cost=0.10)])
+    assert pmp.find_rollbacks(breach, _registry()) == []  # nothing to revert to
+
+
+def test_main_auto_writes_and_signals(tmp_path, capsys):
+    report = _report(
+        [
+            _result("claude-opus-4-6", "anthropic", status="passed", cost=0.10),
+            _result("claude-opus-4-8", "anthropic", status="passed", cost=0.08),
+        ]
+    )
+    bench = tmp_path / "bench.json"
+    reg = tmp_path / "registry.json"
+    out = tmp_path / "out.json"
+    bench.write_text(json.dumps(report))
+    reg.write_text(json.dumps(_registry()))
+    rc = pmp.main(
+        [
+            "--benchmark",
+            str(bench),
+            "--registry",
+            str(reg),
+            "--write",
+            str(out),
+            "--today",
+            "2026-08-01",
+        ]
+    )
+    assert rc == 10  # a change is prepared
+    assert json.loads(out.read_text())["selections"][0]["model_id"] == "claude-opus-4-8"
+
+
+def test_main_noop_when_nothing_qualifies(tmp_path):
+    report = _report(
+        [
+            _result("claude-opus-4-6", "anthropic", status="passed", cost=0.10),
+            _result("claude-sonnet-5", "anthropic", status="passed", cost=0.01),
+        ]
+    )
+    bench = tmp_path / "bench.json"
+    reg = tmp_path / "registry.json"
+    bench.write_text(json.dumps(report))
+    reg.write_text(json.dumps(_registry()))
+    assert (
+        pmp.main(["--benchmark", str(bench), "--registry", str(reg), "--today", "2026-08-01"]) == 0
+    )

tests/workflows/test_workflow_naming.py

@@ -309,6 +309,7 @@ def test_workflow_display_names_are_unique():
     "maint-83-bootstrap-consumer.yml": "Maint 83 Bootstrap Consumer",
     "maint-72-fix-pr-body-conflicts.yml": "Maint 72 Fix PR Body Conflicts",
     "maint-85-keepalive-durability-export.yml": "Maint 85 Keepalive Durability Export",
+    "maint-86-model-promotion-prepare.yml": "Maint 86 Model Promotion Prepare",
     "maint-coverage-guard.yml": "Maint Coverage Guard",
     "maint-metrics-retention.yml": "Maint Metrics Retention",
     "pr-00-gate.yml": "Gate",