Skip to content

feat(maint): batch canonical dev-tool updates - #2902

Merged
stranske merged 4 commits into
mainfrom
codex/issue-2882-batch-dev-tool-pins
Aug 2, 2026
Merged

feat(maint): batch canonical dev-tool updates#2902
stranske merged 4 commits into
mainfrom
codex/issue-2882-batch-dev-tool-pins

Conversation

@stranske

@stranske stranske commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Source: Issue #2882

Closes #2882

Automated Status Summary

Scope

Shared developer-tool pins are currently checked and propagated by several scheduled surfaces: .github/workflows/maint-auto-update-pypi-versions.yml:11-15,45-141, .github/workflows/maint-50-tool-version-check.yml:6-9,43-149, Maint 52, and maint-sync-env-from-pyproject.yml. The repository documentation in docs/ci/TOOL_VERSION_MANAGEMENT.md still describes a partly manual Maint 50 flow even though a daily updater can open source PRs.

This is a current churn source: the same upstream version movement can be detected, commented on, proposed, and propagated by different workflows before the canonical source state settles. That creates extra source PRs, consumer waves, and agent decisions.

Context for Agent

Related Issues/PRs

Tasks

  • Define the canonical pin set and owner across .github/workflows/autofix-versions.env, pyproject.toml, consumer/integration template env files, and requirements.lock.
  • Refactor .github/workflows/maint-auto-update-pypi-versions.yml into the sole source proposal lane with a weekly batch window and a documented security override.
  • Remove duplicate upstream-version proposal/comment behavior from .github/workflows/maint-50-tool-version-check.yml; retain only a bounded policy/freshness audit if it adds unique evidence.
  • Make .github/workflows/maint-52-sync-dev-versions.yml propagation-only: it must consume a settled canonical source commit and must not independently decide newer versions.
  • Keep .github/workflows/maint-sync-env-from-pyproject.yml as a deterministic source-consistency check or replace it with a named equivalent; prevent it from racing the source proposal lane.
  • Group routine ruff, black, mypy, pytest, pytest-cov, pytest-xdist, coverage, isort, and docformatter changes into one bounded source PR per window.
  • Update docs/ci/TOOL_VERSION_MANAGEMENT.md, docs/ops/CONSUMER_REPO_MAINTENANCE.md, and contributor guidance with ownership, cadence, emergency handling, and supersession rules.
  • Close or explicitly supersede overlapping dependency-bot PRs only after the canonical source proposal or merged source commit exists.

Acceptance criteria

  • tests/workflows/test_maint52_sync_dev_versions_pr_body.py proves Maint 52 reports the canonical source commit and never proposes an upstream version independently.
  • tests/workflows/test_sync_dev_dependencies.py proves env, pyproject, template, and lockfile pins move together for every managed tool.
  • A new policy test proves one weekly source proposal can contain multiple routine tool updates and that a security fixture bypasses the window without bypassing tests.
  • Maint 50 and the source updater cannot open or comment competing proposals for the same canonical delta.
  • Consumer propagation starts only after canonical source validation succeeds and creates at most one current wave per repo.
  • Deliberate-break test: introduce one mismatched managed-tool pin in a template or lockfile, verify the named consistency test fails with the exact path/tool, then revert and verify it passes.
  • Run python scripts/dev_check.py --action test and the workflow-validation suite successfully.

Summary by CodeRabbit

  • New Features

    • Added a weekly maintenance proposal process that batches routine updates into one reusable pull request.
    • Added reviewed security overrides for urgent updates outside the weekly schedule.
    • Maintenance checks now report tool-version freshness without creating issues or competing proposals.
    • Improved traceability by recording the canonical source commit used for updates.
  • Documentation

    • Updated maintenance schedules, review procedures, workflow guidance, and security override instructions.
  • Tests

    • Added coverage for update policies, workflow behavior, branch reuse, and source-commit tracking.

@stranske stranske added agent:codex Agent-created issues from Codex agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation labels Aug 2, 2026
Copilot AI review requested due to automatic review settings August 2, 2026 18:08
@cursor

cursor Bot commented Aug 2, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR consolidates shared dev-tool updates into a weekly source-proposal workflow. It adds policy checks and security overrides, makes Maint 50 read-only, records the canonical source commit in Maint 52, and updates documentation and tests.

Changes

Canonical dev-tool updates

Layer / File(s) Summary
Weekly source proposal lane
.github/workflows/maint-auto-update-pypi-versions.yml, scripts/dev_tool_update_policy.py, tests/scripts/test_dev_tool_update_policy.py, tests/workflows/test_dev_tool_source_lane.py
Routine proposals run on Mondays. Security overrides can run at any time. The workflow reuses one weekly branch and edits or creates one source PR.
Read-only freshness reporting
.github/workflows/maint-50-tool-version-check.yml, tests/workflows/test_dev_tool_source_lane.py, docs/WORKFLOW_GUIDE.md, docs/ci/WORKFLOWS.md, docs/ci/WORKFLOW_SYSTEM.md, templates/consumer-repo/WORKFLOW_USER_GUIDE.md
Maint 50 no longer accepts issue-forcing input, writes issues, or creates update proposals. It reports source workflow ownership.
Canonical source commit propagation
.github/workflows/maint-52-sync-dev-versions.yml, tests/workflows/test_dev_tool_source_lane.py, templates/consumer-repo/WORKFLOW_USER_GUIDE.md
Maint 52 records the checked-out source commit and uses it in delivery metadata, PR bodies, and workflow summaries.
Maintenance guidance
docs/ci/TOOL_VERSION_MANAGEMENT.md, docs/ci/WORKFLOW_SYSTEM.md, docs/ops/CONSUMER_REPO_MAINTENANCE.md, templates/consumer-repo/WORKFLOW_USER_GUIDE.md
Documentation describes the weekly source lane, security overrides, read-only freshness checks, central-only ownership, and source commit traceability.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested labels: verify:compare, acceptance-criteria

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler
  participant SourceWorkflow
  participant PolicyTool
  participant WeeklyBranch
  participant SourcePR
  Scheduler->>SourceWorkflow: Trigger Monday run or security_override dispatch
  SourceWorkflow->>PolicyTool: Evaluate timestamp and override
  PolicyTool-->>SourceWorkflow: Return proposal decision and reason
  SourceWorkflow->>WeeklyBranch: Reset or create weekly branch
  SourceWorkflow->>WeeklyBranch: Force-push with lease protection
  SourceWorkflow->>SourcePR: Edit existing PR or create source PR
Loading
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR implements the source lane, weekly batching, security override, read-only Maint 50, and propagation-only Maint 52, but it does not show required pin-consistency and mismatch tests. Add or cite tests for synchronized env, pyproject, template, and lockfile pins, plus deliberate mismatch failure and recovery cases required by #2882.
Docstring Coverage ⚠️ Warning Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Out of Scope Changes check ✅ Passed The workflow, policy, test, and documentation changes support the canonical developer-tool update lane described in #2882.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: batching canonical developer-tool updates through the maintenance workflow.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/issue-2882-batch-dev-tool-pins

Comment @coderabbitai help to get the list of available commands.

@stranske

stranske commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Runner dispatch state for autofix on PR #2902. Do not edit.

Comment thread .github/workflows/maint-52-sync-dev-versions.yml Fixed
Comment thread .github/workflows/maint-52-sync-dev-versions.yml Fixed
Comment thread .github/workflows/maint-52-sync-dev-versions.yml Fixed
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Runner dispatch state for codex on PR #2902. Do not edit.

Comment thread .github/workflows/maint-auto-update-pypi-versions.yml Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR consolidates shared dev-tool version updates into a single canonical “source proposal” workflow lane (weekly batching + explicit security override), makes Maint 50 read-only for freshness evidence, and ensures Maint 52 propagation records the settled canonical source commit for traceability.

Changes:

  • Refactors maint-auto-update-pypi-versions.yml into a weekly (Monday UTC) single-writer PR lane with a security_override dispatch input.
  • Removes issue-creation behavior from maint-50-tool-version-check.yml, leaving it as read-only PyPI freshness reporting.
  • Records and threads a canonical_source_sha through Maint 52 outputs, delivery marker, and PR body; adds tests/docs to lock in the contract.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/workflows/test_dev_tool_source_lane.py Adds workflow-contract assertions for the single-writer lane (Maint Auto-Update), read-only Maint 50, and canonical source SHA propagation.
tests/scripts/test_dev_tool_update_policy.py Adds unit coverage for weekly-window vs security-override gating logic.
scripts/dev_tool_update_policy.py Introduces a small policy helper/CLI to gate whether the source lane may propose updates.
docs/ops/CONSUMER_REPO_MAINTENANCE.md Documents the new single-writer ownership model and the canonical-source SHA traceability expectation.
docs/ci/TOOL_VERSION_MANAGEMENT.md Updates the documented tool-version management flow to reflect the new lane ownership and cadence.
.github/workflows/maint-auto-update-pypi-versions.yml Moves from daily proposals to a weekly batch window + security override; reuses one mutable weekly PR branch.
.github/workflows/maint-52-sync-dev-versions.yml Captures and propagates the settled Workflows source commit SHA into outputs and PR body/marker.
.github/workflows/maint-50-tool-version-check.yml Removes issue-creation permissions and steps; emits freshness evidence only.

Comment thread scripts/dev_tool_update_policy.py
Comment thread .github/workflows/maint-auto-update-pypi-versions.yml

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7f2107331c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/maint-auto-update-pypi-versions.yml Outdated
Comment thread .github/workflows/maint-auto-update-pypi-versions.yml

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.github/workflows/maint-52-sync-dev-versions.yml (1)

279-283: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Pass canonical_source_sha through env: instead of interpolating it directly.

Lines 358, 378, and 421 interpolate ${{ needs.prepare.outputs.canonical_source_sha }} directly into run: script bodies. zizmor flags all three as template-expansion injection risk. The "Create sync PR" step already avoids this pattern for other dynamic values: it threads DELIVERY_PLAN_ID, DELIVERY_GENERATION, DELIVERY_REPOSITORY, and matrix.repo through the env: block at Lines 279-283 and references them as shell variables. Apply the same pattern to canonical_source_sha at all three sites.

Today canonical_source_sha is a git rev-parse HEAD output (hex SHA) from a trusted checkout, so it is not currently attacker-controllable free text. Fix this anyway to close the flagged injection class and keep the pattern consistent within the file.

As per path instructions, "Flag template-injection, unpinned third-party actions, and spoofable bot-actor checks — this workflow YAML is synced across the consumer repos, so one bug replicates fleet-wide."

🔧 Proposed fix: thread the SHA through env at each site
         env:
           GH_TOKEN: ${{ env.REPO_TOKEN }}
           DELIVERY_PLAN_ID: dev-tool-${{ needs.prepare.outputs.versions_hash }}
           DELIVERY_GENERATION: ${{ needs.prepare.outputs.versions_hash }}
           DELIVERY_REPOSITORY: ${{ matrix.repo }}
+          CANONICAL_SOURCE_SHA: ${{ needs.prepare.outputs.canonical_source_sha }}
-            --arg source_commit "${{ needs.prepare.outputs.canonical_source_sha }}" \
+            --arg source_commit "$CANONICAL_SOURCE_SHA" \
-          **Settled source commit:** \`${{ needs.prepare.outputs.canonical_source_sha }}\`
+          **Settled source commit:** \`$CANONICAL_SOURCE_SHA\`

For the summary job (Line 421), add CANONICAL_SOURCE_SHA: ${{ needs.prepare.outputs.canonical_source_sha }} to that step's env: and reference $CANONICAL_SOURCE_SHA instead.

Also applies to: 358-358, 378-378, 421-421

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/maint-52-sync-dev-versions.yml around lines 279 - 283,
Update the three run steps that currently interpolate
needs.prepare.outputs.canonical_source_sha directly in their scripts, including
the summary step, by adding CANONICAL_SOURCE_SHA to each step’s env block and
referencing it as a shell variable. Preserve the existing SHA usage and apply
the same environment-variable pattern already used by DELIVERY_PLAN_ID and
related values.

Sources: Path instructions, Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/maint-auto-update-pypi-versions.yml:
- Around line 166-176: Remove the stale references to the central maintenance
workflows from the consumer template documentation, updating the workflow
entries in WORKFLOW_USER_GUIDE.md rather than adding or duplicating workflow
files. Ensure both template references accurately reflect that these workflows
have no consumer-template counterparts.
- Around line 124-131: The existing weekly branch flow must establish an
explicit push lease when an open PR is found. Update the branch handling around
existing_pr, git fetch, and git checkout -B to check out FETCH_HEAD, capture its
SHA as the expected revision, and pass
--force-with-lease="refs/heads/$branch:$expected_sha" to the later push;
preserve the current behavior for branches without an existing PR.

In `@docs/ci/TOOL_VERSION_MANAGEMENT.md`:
- Around line 72-73: Insert a blank line after the “Canonical Source Proposal
(`maint-auto-update-pypi-versions.yml`)" heading and before its first list item
to satisfy Markdown heading-spacing requirements.
- Around line 69-70: Update the Maint 50 documentation in
TOOL_VERSION_MANAGEMENT.md and the corresponding references in WORKFLOWS.md to
describe only the canonical source PR flow. Remove instructions to create or
close update issues, create separate competing PRs, or otherwise use the
issue-based proposal path, while preserving the read-only freshness-evidence
behavior.

In `@scripts/dev_tool_update_policy.py`:
- Around line 23-34: Add test coverage for the CLI output contract by invoking
scripts.dev_tool_update_policy.main with the specified UTC timestamp and
capturing stdout via capsys. Assert the exact should_propose= and reason= lines,
including a separate invocation with --security-override, while preserving the
existing argument parsing and output format.

---

Outside diff comments:
In @.github/workflows/maint-52-sync-dev-versions.yml:
- Around line 279-283: Update the three run steps that currently interpolate
needs.prepare.outputs.canonical_source_sha directly in their scripts, including
the summary step, by adding CANONICAL_SOURCE_SHA to each step’s env block and
referencing it as a shell variable. Preserve the existing SHA usage and apply
the same environment-variable pattern already used by DELIVERY_PLAN_ID and
related values.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 013fb5b8-0491-45e6-8ac1-642e03643871

📥 Commits

Reviewing files that changed from the base of the PR and between 22f2393 and 7f21073.

📒 Files selected for processing (8)
  • .github/workflows/maint-50-tool-version-check.yml
  • .github/workflows/maint-52-sync-dev-versions.yml
  • .github/workflows/maint-auto-update-pypi-versions.yml
  • docs/ci/TOOL_VERSION_MANAGEMENT.md
  • docs/ops/CONSUMER_REPO_MAINTENANCE.md
  • scripts/dev_tool_update_policy.py
  • tests/scripts/test_dev_tool_update_policy.py
  • tests/workflows/test_dev_tool_source_lane.py

Comment thread .github/workflows/maint-auto-update-pypi-versions.yml
Comment thread .github/workflows/maint-auto-update-pypi-versions.yml
Comment thread docs/ci/TOOL_VERSION_MANAGEMENT.md
Comment thread docs/ci/TOOL_VERSION_MANAGEMENT.md
Comment thread scripts/dev_tool_update_policy.py
@agents-workflows-bot

agents-workflows-bot Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

🤖 Bot Comment Handler

  • Agent: codex
  • Bot comments to address: 8

The agent has been assigned to this PR to address the bot review comments.

Instructions for agent

  1. Implement suggested fixes that improve the code
  2. Skip suggestions that don't apply (note why in your response)

The bot comment handler workflow has prepared context in the artifacts.

@stranske
stranske temporarily deployed to agent-high-privilege August 2, 2026 18:31 — with GitHub Actions Inactive
@stranske

stranske commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Addressed the current review set in cf5a7fae:

  • moved GitHub-expression values into step environment variables for shell safety;
  • made weekly-branch refreshes use an explicit remote lease even when no PR is open;
  • corrected the policy CLI’s skipped-window reason and added output-contract tests;
  • aligned operator, workflow-inventory, and consumer-template guidance with the canonical weekly source lane.

Validation: 9 focused tests passed; Actionlint, YAML parsing, sync_tool_versions --check, sync_dev_dependencies --check --lockfile, and git diff --check passed. The next scheduled closer pass should re-open the PR for fresh CI/review signals.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/ci/TOOL_VERSION_MANAGEMENT.md`:
- Line 181: Update the troubleshooting step for the missing canonical source PR
to reference maint-auto-update-pypi-versions.yml and its 03:00 UTC Monday
schedule (0 3 * * 1), replacing the Maint 50 freshness-report schedule.

In `@docs/ci/WORKFLOW_SYSTEM.md`:
- Line 713: Update the Maint 52 entry in docs/ci/WORKFLOW_SYSTEM.md at line 713
to describe post-source-commit propagation instead of Sunday and push triggers.
Update docs/ci/WORKFLOWS.md at line 125 to remove “weekly or on version changes”
and state that Maint 52 consumes the validated canonical source commit.

In `@templates/consumer-repo/WORKFLOW_USER_GUIDE.md`:
- Around line 1118-1120: Update the Maint 50 checklist near the central
freshness-audit purpose and trigger so it matches the canonical PyPI freshness
contract for the central developer-tool pin set; remove unrelated runtime and
CLI version checks, or explicitly separate and document them if the workflow
validates both scopes.

In `@tests/scripts/test_dev_tool_update_policy.py`:
- Line 4: Add a boundary test alongside the existing tests for
should_propose_update that supplies a timestamp whose local date is Sunday but
whose UTC conversion falls on Monday, then assert the policy evaluates the
weekday using the converted UTC value. Keep the test focused on the UTC day
boundary and use the existing test conventions and policy inputs.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 94a0336f-40a0-477f-a2b8-2a747ff3f90a

📥 Commits

Reviewing files that changed from the base of the PR and between 7f21073 and cf5a7fa.

📒 Files selected for processing (10)
  • .github/workflows/maint-52-sync-dev-versions.yml
  • .github/workflows/maint-auto-update-pypi-versions.yml
  • docs/WORKFLOW_GUIDE.md
  • docs/ci/TOOL_VERSION_MANAGEMENT.md
  • docs/ci/WORKFLOWS.md
  • docs/ci/WORKFLOW_SYSTEM.md
  • scripts/dev_tool_update_policy.py
  • templates/consumer-repo/WORKFLOW_USER_GUIDE.md
  • tests/scripts/test_dev_tool_update_policy.py
  • tests/workflows/test_dev_tool_source_lane.py

Comment thread docs/ci/TOOL_VERSION_MANAGEMENT.md
Comment thread docs/ci/WORKFLOW_SYSTEM.md Outdated
Comment thread templates/consumer-repo/WORKFLOW_USER_GUIDE.md
Comment thread tests/scripts/test_dev_tool_update_policy.py
Align Maint 50/52 docs with the settled-source contract, correct the
source-lane troubleshooting schedule, and cover UTC weekday boundary evaluation.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stranske

stranske commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Closer review recovery (cursor) — second pass

Addressed the 4 remaining non-outdated CodeRabbit threads on head eddf5c6e:

  1. docs/ci/TOOL_VERSION_MANAGEMENT.md — missing-source-PR troubleshooting now points at maint-auto-update-pypi-versions.yml / 0 3 * * 1 (not Maint 50’s 0 8 * * 1).
  2. docs/ci/WORKFLOW_SYSTEM.md + docs/ci/WORKFLOWS.md — Maint 52 Sync Dev Versions described as post-settled-source-commit propagation only (no independent upstream proposal).
  3. templates/consumer-repo/WORKFLOW_USER_GUIDE.md — Maint 50 checklist aligned to central PyPI developer-tool freshness evidence.
  4. tests/scripts/test_dev_tool_update_policy.py — added UTC weekday boundary coverage (2026-08-02T20:30:00-04:00 → Monday UTC).

Validation: tests/scripts/test_dev_tool_update_policy.py (7 passed) + tests/workflows/test_dev_tool_source_lane.py (3 passed); git diff --check clean. Threads resolved. Next: wait for fresh CI on eddf5c6e, then merge when checks/reviews are clear.

@stranske
stranske temporarily deployed to agent-high-privilege August 2, 2026 18:38 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Autofix updated these files:

  • tests/scripts/test_dev_tool_update_policy.py

@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 2, 2026 18:43 Inactive
@stranske stranske added the agent:retry Add to trigger agent retry after rate limit or pause label Aug 2, 2026
@stranske
stranske temporarily deployed to agent-high-privilege August 2, 2026 19:02 — with GitHub Actions Inactive
@stranske
stranske merged commit 4445fd7 into main Aug 2, 2026
80 checks passed
@stranske
stranske deleted the codex/issue-2882-batch-dev-tool-pins branch August 2, 2026 19:22
@stranske stranske added the verify:compare Compare multiple LLM evaluations label Aug 2, 2026
@stranske
stranske temporarily deployed to agent-high-privilege August 2, 2026 19:23 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Provider Comparison Report

Provider Summary

Provider Model Verdict Confidence Summary
openai gpt-5.6-terra CONCERNS 77% The PR is directionally correct: it simplifies Maint 50, introduces a dedicated dev-tool update policy, changes the source updater toward a scheduled batch lane, updates Maint 52, and documents the...
anthropic claude-sonnet-5 CONCERNS 55% The PR makes substantive, plausible progress toward acceptance criteria: it dramatically slims maint-50-tool-version-check.yml (removing duplicate proposal/comment logic), expands maint-auto-update...
📋 Full Provider Details (click to expand)

openai

  • Model: gpt-5.6-terra
  • Verdict: CONCERNS
  • Confidence: 77%
  • Scores:
    • Correctness: 7.0/10
    • Completeness: 6.0/10
    • Quality: 8.0/10
    • Testing: 6.0/10
    • Risks: 6.0/10
  • Summary: The PR is directionally correct: it simplifies Maint 50, introduces a dedicated dev-tool update policy, changes the source updater toward a scheduled batch lane, updates Maint 52, and documents the intended ownership/cadence model. The implementation is readable and the new policy tests are useful. However, several acceptance criteria require stronger demonstrated behavior than is covered by the visible changed tests, particularly validated canonical-commit propagation, one-wave-per-consumer enforcement, deliberate consistency-break diagnostics, workflow-level security-gate ordering, and dependency-bot supersession behavior. These are significant completeness and verification gaps, so the result is CONCERNS rather than PASS.
  • Concerns:
    • The change set substantially reduces Maint 50's proposal behavior and adds a policy helper/source-lane test, but the visible test changes do not include the specifically required Maint 52 PR-body test or the managed-pin synchronization test. Those may exist already, but this PR does not demonstrate that they were updated or expanded to prove the newly required canonical-source semantics.
    • The new tests appear focused on policy and workflow structure. They do not provide end-to-end evidence that Maint 52 waits for successful validation of a settled canonical source commit, reports that exact commit in its consumer PR body, and limits each consumer repository to one current propagation wave.
    • The required deliberate-break coverage is not represented in the changed tests: there is no visible test change that injects a mismatched template or lockfile pin and asserts a named consistency check fails with the exact path and tool before passing after restoration.
    • The security-window behavior is unit-tested through the new policy helper, but workflow-level coverage should also establish that the security override still executes the required validation/test gate before a source proposal can be created. A helper-level decision test alone cannot prove ordering in the GitHub Actions workflow.
    • The diff summary does not show a concrete implementation or test for closing/superseding overlapping dependency-bot PRs only after a canonical proposal or merged canonical commit exists. Documentation changes alone would not enforce that operational requirement.

anthropic

  • Model: claude-sonnet-5
  • Verdict: CONCERNS
  • Confidence: 55%
  • Scores:
    • Correctness: 6.0/10
    • Completeness: 5.0/10
    • Quality: 7.0/10
    • Testing: 5.0/10
    • Risks: 5.0/10
  • Summary: The PR makes substantive, plausible progress toward acceptance criteria: it dramatically slims maint-50-tool-version-check.yml (removing duplicate proposal/comment logic), expands maint-auto-update-pypi-versions.yml into a broader source-proposal lane, adds a new dev_tool_update_policy.py script with accompanying tests, tweaks maint-52 for propagation, and updates documentation across several files. CI (pr-11-ci-smoke, selftest-ci, and most of pr-00-gate) passed on the merge commit. However, several acceptance criteria reference specific pre-existing test files (test_maint52_sync_dev_versions_pr_body.py, test_sync_dev_dependencies.py) and a deliberate-break verification step that are not part of this diff, so their continued validity against the refactored workflows cannot be confirmed from the code changes alone. The maint-sync-env-from-pyproject.yml workflow, explicitly named in the task list, is untouched, and no canonical pin-set files (env, pyproject, templates, lockfile) were modified in this PR, raising questions about whether that foundational task was completed elsewhere or is still outstanding. Given the AC's testing/traceability gaps around consumer propagation gating and cross-surface exclusivity, this warrants a CONCERNS verdict despite reasonable code quality and CI success for what was included.
  • Concerns:
    • The two test files explicitly named in the acceptance criteria (tests/workflows/test_maint52_sync_dev_versions_pr_body.py and tests/workflows/test_sync_dev_dependencies.py) are not touched by this diff, so there is no direct evidence in the PR that Maint 52's canonical-commit reporting or the env/pyproject/template/lockfile consistency checks were validated against the refactored workflows.
    • .github/workflows/maint-sync-env-from-pyproject.yml, explicitly called out in the task list as needing to remain a deterministic consistency check (or be replaced), is not modified in this diff, leaving its relationship to the new source lane unverified.
    • No file changes to .github/workflows/autofix-versions.env, pyproject.toml, consumer/integration template env files, or requirements.lock are present, so the 'canonical pin set' definition task is not evidenced in this PR (may have been done in a prior PR, but not traceable here).
    • The 'deliberate-break' acceptance criterion (introduce a mismatched pin, confirm named test fails with exact path/tool, revert) is not demonstrated anywhere in the diff or new test files.
    • maint-52-sync-dev-versions.yml only received a small +18/-4 change; it's unclear from the diff summary alone whether it now fully avoids independently deciding newer versions versus just consuming a settled canonical commit — this needs closer inspection of the actual workflow logic, which is not visible in the summary.
    • New tests (test_dev_tool_update_policy.py, test_dev_tool_source_lane.py) are small (48 and 34 lines) and their assertions cannot be fully verified from the summary alone; there's uncertainty whether they adequately cover the weekly batching + security bypass requirement together with the exclusivity of the source lane.

Agreement

  • Verdict: CONCERNS (all providers)
  • Correctness: scores within 1 point (avg 6.5/10, range 6.0-7.0)
  • Completeness: scores within 1 point (avg 5.5/10, range 5.0-6.0)
  • Quality: scores within 1 point (avg 7.5/10, range 7.0-8.0)
  • Testing: scores within 1 point (avg 5.5/10, range 5.0-6.0)
  • Risks: scores within 1 point (avg 5.5/10, range 5.0-6.0)

Disagreement

No major disagreements detected.

Unique Insights

  • openai: The change set substantially reduces Maint 50's proposal behavior and adds a policy helper/source-lane test, but the visible test changes do not include the specifically required Maint 52 PR-body test or the managed-pin synchronization test. Those may exist already, but this PR does not demonstrate that they were updated or expanded to prove the newly required canonical-source semantics.; The new tests appear focused on policy and workflow structure. They do not provide end-to-end evidence that Maint 52 waits for successful validation of a settled canonical source commit, reports that exact commit in its consumer PR body, and limits each consumer repository to one current propagation wave.; The required deliberate-break coverage is not represented in the changed tests: there is no visible test change that injects a mismatched template or lockfile pin and asserts a named consistency check fails with the exact path and tool before passing after restoration.; The security-window behavior is unit-tested through the new policy helper, but workflow-level coverage should also establish that the security override still executes the required validation/test gate before a source proposal can be created. A helper-level decision test alone cannot prove ordering in the GitHub Actions workflow.; The diff summary does not show a concrete implementation or test for closing/superseding overlapping dependency-bot PRs only after a canonical proposal or merged canonical commit exists. Documentation changes alone would not enforce that operational requirement.
  • anthropic: The two test files explicitly named in the acceptance criteria (tests/workflows/test_maint52_sync_dev_versions_pr_body.py and tests/workflows/test_sync_dev_dependencies.py) are not touched by this diff, so there is no direct evidence in the PR that Maint 52's canonical-commit reporting or the env/pyproject/template/lockfile consistency checks were validated against the refactored workflows.; .github/workflows/maint-sync-env-from-pyproject.yml, explicitly called out in the task list as needing to remain a deterministic consistency check (or be replaced), is not modified in this diff, leaving its relationship to the new source lane unverified.; No file changes to .github/workflows/autofix-versions.env, pyproject.toml, consumer/integration template env files, or requirements.lock are present, so the 'canonical pin set' definition task is not evidenced in this PR (may have been done in a prior PR, but not traceable here).; The 'deliberate-break' acceptance criterion (introduce a mismatched pin, confirm named test fails with exact path/tool, revert) is not demonstrated anywhere in the diff or new test files.; maint-52-sync-dev-versions.yml only received a small +18/-4 change; it's unclear from the diff summary alone whether it now fully avoids independently deciding newer versions versus just consuming a settled canonical commit — this needs closer inspection of the actual workflow logic, which is not visible in the summary.; New tests (test_dev_tool_update_policy.py, test_dev_tool_source_lane.py) are small (48 and 34 lines) and their assertions cannot be fully verified from the summary alone; there's uncertainty whether they adequately cover the weekly batching + security bypass requirement together with the exclusivity of the source lane.

🔍 LangSmith Traces

@stranske

stranske commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Closer disposition — verifier CONCERNS follow-up opened

Durable Provider Comparison Report on this PR is CONCERNS (2026-08-02T19:30:48Z; openai + anthropic agreed).

Human-disposition audit against current main:

  • Already present on main after feat(maint): batch canonical dev-tool updates #2902: Maint 50 exclusivity, weekly/security policy helper, Maint 52 canonical_source_sha recording, template/lockfile mismatch coverage in test_sync_tool_versions.py / test_sync_dev_dependencies.py.
  • Residual gaps addressed in bounded follow-up fix(maint): close #2882 verifier CONCERNS gaps #2903 (codex/followup-2882-verifier-concerns @ a19482d6): pre-PR pin validation gate (security_override cannot skip), Dependabot/Renovate supersession after canonical proposal, strengthened Maint 52 PR-body / no-independent-PyPI assertions, explicit deliberate-break path:tool assertion, docs update.

Source issue #2882 was reopened for sequencing until #2903 verifies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:codex Agent-created issues from Codex agent:retry Add to trigger agent retry after rate limit or pause agents:keepalive Use to initiate keepalive functionality with agents autofix:patch autofix Opt-in automated formatting & lint remediation verify:compare Compare multiple LLM evaluations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Dependency/Sync] Batch shared dev-tool pin updates through one source lane

4 participants