[codex] Add verifier CLI compatibility telemetry - #1913
Merged
Conversation
Contributor
Automated Status SummaryHead SHA: a57ae50
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeSync/Dependabot Campaign QueueRemote GitHub Actions owns discovery for sync-generated and Dependabot PR rounds. Local Codex should only claim items from this issue when SummaryContext for AgentRelated Issues/PRs
Tasks
Acceptance criteria
|
stranske
temporarily deployed
to
agent-high-privilege
April 26, 2026 18:40 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
There was a problem hiding this comment.
Pull request overview
Adds telemetry and aggregation for the installed @openai/codex CLI version used by the verifier, and enforces a reviewed compatibility contract between verifier model candidates and the pinned CLI version.
Changes:
- Capture the installed Codex CLI version in
reusable-agents-verifier.yml, and emit it into verifier metrics + terminal-disposition records. - Normalize/accept
llm_cli_versionin terminal-disposition normalization (source + consumer template), with tests updated accordingly. - Aggregate and surface verifier CLI-version counts in weekly metrics summaries/contracts, with Python tests updated.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/reusable-agents-verifier.yml |
Pins and captures Codex CLI version; threads it into metrics and terminal-disposition output. |
.github/scripts/terminal_disposition.js |
Normalizes llm_cli_version into terminal disposition records. |
templates/consumer-repo/.github/scripts/terminal_disposition.js |
Mirrors terminal-disposition normalization change for consumer template. |
.github/scripts/__tests__/terminal-disposition.test.js |
Extends JS normalization tests to assert llm_cli_version. |
scripts/aggregate_agent_metrics.py |
Adds verifier_cli_versions aggregation and includes it in summary output. |
tests/scripts/test_aggregate_agent_metrics.py |
Updates/extends aggregator tests to validate CLI-version counting and contract output. |
tests/workflows/test_verifier_terminal_disposition.py |
Adds workflow-level contract test ensuring model candidates have a reviewed minimum CLI mapping and the pinned CLI meets it. |
Comment on lines
506
to
512
| terminal_sources = Counter() | ||
| verifier_models = Counter() | ||
| model_selection_reasons = Counter() | ||
| verifier_cli_versions = Counter() | ||
| unsupported_verifier_models = Counter() | ||
| unsupported_model_dispositions = Counter() | ||
| missing_verifier_model_metadata = Counter() |
There was a problem hiding this comment.
scripts/aggregate_agent_metrics.py is marked template_sync: exact in .github/sync-manifest.yml (so the consumer template copy must be updated in the same PR). The template version at templates/consumer-repo/scripts/aggregate_agent_metrics.py does not currently include the new verifier_cli_versions counter/contract field, which will cause template drift and likely fail sync validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to campaign issue #1836
Automated Status Summary
Scope
Sync/Dependabot Campaign Queue
Remote GitHub Actions owns discovery for sync-generated and Dependabot PR rounds. Local Codex should only claim items from this issue when
needs-local-codexwork is queued.Summary
Context for Agent
Related Issues/PRs
Tasks
Acceptance criteria
stranske/Workflowsand opened PR: fix: address sync review feedback for verify followups #1911 Addressed all 3 active review threads from `stranske/Travel-Pla...Head SHA: af12050
Latest Runs: ✅ success — Gate
Required: gate: ✅ success