Skip to content

chore: sync workflow templates - #625

Closed
stranske wants to merge 1 commit into
mainfrom
sync/workflows-8c201f72e3ed
Closed

chore: sync workflow templates#625
stranske wants to merge 1 commit into
mainfrom
sync/workflows-8c201f72e3ed

Conversation

@stranske

Copy link
Copy Markdown
Owner

Sync Summary

Files Updated

  • agents-weekly-metrics.yml: Weekly metrics - aggregates auto-pilot, keepalive, autofix and verifier metrics into summary reports
  • aggregate_agent_metrics.py: Aggregates downloaded weekly agent metrics - required by agents-weekly-metrics.yml
  • terminal_disposition.js: Machine-readable terminal disposition records and source summaries
  • terminal_disposition_coverage.js: Warning-only terminal disposition source coverage preflight
  • weekly_metrics_artifacts.js: Bounded weekly metrics artifact selection contract
  • weekly_metrics_download_manifest.js: Weekly metrics artifact download and extraction manifest contract

Files Skipped

  • pr-00-gate.yml: File exists and sync_mode is create_only
  • ci.yml: File exists and sync_mode is create_only
  • dependabot.yml: File exists and sync_mode is create_only
  • llm_slots.json: None

Review Checklist

  • CI passes with updated workflows
  • No repo-specific customizations were overwritten

Source: stranske/Workflows
Source SHA: 9686fb727c01090a0a60095ef09bafdf10fb821b
Template hash: 8c201f72e3ed
Sync branch: sync/workflows-8c201f72e3ed
Consumer repo: stranske/Template
Manifest: .github/sync-manifest.yml

Automated sync from stranske/Workflows
Template hash: 8c201f72e3ed

Changes synced from sync-manifest.yml
@stranske stranske added sync Automated sync from Workflows automated Automated sync from Workflows labels Apr 26, 2026
Copilot AI review requested due to automatic review settings April 26, 2026 21:06
@stranske stranske added sync Automated sync from Workflows automated Automated sync from Workflows labels Apr 26, 2026
@stranske-keepalive

Copy link
Copy Markdown
Contributor

⚠️ Action Required: Unable to determine source issue for PR #625. The PR title, branch name, or body must contain the issue number (e.g. #123, branch: issue-123, or the hidden marker ).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Syncs workflow template updates from stranske/Workflows to extend weekly metrics reporting (notably adding Codex CLI freshness metrics) and to harden/standardize metrics artifact download + terminal disposition normalization/coverage checks.

Changes:

  • Add codex-cli-freshness as a first-class weekly metrics artifact family and summarize it in the aggregated report/contract.
  • Centralize artifact name sanitization + enrich the weekly download manifest with more compact selection details and result-shape normalization.
  • Refactor terminal disposition normalization of optional fields and adjust verifier model metadata coverage logic.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
scripts/aggregate_agent_metrics.py Adds codex CLI freshness bucketing + summarization, and includes verifier CLI version aggregation.
.github/workflows/agents-weekly-metrics.yml Uses shared safeArtifactPathSegment() helper when creating artifact directories during download.
.github/scripts/weekly_metrics_download_manifest.js Adds compactSelectionDetails() and normalizes artifact download/unzip result shapes before updates/finalization.
.github/scripts/weekly_metrics_artifacts.js Adds codex-cli-freshness to exact/prefixed artifact sets and priority families.
.github/scripts/terminal_disposition_coverage.js Updates unsupported model env var fallback and changes “requires Codex model” logic.
.github/scripts/terminal_disposition.js Introduces normalizeOptionalValue() to normalize numeric/boolean optional fields (incl. llm_cli_version).

elif is_verifier_terminal and model_metadata_required:
verifier_mode = str(entry.get("verifier_mode") or "").strip().lower()
if verifier_mode and verifier_mode != "evaluate":
if verifier_mode != "evaluate":

Copilot AI Apr 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verifier_mode is normalized to an empty string when missing, but this condition now treats the empty value as "not evaluate". That changes behavior from “only enforce model metadata for explicit non-evaluate modes” to “enforce for missing/unknown modes too”, which can inflate missing-model-metadata counts and potentially trigger coverage warnings/hard-blocks for records that omit verifier_mode.

Consider restoring the prior guard (only enforce when verifier_mode is present and not evaluate) or explicitly defaulting missing verifier_mode to evaluate before this comparison.

Suggested change
if verifier_mode != "evaluate":
if verifier_mode and verifier_mode != "evaluate":

Copilot uses AI. Check for mistakes.
Comment on lines +755 to +760
status = str(entry.get("status") or "unknown")
statuses[status] += 1
package = str(entry.get("package") or "unknown")
packages[package] += 1
pinned = str(entry.get("pinned_version") or "unknown")
latest = str(entry.get("latest_version") or "unknown")

Copilot AI Apr 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In _summarise_codex_cli_freshness, status is counted using the raw string (no strip/lower), but later the summary derives outdated_records/latest_unavailable_records via exact-key lookups. Any casing/whitespace differences in incoming data will cause those derived counts to be wrong.

Normalize status (and potentially package/versions) consistently (e.g., strip().lower()) before incrementing counters and before doing key-based lookups.

Suggested change
status = str(entry.get("status") or "unknown")
statuses[status] += 1
package = str(entry.get("package") or "unknown")
packages[package] += 1
pinned = str(entry.get("pinned_version") or "unknown")
latest = str(entry.get("latest_version") or "unknown")
status = str(entry.get("status") or "").strip().lower() or "unknown"
statuses[status] += 1
package = str(entry.get("package") or "").strip().lower() or "unknown"
packages[package] += 1
pinned = str(entry.get("pinned_version") or "").strip() or "unknown"
latest = str(entry.get("latest_version") or "").strip() or "unknown"

Copilot uses AI. Check for mistakes.
const reason = cleanString(record.model_selection_reason);
const verifierMode = cleanString(record.verifier_mode).toLowerCase();
const requiresCodexModel = Boolean(verifierMode) && verifierMode !== 'evaluate';
const requiresCodexModel = verifierMode !== 'evaluate';

Copilot AI Apr 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verifier_mode is cleaned to an empty string when absent, but requiresCodexModel now treats empty/unknown as requiring a Codex model ('' !== 'evaluate'). This is a behavior change from the previous logic and will start flagging missing model metadata for verifier records that omit verifier_mode.

If the intended behavior is “only require a Codex model when a non-evaluate mode is explicitly set”, reintroduce the Boolean(verifierMode) guard or default missing verifier_mode to evaluate before comparison.

Suggested change
const requiresCodexModel = verifierMode !== 'evaluate';
const requiresCodexModel = Boolean(verifierMode) && verifierMode !== 'evaluate';

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automated Automated sync from Workflows sync Automated sync from Workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants