Skip to content

Tighten weekly auth metrics contracts - #1855

Merged
stranske merged 1 commit into
mainfrom
codex/bot-auth-ancestor-artifacts-20260425
Apr 25, 2026
Merged

stranske merged 1 commit into
mainfrom
codex/bot-auth-ancestor-artifacts-20260425

Conversation

@stranske

Copy link
Copy Markdown
Owner

Summary

  • report missing organic auth evidence blockers even when no auth records are present, while keeping disabled organic checks as pass and pure no-data out of hard-block eligibility
  • sync scripts/aggregate_agent_metrics.py into the consumer template and add it to the sync manifest so agents-weekly-metrics sparse checkouts include the entrypoint they invoke
  • add regression coverage for both contracts

Verification

  • node --test .github/scripts/tests/bot-comment-auth-coverage.test.js
  • node --check .github/scripts/bot_comment_auth_coverage.js && node --check templates/consumer-repo/.github/scripts/bot_comment_auth_coverage.js
  • python -m py_compile scripts/aggregate_agent_metrics.py templates/consumer-repo/scripts/aggregate_agent_metrics.py
  • python -m pytest tests/scripts/test_aggregate_agent_metrics.py tests/workflows/test_workflow_agents_consolidation.py -q
  • python scripts/validate_template_sync.py
  • python scripts/validate_template_completeness.py
  • git diff --check

Source queue items: stranske/Travel-Plan-Permission#895 and stranske/Template#582.

Copilot AI review requested due to automatic review settings April 25, 2026 21:38
@stranske-automation-bot

Copy link
Copy Markdown
Collaborator

⚠️ Action Required: Unable to determine source issue for PR #1855. The PR title, branch name, or body must contain the issue number (e.g. #123, branch: issue-123, or the hidden marker ).

@stranske-keepalive

stranske-keepalive Bot commented Apr 25, 2026 •

Copy link
Copy Markdown
Contributor

Automated Status Summary

Head SHA: b1766d2
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / guard
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job Result Logs
(no jobs reported) ⏳ pending —

Coverage Overview

  • Coverage history entries: 1

Coverage Trend

Metric Value
Current 93.12%
Baseline 85.00%
Delta +8.12%
Minimum 70.00%
Status ✅ Pass

Top Coverage Hotspots (lowest coverage)

File Coverage Missing
src/cli_parser.py 81.8% 4
src/percentile_calculator.py 95.0% 1
src/aggregator.py 95.0% 2
src/__init__.py 100.0% 0
src/ndjson_parser.py 100.0% 0

Updated automatically; will refresh on subsequent CI/Docker completions.


Keepalive checklist

Scope

No scope information available

Tasks

  • No tasks defined

Acceptance criteria

  • No acceptance criteria defined

@stranske-automation-bot

stranske-automation-bot commented Apr 25, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Keepalive Loop Status

PR #1855 | Agent: Codex | Iteration 0/5

Current State

Metric Value
Iteration progress [----------] 0/5
Action wait (missing-agent-label)
Disposition skipped (transient)
Gate success
Tasks 0/10 complete
Timeout 45 min (default)
Timeout usage 3m elapsed (9%, 42m remaining)
Keepalive ❌ disabled
Autofix ❌ disabled

🔍 Failure Classification

| Error type | infrastructure |
| Error category | resource |
| Suggested recovery | Confirm the referenced resource exists (repo, PR, branch, workflow, or file). |

@stranske-automation-bot

stranske-automation-bot commented Apr 25, 2026 •

Copy link
Copy Markdown
Collaborator
Keepalive Work Log (click to expand)
# Time (UTC) Agent Action Result Files Tasks Progress Commit Gate
0 2026-04-25 21:39:19 Codex wait (missing-agent-label-transient) skipped — 0 0/10 — —
0 2026-04-25 21:40:02 Codex wait (missing-agent-label-transient) skipped — 0 0/10 — cancelled
0 2026-04-25 21:42:41 Codex wait (missing-agent-label-transient) skipped — 0 0/10 — success

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates weekly auth-metrics contracts so “missing organic evidence” blockers are still surfaced even when no auth coverage records are present, and ensures the weekly metrics aggregation entrypoint is available in consumer sparse checkouts via template + sync-manifest alignment.

Changes:

  • Extend bot-comment auth coverage summarization to emit organic “missing-*” blockers even when records.length === 0, while keeping overall status as no-data in pure no-record scenarios.
  • Sync scripts/aggregate_agent_metrics.py into the consumer template and include it in .github/sync-manifest.yml; add a workflow regression test to prevent drift.
  • Add/adjust Node tests to cover the updated organic-evidence/no-auth-record behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/workflows/test_workflow_agents_consolidation.py Adds a regression test ensuring the weekly metrics aggregator script is referenced in workflows and present in the sync manifest + template.
templates/consumer-repo/scripts/aggregate_agent_metrics.py Adds the aggregator script into the consumer template so sparse checkouts can run it.
templates/consumer-repo/.github/scripts/bot_comment_auth_coverage.js Mirrors bot-comment auth organic-evidence contract tightening in the consumer template copy.
.github/sync-manifest.yml Adds scripts/aggregate_agent_metrics.py to the consumer sync manifest.
.github/scripts/bot_comment_auth_coverage.js Implements organic “missing evidence” blockers for the no-auth-record path and refines no-data blocker classification.
.github/scripts/__tests__/bot-comment-auth-coverage.test.js Updates assertions to validate missing-organic blockers are emitted and carried into enforcement blockers even with zero auth records.

@stranske
stranske merged commit ecc8f25 into main Apr 25, 2026
67 of 75 checks passed
@stranske
stranske deleted the codex/bot-auth-ancestor-artifacts-20260425 branch April 25, 2026 21:42

This branch was previously deployed

1 inactive deployment
agent-standard — cabfef9a Deployed Apr 25, 2026 by stranske via privilege environment gate #6058
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants