[codex] Harden weekly metrics telemetry contracts - #1903
Merged
Conversation
Contributor
Contributor
Automated Status SummaryHead SHA: f08eca1
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeSync/Dependabot Campaign QueueRemote GitHub Actions owns discovery for sync-generated and Dependabot PR rounds. Local Codex should only claim items from this issue when SummaryContext for AgentRelated Issues/PRs
Tasks
Acceptance criteria
|
Contributor
🤖 Keepalive Loop StatusPR #1903 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
Contributor
Keepalive Work Log (click to expand)
|
Contributor
There was a problem hiding this comment.
Pull request overview
Hardens the weekly metrics telemetry contracts by bounding legacy pretty-JSON fallback buffering during metrics aggregation and extending the artifact-selection contract so downstream telemetry (terminal-disposition coverage) can preserve “latest candidate” metadata across legacy/new report shapes.
Changes:
- Add bounded legacy pretty-JSON fallback buffering in
_read_ndjson()(lines + bytes) and emit a buffer-limit parse error. - Extend weekly metrics artifact selection output with
latest_candidate_by_familyand update terminal-disposition coverage to read it (and a legacy spelling). - Mirror shared script changes into the consumer template and add focused regression tests; apply Black-driven formatting fixes.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
scripts/aggregate_agent_metrics.py |
Bounds legacy pretty-JSON fallback buffering; emits buffer-limit parse error telemetry. |
templates/consumer-repo/scripts/aggregate_agent_metrics.py |
Mirrors the aggregation hardening into the consumer template. |
tests/scripts/test_aggregate_agent_metrics.py |
Adds regression coverage for the legacy fallback buffer limit behavior. |
.github/scripts/weekly_metrics_artifacts.js |
Adds latest_candidate_by_family to the selection report and exports helper. |
templates/consumer-repo/.github/scripts/weekly_metrics_artifacts.js |
Mirrors the selection contract changes into the consumer template. |
.github/scripts/terminal_disposition_coverage.js |
Preserves latest-candidate metadata by consuming new/legacy selection-map fields. |
templates/consumer-repo/.github/scripts/terminal_disposition_coverage.js |
Mirrors terminal coverage contract handling into the consumer template. |
.github/scripts/__tests__/weekly-metrics-artifacts.test.js |
Adds assertions for latest_candidate_by_family and a unit test for helper mapping. |
.github/scripts/__tests__/terminal-disposition-coverage.test.js |
Adds regression test for legacy latest-candidate maps normalization. |
scripts/repo_review_evaluator.py |
Formatting-only changes to satisfy Black gate. |
tests/scripts/test_repo_review_evaluator.py |
Formatting-only change to satisfy Black gate. |
Comments suppressed due to low confidence (2)
scripts/aggregate_agent_metrics.py:208
- _read_ndjson() still appends a ParseErrorDetail for every invalid JSON line. For a very large malformed metrics file this can still consume unbounded memory (even though the pretty-JSON fallback buffer is now bounded), since
file_errorsgrows with the number of lines. Consider capping stored error details (e.g., keep only the first N details plus a final "details-truncated" sentinel) while separately tracking total error counts/counters so the JSON/markdown contract can still report accurate totals.
try:
parsed = json.loads(raw)
except json.JSONDecodeError:
file_errors.append(_parse_error_detail(path, line_number, "invalid-json"))
continue
templates/consumer-repo/scripts/aggregate_agent_metrics.py:208
- _read_ndjson() still accumulates a ParseErrorDetail per invalid JSON line in
file_errors. A sufficiently large malformed metrics file can therefore still drive unbounded memory growth despite the new bounds on the legacy pretty-JSON fallback buffer. Consider capping stored error details (keeping only the first N + a truncation sentinel) while tracking total error counts separately so reporting remains accurate.
try:
parsed = json.loads(raw)
except json.JSONDecodeError:
file_errors.append(_parse_error_detail(path, line_number, "invalid-json"))
continue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to campaign issue #1836
Automated Status Summary
Scope
Sync/Dependabot Campaign Queue
Remote GitHub Actions owns discovery for sync-generated and Dependabot PR rounds. Local Codex should only claim items from this issue when
needs-local-codexwork is queued.Summary
Context for Agent
Related Issues/PRs
Tasks
Acceptance criteria
Head SHA: a4d2a8d
Latest Runs: ✅ success — Gate
Required: gate: ✅ success