fix: improve Codex coverage verification to prevent timeouts#386
Merged
fix: improve Codex coverage verification to prevent timeouts#386
Conversation
- Add @slow marker to 3 integration tests that run external tools - Update all keepalive prompts to use targeted coverage verification: - Use -m 'not slow' to skip slow tests - Use --cov=scripts/specific_module for faster feedback - Run specific test file when available - Prevents Codex timeout issues during coverage verification
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
stranske
temporarily deployed
to
agent-standard
GitHub Actions
Inactive
github-actions
bot
added
the
autofix
Contributor
Contributor
|
Status | ✅ no new diagnostics |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR prevents CI timeout issues during Codex coverage verification by optimizing test execution strategies. The solution involves marking slow integration tests for selective exclusion and providing Codex with targeted coverage verification instructions to avoid running the full test suite unnecessarily.
Key Changes:
- Added
@slowmarkers to three integration tests that invoke external tools (ruff, isort, black, mypy) - Updated coverage verification instructions across all Codex prompt templates to use targeted coverage checks and skip slow tests
- Introduced pytest options (
-m "not slow",-x, targeted--cov) to reduce execution time
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/workflows/test_autofix_pipeline_diverse.py | Added @slow marker to integration test that handles diverse autofix scenarios |
| tests/workflows/test_autofix_pipeline.py | Added @slow marker to integration test for trivial ruff issues |
| tests/workflows/test_autofix_full_pipeline.py | Added @slow marker to integration test for full lint and typing pipeline |
| .github/templates/keepalive-instruction.md | Updated coverage verification instructions with targeted pytest commands and slow test exclusion |
| .github/codex/prompts/keepalive_next_task.md | Updated coverage verification instructions with targeted pytest commands and slow test exclusion |
| templates/consumer-repo/.github/templates/keepalive-instruction.md | Updated coverage verification instructions with targeted pytest commands and slow test exclusion (template) |
| templates/consumer-repo/.github/codex/prompts/keepalive_next_task.md | Updated coverage verification instructions with targeted pytest commands and slow test exclusion (template) |
| agents/codex-prompt.md | Updated coverage verification instructions with targeted pytest commands and slow test exclusion |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Automated Status SummaryHead SHA: 590246a
Coverage Overview
Coverage Trend
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeTest coverage is at 71.60% with 17 scripts below 95% coverage. Low coverage makes scripts risky to modify and harder to maintain. Tasks
Acceptance criteria
|
stranske
temporarily deployed
to
agent-standard
GitHub Actions
Inactive
Contributor
🤖 Keepalive Loop StatusPR #386 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated Status Summary
Scope
Test coverage is at 71.60% with 17 scripts below 95% coverage. Low coverage makes scripts risky to modify and harder to maintain.
Tasks
pytest tests/ --cov=scripts --cov-report=term-missingand verify the script shows ≥95% in the Cover column. Do not mark a task complete without verifying the actual coverage percentage.scripts/sync_tool_versions.pyuntil coverage ≥95% (currently 0.00%)scripts/update_residual_history.pyuntil coverage ≥95% (currently 0.00%)scripts/validate_version_pins.pyuntil coverage ≥95% (currently 0.00%)scripts/sync_test_dependencies.pyuntil coverage ≥95% (currently 15.32%)scripts/auto_type_hygiene.pyuntil coverage ≥95% (currently 34.78%)scripts/keepalive_metrics_collector.pyuntil coverage ≥95% (currently 46.48%)scripts/keepalive_metrics_dashboard.pyuntil coverage ≥95% (currently 56.67%)scripts/workflow_health_check.pyuntil coverage ≥95% (currently 62.62%)scripts/classify_test_failures.pyuntil coverage ≥95% (currently 62.87%)scripts/mypy_autofix.pyuntil coverage ≥95% (currently 63.08%)scripts/ledger_validate.pyuntil coverage ≥95% (currently 65.32%)scripts/mypy_return_autofix.pyuntil coverage ≥95% (currently 82.55%)scripts/ledger_migrate_base.pyuntil coverage ≥95% (currently 85.48%)scripts/ci_failure_analyzer.pyuntil coverage ≥95% (currently 87.35%)scripts/fix_cosmetic_aggregate.pyuntil coverage ≥95% (currently 92.31%)scripts/coverage_history_append.pyuntil coverage ≥95% (currently 92.75%)scripts/workflow_validator.pyuntil coverage ≥95% (currently 93.27%)Acceptance criteria
pytest tests/ --cov=scripts --cov-report=term-missingCovercolumn shows ≥95%scripts/shows ≥95% in coverage outputtests/scripts/directoryHead SHA: 19f0595
Latest Runs: ❔ in progress — Gate
Required: gate: ❔ in progress