From 1dfdfdbd4d495491e44192ca72394a9df0c75966 Mon Sep 17 00:00:00 2001 From: stranske Date: Wed, 31 Dec 2025 12:06:07 +0000 Subject: [PATCH] fix: add coverage verification rules to Codex prompts Codex was marking coverage tasks complete after adding tests without verifying actual coverage reached the target percentage. This adds a COVERAGE TASKS - SPECIAL RULES section to all keepalive prompt templates that requires Codex to: 1. Run pytest with --cov after adding tests 2. Find the specific script in the coverage output 3. Verify coverage meets the target before marking complete 4. Continue adding tests if below target This addresses the issue where PR #378 marked all 17 coverage tasks as complete despite only adding one test file. Files updated: - .github/templates/keepalive-instruction.md - .github/codex/prompts/keepalive_next_task.md - templates/consumer-repo/.github/templates/keepalive-instruction.md - templates/consumer-repo/.github/codex/prompts/keepalive_next_task.md - agents/codex-prompt.md --- .github/codex/prompts/keepalive_next_task.md | 10 ++++++++++ .github/templates/keepalive-instruction.md | 10 ++++++++++ agents/codex-prompt.md | 10 ++++++++++ .../.github/codex/prompts/keepalive_next_task.md | 10 ++++++++++ .../.github/templates/keepalive-instruction.md | 10 ++++++++++ 5 files changed, 50 insertions(+) diff --git a/.github/codex/prompts/keepalive_next_task.md b/.github/codex/prompts/keepalive_next_task.md index 5d9d5cccc..ed4365bc2 100644 --- a/.github/codex/prompts/keepalive_next_task.md +++ b/.github/codex/prompts/keepalive_next_task.md @@ -14,4 +14,14 @@ Your objective is to satisfy the **Acceptance Criteria** by completing each **Ta - Prefer small, reviewable commits; leave clear notes when follow-up is required. - Do NOT work on unrelated improvements until all PR tasks are complete. +**COVERAGE TASKS - SPECIAL RULES:** +If a task mentions "coverage" or a percentage target (e.g., "≥95%", "to 95%"), you MUST: +1. Run `pytest tests/ --cov=scripts --cov-report=term-missing` after adding tests +2. Find the specific script in the coverage output table +3. Verify the `Cover` column shows the target percentage or higher +4. Only mark the task complete if the actual coverage meets the target +5. If coverage is below target, add more tests until it meets the target + +A coverage task is NOT complete just because you added tests. It is complete ONLY when the coverage command output confirms the target is met. + **The Tasks and Acceptance Criteria are provided in the appendix below.** Work through them in order. diff --git a/.github/templates/keepalive-instruction.md b/.github/templates/keepalive-instruction.md index 0768a71e9..941efb5f4 100644 --- a/.github/templates/keepalive-instruction.md +++ b/.github/templates/keepalive-instruction.md @@ -28,6 +28,16 @@ After: `- [x] Add validation for user input` - Close the round without source-code changes when acceptance criteria require them. - Change the text of checkboxes—only change `[ ]` to `[x]`. +**COVERAGE TASKS - SPECIAL RULES:** +If a task mentions "coverage" or a percentage target (e.g., "≥95%", "to 95%"), you MUST: +1. Run `pytest tests/ --cov=scripts --cov-report=term-missing` after adding tests +2. Find the specific script in the coverage output table +3. Verify the `Cover` column shows the target percentage or higher +4. Only mark the task complete if the actual coverage meets the target +5. If coverage is below target, add more tests until it meets the target + +A coverage task is NOT complete just because you added tests. It is complete ONLY when the coverage command output confirms the target is met. + **CONTEXT TIP:** If the PR body includes a **Source** section with links to a parent issue or original PR, those contain additional context about the problem being solved. Check the linked issue/PR for background information, related discussions, or details not captured in the Scope section. diff --git a/agents/codex-prompt.md b/agents/codex-prompt.md index c1c999b9c..b450d17d8 100644 --- a/agents/codex-prompt.md +++ b/agents/codex-prompt.md @@ -117,6 +117,16 @@ Your objective is to satisfy the **Acceptance Criteria** by completing each **Ta - Prefer small, reviewable commits; leave clear notes when follow-up is required. - Do NOT work on unrelated improvements until all PR tasks are complete. +**COVERAGE TASKS - SPECIAL RULES:** +If a task mentions "coverage" or a percentage target (e.g., "≥95%", "to 95%"), you MUST: +1. Run `pytest tests/ --cov=scripts --cov-report=term-missing` after adding tests +2. Find the specific script in the coverage output table +3. Verify the `Cover` column shows the target percentage or higher +4. Only mark the task complete if the actual coverage meets the target +5. If coverage is below target, add more tests until it meets the target + +A coverage task is NOT complete just because you added tests. It is complete ONLY when the coverage command output confirms the target is met. + **The Tasks and Acceptance Criteria are provided in the appendix below.** Work through them in order. ## Run context diff --git a/templates/consumer-repo/.github/codex/prompts/keepalive_next_task.md b/templates/consumer-repo/.github/codex/prompts/keepalive_next_task.md index 5d9d5cccc..ed4365bc2 100644 --- a/templates/consumer-repo/.github/codex/prompts/keepalive_next_task.md +++ b/templates/consumer-repo/.github/codex/prompts/keepalive_next_task.md @@ -14,4 +14,14 @@ Your objective is to satisfy the **Acceptance Criteria** by completing each **Ta - Prefer small, reviewable commits; leave clear notes when follow-up is required. - Do NOT work on unrelated improvements until all PR tasks are complete. +**COVERAGE TASKS - SPECIAL RULES:** +If a task mentions "coverage" or a percentage target (e.g., "≥95%", "to 95%"), you MUST: +1. Run `pytest tests/ --cov=scripts --cov-report=term-missing` after adding tests +2. Find the specific script in the coverage output table +3. Verify the `Cover` column shows the target percentage or higher +4. Only mark the task complete if the actual coverage meets the target +5. If coverage is below target, add more tests until it meets the target + +A coverage task is NOT complete just because you added tests. It is complete ONLY when the coverage command output confirms the target is met. + **The Tasks and Acceptance Criteria are provided in the appendix below.** Work through them in order. diff --git a/templates/consumer-repo/.github/templates/keepalive-instruction.md b/templates/consumer-repo/.github/templates/keepalive-instruction.md index 0768a71e9..941efb5f4 100644 --- a/templates/consumer-repo/.github/templates/keepalive-instruction.md +++ b/templates/consumer-repo/.github/templates/keepalive-instruction.md @@ -28,6 +28,16 @@ After: `- [x] Add validation for user input` - Close the round without source-code changes when acceptance criteria require them. - Change the text of checkboxes—only change `[ ]` to `[x]`. +**COVERAGE TASKS - SPECIAL RULES:** +If a task mentions "coverage" or a percentage target (e.g., "≥95%", "to 95%"), you MUST: +1. Run `pytest tests/ --cov=scripts --cov-report=term-missing` after adding tests +2. Find the specific script in the coverage output table +3. Verify the `Cover` column shows the target percentage or higher +4. Only mark the task complete if the actual coverage meets the target +5. If coverage is below target, add more tests until it meets the target + +A coverage task is NOT complete just because you added tests. It is complete ONLY when the coverage command output confirms the target is met. + **CONTEXT TIP:** If the PR body includes a **Source** section with links to a parent issue or original PR, those contain additional context about the problem being solved. Check the linked issue/PR for background information, related discussions, or details not captured in the Scope section.