-
Notifications
You must be signed in to change notification settings - Fork 11
Add test-coverage-detector workflow (detector/fixer pair for test-improver) #488
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
strawgate
merged 8 commits into
main
from
copilot/create-detector-version-test-improver
Feb 28, 2026
Merged
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
f51f54e
Initial plan
Copilot acd07f0
Add test-coverage-detector workflow (detector version of test-improver)
Copilot 4815614
Fix typo: end-to-one → end-to-end in test-coverage-detector prompt
Copilot 1d7b1b7
Merge branch 'main' into copilot/create-detector-version-test-improver
strawgate 51e10aa
Fix broken test coverage docs links
github-actions[bot] 340573f
ci: trigger CI checks
github-actions[bot] 6904e05
Merge main and resolve conflicts in docs catalog and quick-setup
strawgate fbc6c72
Merge remote changes and resolve docs catalog conflict
strawgate File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
1,502 changes: 1,502 additions & 0 deletions
1,502
.github/workflows/gh-aw-test-coverage-detector.lock.yml
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,158 @@ | ||
| --- | ||
| inlined-imports: true | ||
| name: "Test Coverage Detector" | ||
| description: "Find under-tested code paths and file a test coverage report" | ||
| imports: | ||
| - gh-aw-fragments/elastic-tools.md | ||
| - gh-aw-fragments/runtime-setup.md | ||
| - gh-aw-fragments/formatting.md | ||
| - gh-aw-fragments/rigor.md | ||
| - gh-aw-fragments/mcp-pagination.md | ||
| - gh-aw-fragments/messages-footer.md | ||
| - gh-aw-fragments/safe-output-create-issue.md | ||
| - gh-aw-fragments/previous-findings.md | ||
| - gh-aw-fragments/pick-three-keep-one.md | ||
| - gh-aw-fragments/scheduled-audit.md | ||
| - gh-aw-fragments/network-ecosystems.md | ||
| engine: | ||
| id: copilot | ||
| model: ${{ inputs.model }} | ||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| model: | ||
| description: "AI model to use" | ||
| type: string | ||
| required: false | ||
| default: "gpt-5.3-codex" | ||
| additional-instructions: | ||
| description: "Repo-specific instructions appended to the agent prompt" | ||
| type: string | ||
| required: false | ||
| default: "" | ||
| setup-commands: | ||
| description: "Shell commands to run before the agent starts (dependency install, build, etc.)" | ||
| type: string | ||
| required: false | ||
| default: "" | ||
| allowed-bot-users: | ||
| description: "Allowlisted bot actor usernames (comma-separated)" | ||
| type: string | ||
| required: false | ||
| default: "github-actions[bot]" | ||
| messages-footer: | ||
| description: "Footer appended to all agent comments and reviews" | ||
| type: string | ||
| required: false | ||
| default: "" | ||
| title-prefix: | ||
| description: "Title prefix for created issues (e.g. '[test-coverage]')" | ||
| type: string | ||
| required: false | ||
| default: "[test-coverage]" | ||
| secrets: | ||
| COPILOT_GITHUB_TOKEN: | ||
| required: true | ||
| roles: [admin, maintainer, write] | ||
| bots: | ||
| - "${{ inputs.allowed-bot-users }}" | ||
| concurrency: | ||
| group: test-coverage-detector | ||
| cancel-in-progress: true | ||
| permissions: | ||
| actions: read | ||
| contents: read | ||
| issues: read | ||
| pull-requests: read | ||
| tools: | ||
| github: | ||
| toolsets: [repos, issues, pull_requests, search, labels, actions] | ||
| bash: true | ||
| web-fetch: | ||
| strict: false | ||
| safe-outputs: | ||
| activation-comments: false | ||
| noop: | ||
| create-issue: | ||
| max: 1 | ||
| title-prefix: "${{ inputs.title-prefix }} " | ||
| close-older-issues: false | ||
| expires: 7d | ||
| timeout-minutes: 90 | ||
| steps: | ||
| - name: Repo-specific setup | ||
| if: ${{ inputs.setup-commands != '' }} | ||
| env: | ||
| SETUP_COMMANDS: ${{ inputs.setup-commands }} | ||
| run: eval "$SETUP_COMMANDS" | ||
| --- | ||
|
|
||
| Identify under-tested code paths that would benefit from focused tests and file a report issue with specific, actionable recommendations. | ||
|
|
||
| **The bar is high: you must identify concrete, high-value test gaps before filing.** Most runs should end with `noop` — that means test coverage is adequate. | ||
|
|
||
| ### Data Gathering | ||
|
|
||
| 1. Determine required repo commands (lint/build/test) and how to run tests: | ||
| - Check README, CONTRIBUTING, DEVELOPING, Makefile, CI config, package.json, pyproject.toml, and similar files. | ||
| 2. Identify coverage tooling (nyc, jest --coverage, pytest --cov, go test -cover, etc.). | ||
| - If coverage is available and reasonably fast, run it to find low-coverage files. | ||
| - If coverage tooling is not available, use code analysis to identify untested code paths. | ||
| 3. Review recent changes: | ||
| - Run `git log --since="28 days ago" --stat` and identify recent changes without corresponding test updates. | ||
| 4. Use the **Pick Three, Keep One** pattern for the investigation phase: spawn 3 `general-purpose` sub-agents, each searching for test gaps from a different angle (e.g., one analyzing coverage reports, one reviewing recent commits for missing tests, one examining error paths and edge cases in public APIs). Include the repo conventions, coverage data, and the full "What to Look For" / "What to Skip" criteria in each sub-agent prompt. Each sub-agent should return its best candidate finding or recommend `noop`. | ||
|
|
||
| ### What to Look For | ||
|
|
||
| - **Untested public APIs**: exported functions, CLI commands, API endpoints, or config parsing that have no or minimal test coverage. | ||
| - **Error paths**: exception handling, validation failures, and edge cases that are not exercised by any test. | ||
| - **Recent changes without tests**: code modified in the last 28 days where no corresponding test was added or updated. | ||
| - **Critical business logic**: core algorithms, data transformations, or state machines with insufficient coverage. | ||
| - **Trace to user-facing behavior.** For every candidate, answer: "What end-to-end user action would exercise this code path?" Only report gaps that map to a concrete user scenario. | ||
|
|
||
| ### What to Skip | ||
|
|
||
| - Trivial getters, setters, or simple constructors — testing these adds maintenance burden without catching regressions. | ||
| - Generated code, vendored dependencies, or third-party code. | ||
| - Test files themselves. | ||
| - Code paths that are adequately covered by integration or end-to-end tests, even if unit coverage is low. | ||
| - Internal helper functions that are only reachable through already-tested public APIs. | ||
| - Subjective "should have more tests" observations without a concrete user scenario. | ||
|
|
||
| ### Quality Gate — When to Noop | ||
|
|
||
| Call `noop` if any of these are true: | ||
| - Coverage tooling shows adequate coverage and no clear gaps in critical paths. | ||
| - The only gaps found are trivial (getters, constructors, simple pass-through functions). | ||
| - You cannot describe a concrete user scenario for any identified gap. | ||
| - All gaps are already tracked in open issues. | ||
| - The codebase has no testable code (e.g., pure configuration, documentation-only). | ||
|
|
||
| ### Issue Format | ||
|
|
||
| **Issue title:** Test coverage gaps (date) | ||
|
|
||
| **Issue body:** | ||
|
|
||
| > ## Summary | ||
| > - Coverage method: [tool used or manual analysis] | ||
| > - Files analyzed: [count] | ||
| > - Gaps identified: [count] | ||
| > | ||
| > ## Findings | ||
| > | ||
| > ### 1. [Brief description of the gap] | ||
| > **File:** `path/to/file.ext` | ||
| > **Function/Method:** `FuncName(...)` | ||
| > **User scenario:** [Describe the real-world user action that exercises this code path] | ||
| > **Why it matters:** [What regression could this catch? What could break?] | ||
| > **Suggested test approach:** [Brief description of what a test should verify] | ||
| > | ||
| > ## Suggested Actions | ||
| > - [ ] [Concrete action for each finding, e.g., "Add unit test for error handling in `ParseConfig` when config file is missing"] | ||
|
|
||
| ### Labeling | ||
|
|
||
| - If the `test-coverage` label exists (check with `github-get_label`), include it in the `create_issue` call; otherwise, rely on the `${{ inputs.title-prefix }}` title prefix only. | ||
|
|
||
| ${{ inputs.additional-instructions }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # This file is auto-generated by scripts/dogfood.sh. Do not edit directly. | ||
| # Edit gh-agent-workflows/test-coverage-detector/example.yml and run 'make compile' to regenerate. | ||
| name: Trigger Test Coverage Detector | ||
| on: | ||
| schedule: | ||
| - cron: "0 10 * * 1-5" | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| actions: read | ||
| contents: read | ||
| issues: write | ||
| pull-requests: read | ||
|
|
||
| jobs: | ||
| run: | ||
| uses: ./.github/workflows/gh-aw-test-coverage-detector.lock.yml | ||
| secrets: | ||
| COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| # Test Coverage | ||
|
|
||
| Find under-tested code and automatically add focused tests. | ||
|
|
||
| **Test Coverage Detector** identifies code paths with no or minimal test coverage by running coverage tools (when available) and analyzing recent changes for missing tests. **Test Improver** adds focused tests that validate real behavior and cleans up redundant tests. Most runs of either workflow end with `noop`. | ||
|
|
||
| ## Quick install | ||
|
|
||
| ### Detector only | ||
|
|
||
| Install the detector alone if you want to review test coverage reports before acting. | ||
|
|
||
| ```bash | ||
| mkdir -p .github/workflows && curl -sL \ | ||
| https://raw.githubusercontent.com/elastic/ai-github-actions/v0/gh-agent-workflows/test-coverage-detector/example.yml \ | ||
| -o .github/workflows/test-coverage-detector.yml | ||
| ``` | ||
|
|
||
| ### Full loop (detector + fixer) | ||
|
|
||
| Install both for autonomous detection and test improvement. | ||
|
|
||
| ```bash | ||
| mkdir -p .github/workflows && \ | ||
| curl -sL https://raw.githubusercontent.com/elastic/ai-github-actions/v0/gh-agent-workflows/test-coverage-detector/example.yml \ | ||
| -o .github/workflows/test-coverage-detector.yml && \ | ||
| curl -sL https://raw.githubusercontent.com/elastic/ai-github-actions/v0/gh-agent-workflows/test-improver/example.yml \ | ||
| -o .github/workflows/test-improver.yml | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Test Coverage Detector | ||
|
|
||
| Identifies code paths with no or minimal test coverage by running coverage tools (when available) and analyzing recent changes for missing tests. Files a report with specific, actionable recommendations for each gap — including the user scenario and suggested test approach. | ||
|
|
||
| ### Trigger | ||
|
|
||
| | Event | Schedule | | ||
| | --- | --- | | ||
| | `schedule` | Weekdays | | ||
| | `workflow_dispatch` | Manual | | ||
|
|
||
| ### Inputs | ||
|
|
||
| | Input | Description | Default | | ||
| | --- | --- | --- | | ||
| | `additional-instructions` | Repo-specific instructions appended to the agent prompt | `""` | | ||
| | `setup-commands` | Shell commands run before the agent starts | `""` | | ||
|
|
||
| ### Safe outputs | ||
|
|
||
| - `create-issue` — file a test coverage report (max 1, auto-closes older reports) | ||
|
|
||
| ### Example workflow | ||
|
|
||
| ```yaml | ||
| name: Test Coverage Detector | ||
| on: | ||
| schedule: | ||
| - cron: "0 10 * * 1-5" | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| actions: read | ||
| contents: read | ||
| issues: write | ||
| pull-requests: read | ||
|
|
||
| jobs: | ||
| run: | ||
| uses: elastic/ai-github-actions/.github/workflows/gh-aw-test-coverage-detector.lock.yml@v0 | ||
| secrets: | ||
| COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Test Improver (fixer) | ||
|
|
||
| Identifies code paths with no or minimal test coverage, adds focused tests that validate real behavior, and removes or consolidates clearly redundant tests. Only opens a PR for changes that would catch actual regressions — not trivial getters or incidental coverage. | ||
|
|
||
| ### Trigger | ||
|
|
||
| | Event | Schedule | | ||
| | --- | --- | | ||
| | `schedule` | Weekly | | ||
| | `workflow_dispatch` | Manual | | ||
|
|
||
| ### Inputs | ||
|
|
||
| | Input | Description | Default | | ||
| | --- | --- | --- | | ||
| | `additional-instructions` | Repo-specific instructions appended to the agent prompt | `""` | | ||
| | `setup-commands` | Shell commands run before the agent starts | `""` | | ||
|
|
||
| ### Safe outputs | ||
|
|
||
| - `create-pull-request` — open a PR with test improvements | ||
|
|
||
| ### Example workflow | ||
|
|
||
| ```yaml | ||
| name: Test Improver | ||
| on: | ||
| schedule: | ||
| - cron: "0 9 * * 1" | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| actions: read | ||
| contents: write | ||
| issues: write | ||
| pull-requests: write | ||
|
|
||
| jobs: | ||
| run: | ||
| uses: elastic/ai-github-actions/.github/workflows/gh-aw-test-improver.lock.yml@v0 | ||
| secrets: | ||
| COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| # Test Coverage Detector | ||
|
|
||
| Find under-tested code paths and file a test coverage report. | ||
|
|
||
| ## How it works | ||
|
|
||
| Identifies code paths with no or minimal test coverage by running coverage tools (when available) and analyzing recent changes for missing tests. Files a report with specific, actionable recommendations for each gap — including the user scenario and suggested test approach. The bar is high; most runs end with `noop`. | ||
|
|
||
| ## Quick Install | ||
|
|
||
| ```bash | ||
| mkdir -p .github/workflows && curl -sL \ | ||
| https://raw.githubusercontent.com/elastic/ai-github-actions/v0/gh-agent-workflows/test-coverage-detector/example.yml \ | ||
| -o .github/workflows/test-coverage-detector.yml | ||
| ``` | ||
|
|
||
| See [example.yml](example.yml) for the full workflow file. | ||
|
|
||
| ## Trigger | ||
|
|
||
| | Event | Schedule | | ||
| | --- | --- | | ||
| | `schedule` | Weekdays | | ||
| | `workflow_dispatch` | Manual | | ||
|
|
||
| ## Inputs | ||
|
|
||
| | Input | Description | Required | Default | | ||
| | --- | --- | --- | --- | | ||
| | `additional-instructions` | Repo-specific instructions appended to the agent prompt | No | `""` | | ||
| | `setup-commands` | Shell commands run before the agent starts | No | `""` | | ||
| | `allowed-bot-users` | Allowlisted bot actor usernames (comma-separated) | No | `github-actions[bot]` | | ||
|
|
||
| ## Safe Outputs | ||
|
|
||
| - `create-issue` — file a test coverage report (max 1, auto-closes older reports) | ||
|
|
||
| ## Pairing | ||
|
|
||
| This workflow is the read-only companion to [Test Improver](../test-improver/). The detector finds test coverage gaps; the improver adds the tests. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[LOW] Broken docs link for Test Coverage detector/fixer pair
This row links to
gh-agent-workflows/test-coverage-detector.md, but that page does not exist in this PR (mkdocs.ymland docs addworkflows/gh-agent-workflows/test-coverage.md). The current link will 404 in rendered docs.