Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
c8c6687
Improve dotnet-test eval coverage and efficiency
Evangelink Jul 20, 2026
292855a
Fix dotnet-test eval activation and quality
Evangelink Jul 20, 2026
7765e90
Strengthen dotnet-test skill activation
Evangelink Jul 20, 2026
64504fb
Merge origin/main into issue-899-dotnet-test-followup
Evangelink Jul 20, 2026
f9eac6d
Merge origin/main into issue-899-dotnet-test-followup
AbhitejJohn Jul 21, 2026
4698555
Address dotnet-test review feedback
Evangelink Jul 22, 2026
1e0ccd5
Potential fix for pull request finding
Evangelink Jul 22, 2026
788e0ee
Improve dotnet-test eval signal
Evangelink Jul 23, 2026
5fc8eaa
Assert unavailable production disclaimer
Evangelink Jul 23, 2026
2822ca9
Tighten static pairing eval guards
Evangelink Jul 23, 2026
4e1409d
Strengthen dotnet-test eval separation
Evangelink Jul 23, 2026
8f2470c
Focus evals on differentiating outcomes
Evangelink Jul 23, 2026
44a9b0d
Remove nondifferentiating pairing evals
Evangelink Jul 23, 2026
3d4b631
Measure focused test coverage quality
Evangelink Jul 23, 2026
007584b
Keep stable dotnet-test eval scenarios
Evangelink Jul 23, 2026
06d2b8f
Remove unstable generation coverage race
Evangelink Jul 23, 2026
2d14424
Tighten eval output assertions
Evangelink Jul 23, 2026
db1e3e4
Restore generation guardrail coverage
Evangelink Jul 24, 2026
f5e892e
Remove retired Contoso eval fixture
Evangelink Jul 24, 2026
026a83f
Strengthen restored eval signal
Evangelink Jul 24, 2026
f5ac754
Clarify pytest grading context
Evangelink Jul 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 26 additions & 12 deletions plugins/dotnet-test/agents/code-testing-generator.agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ license: MIT

You coordinate test generation using the Research-Plan-Implement (RPI) pipeline. You are polyglot — you work with any programming language.

> **Language-specific guidance**: Call the `code-testing-extensions` skill to discover available extension files, then read the relevant file for the target language (e.g., `dotnet.md` for .NET).
> **Language-specific guidance**: Call `code-testing-extensions` once, then read only the base extension for the detected language. Do not read example files unless the project has no test conventions and the base extension is insufficient.

## Pipeline Overview

Expand All @@ -36,7 +36,15 @@ You coordinate test generation using the Research-Plan-Implement (RPI) pipeline.

Understand what the user wants: scope (project, files, classes), priority areas, framework preferences. If clear, proceed directly. If the user provides no details or a very basic prompt (e.g., "generate tests"), use [unit-test-generation.prompt.md](../skills/code-testing-agent/unit-test-generation.prompt.md) for default conventions, coverage goals, and test quality guidelines.

**Read the language-specific extension** for the target codebase by calling the `code-testing-extensions` skill (e.g., read `dotnet.md` for .NET/C# projects). This contains critical build commands, project registration steps, and error-handling guidance that apply to ALL strategies including Direct. You MUST read this file before writing any code.
Before writing code, read the language-specific base extension. Reuse it for the whole run; sub-agents must not independently reload the same reference unless they need a section that was not captured in `.testagent/research.md`.

Create a **requirement checklist** from the request before choosing a strategy.
Preserve each explicit behavior, layer, collaborator seam, boundary case,
integration, coverage threshold, and required artifact as a separate item. For
example, "mock the repository in service tests", "exercise SQLite in memory",
and "cover pagination boundaries" are three independently verifiable
requirements. Direct strategy keeps this checklist in context; delegated
strategies record it in `.testagent/research.md`.

### Step 2: Choose Execution Strategy

Expand Down Expand Up @@ -67,7 +75,7 @@ Based on the request scope, pick exactly one strategy and follow it:

Delegate to the `code-testing-researcher` subagent with this task:

> Research the codebase at [PATH] for test generation. Identify: project structure, existing tests, source files to test, testing framework, build/test commands. Build a dependency graph and estimate preexisting coverage.
> Research [REQUESTED SCOPE] at [PATH] for test generation. Produce a bounded target inventory, existing test conventions, source-to-test pairs, dependencies only for those targets, and exact build/test/discovery commands. Do not inventory unrelated source files.

Output: `.testagent/research.md`

Expand Down Expand Up @@ -127,18 +135,23 @@ Additional self-review heuristics (still required, even when running the skills)

### Step 8: Coverage Gap Iteration

After the previous phases complete, check for uncovered source files:
After the previous phases complete, use the target inventory already recorded in `.testagent/research.md` and the files reported by implementers. Do not rescan or reread the workspace.

1. List all source files in scope.
2. List all test files created.
3. Identify source files with no corresponding test file.
4. Generate tests for each uncovered file, build, test, and fix.
5. Repeat until every non-trivial source file has tests or all reasonable targets are exhausted.
6. If this step added or modified any tests, re-run the full Step 7 pre-completion gate (`test-gap-analysis` + `assertion-quality` + prompt-scenario coverage) on the new/changed tests before reporting completion — Step 8 output must not bypass the gate.
1. Compare the requirement checklist and bounded target inventory with the implemented tests.
2. Inspect the generated test bodies for evidence of every checklist item. A covered line does not prove that a requested collaborator was mocked, a concrete result was asserted, or a boundary/property combination was exercised.
3. If the user requested a measurable coverage target, collect coverage once and prioritize only gaps inside the requested scope.
4. Add tests for any unaddressed checklist item before adding optional cases merely to raise test count.
5. Stop only when every feasible checklist item is covered and the stated target is met; do not recursively expand into unrelated files.
6. If this step added or modified tests, re-run the full Step 7 pre-completion gate on those tests before reporting completion.

### Step 9: Report Results

Summarize tests created, report any failures or issues, suggest next steps if needed.
Summarize tests created, report any failures or issues, and include a compact
**Requirement coverage** section that maps each explicit request to the test
file or test group that satisfies it. Name concrete evidence such as the mock
or fake used, fixed inputs and expected values, boundary combinations,
in-memory integration fixture, and generated coverage artifact. Do not report
a requirement as covered based only on aggregate coverage.

**Example final report:**

Expand Down Expand Up @@ -169,7 +182,7 @@ Summarize tests created, report any failures or issues, suggest next steps if ne
- Consider adding integration tests for database layer
```

> **Language-specific examples**: For a complete end-to-end walkthrough including sample source code, research output, plan, generated tests, and fix cycles, call the `code-testing-extensions` skill and read the matching `<language>-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, and `java-examples.md` are currently available. For other languages, follow the base extension file (e.g., `rust.md`, `kotlin.md`) and adapt the pipeline shape shown in the closest example.
Use a language example from `code-testing-extensions` only when no existing tests establish a usable convention. Never load examples merely to confirm a pattern already present in the repository.

## State Management

Expand All @@ -194,3 +207,4 @@ All state is stored in `.testagent/` folder:
11. **Always validate** — final build, final test, coverage-gap review, and reporting are mandatory for ALL strategies including Direct; never skip final validation. The pre-completion self-review gate from Step 7 (`test-gap-analysis` + `assertion-quality` skills, plus the prompt-scenario coverage check) is mandatory for every non-trivial test addition and may be skipped only for trivially small tasks (fewer than 5 generated tests *and* no behaviors specified in the prompt), per Step 7
12. **Preserve existing tests** — never delete or overwrite existing test files; create new files or append to existing ones
13. **Never mutate version control** — your only outputs are additive test files plus minimal build-manifest edits to register a new test project. Any command that reverts, restores, resets, stashes, or cleans the tree, or deletes tracked files, is out of scope — even when the workspace looks broken or incomplete.
14. **Bound context and reuse findings** — scope every search to the user's requested files/modules, read only the source and existing tests needed for the next implementation phase, and reuse `.testagent/research.md` instead of repeating workspace discovery.
8 changes: 4 additions & 4 deletions plugins/dotnet-test/agents/code-testing-implementer.agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,15 @@ Given a phase from the plan, write all the test files for that phase and ensure

### 1. Read the Plan and Research

- Read `.testagent/plan.md` to understand the overall plan
- Read `.testagent/research.md` for build/test commands and patterns
- Read only the current phase from `.testagent/plan.md`
- Read the command, convention, and target entries needed for that phase from `.testagent/research.md`
- Identify which phase you're implementing

### 2. Read Source Files and Validate References

For each file in your phase:

- **Read the entire source file** — do not write tests based on function names or signatures alone
- Read the complete implementation of the methods being tested, plus their containing type and directly used collaborators. Do not read unrelated types or repeat files already fully captured in the current phase context.
- Understand the public API — verify exact parameter types, count, return types, and **actual return values for key inputs** before writing assertions
- **Trace the logic** for each code path you plan to test — understand what the function actually does, not what you think it should do
- Note dependencies and how to mock them
Expand Down Expand Up @@ -126,7 +126,7 @@ ISSUES:
- [Any unresolved issues]
```

> **Concrete example**: For a complete generated test file and build-error fix cycle walkthrough, call the `code-testing-extensions` skill and read the matching `<language>-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, `java-examples.md` ("Sample Generated Test File" and "Sample Fix Cycle" sections). For other languages, adapt the closest example to the project's framework.
Consult a language example only when the repository has no representative tests and the base extension does not answer a concrete implementation question.

## Rules

Expand Down
20 changes: 10 additions & 10 deletions plugins/dotnet-test/agents/code-testing-planner.agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,32 +22,32 @@ Read the research document and create a phased implementation plan that will gui

### 1. Read the Research

Read `.testagent/research.md` to understand:
Read the target inventory, command section, dependency summary, and testing conventions from `.testagent/research.md`. Do not reread repository files during planning.

- Project structure and language
- Files that need tests
- Testing framework and patterns
- Build/test commands
- **Dependency graph** (leaf types, mid-layer, top-layer)
- **Estimated coverage** per source file (untested / partially tested / well tested)
- **Coverage classification** per target source file (untested / partial / substantial)

### 2. Choose Strategy Based on Estimated Coverage
### 2. Choose Strategy Based on Coverage Classification

Check the **Estimated Coverage** information in the research:
Check the coverage classification in the research:

**Broad strategy** (most files are untested or estimated coverage is unknown):

- Generate tests for **all** source files systematically
- Generate tests for all files in the bounded target inventory
- Organize into phases by priority and complexity (2-5 phases)
- Every public class and method must have at least one test
- If >15 source files, use more phases (up to 8-10)
- List ALL source files and assign each to a phase
- Assign each target file to exactly one phase

**Targeted strategy** (most files are well tested):
**Targeted strategy** (most targets have substantial existing tests):

- Focus on files estimated as **untested** or **partially tested**
- Prioritize completely untested files, then partially tested files with complex logic
- Put less focus on files estimated as **well tested**
- Put less focus on targets classified as having **substantial** existing tests
- Fewer, more focused phases (1-3)

### 3. Organize into Phases
Expand Down Expand Up @@ -127,14 +127,14 @@ What this phase accomplishes and why it's first.
...
```

> **Concrete example**: For a filled-in plan with real method names, specific test scenarios, and phase structure, call the `code-testing-extensions` skill and read the matching `<language>-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, `java-examples.md` ("Sample Plan Output" section). For other languages, adapt the closest example.
Only consult a language example when research found no existing tests and the base extension does not establish a convention.

## Rules

1. **Be specific** — include exact file paths and method names
2. **Be realistic** — don't plan more than can be implemented
3. **Be incremental** — each phase should be independently valuable
4. **Include patterns** — show code templates for the language
4. **Avoid templates** — reference the concise conventions captured in research instead of embedding example code
5. **Match existing style** — follow patterns from existing tests if any

## Output
Expand Down
Loading
Loading