From 111e8388be9d065b81ea9da276bde7942ca5afcd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Thu, 28 May 2026 18:41:01 +0200 Subject: [PATCH 1/3] dotnet-test: add Rule #0 (Confirm the Test Target) to ruby/powershell extensions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When the prompt does not name a specific file ("test the repository", "one core module", "comprehensive suite"), agents frequently target the wrong code — typically the largest upstream module that already has rich existing tests — instead of the newly-added module the user actually wants tested. Recent benchmark runs (top5-{ruby,powershell}-*-{simple,complex}) showed this is a dominant failure mode for Ruby/PowerShell: the agent burns 50+ turns writing tests for files the verifier never measures, while the real target (a small untracked `lib/string_utils.rb` or `tools/StringUtils.psm1`) sits one `git status` away. Adds a Rule #0 to both ruby.md and powershell.md, ahead of the existing "Rule #1: Investigate the Repo First". The rule: - Tells the agent to use git history (`git status -s`, `git ls-files --others --exclude-standard`, `git log --diff-filter=A --name-only -5`) to find the actual target rather than guessing from repo size. - Documents a Test Placement Contract — RSpec scopes to `spec/`, Pester scopes to whatever directory the harness passes to `Invoke-Pester -Path`; tests placed elsewhere are invisible. - Adds a First-Test Sanity Loop: write one test, run --dry-run / -PassThru to confirm discovery > 0, fix LoadError / Import-Module issues before expanding. Catches placement mistakes on turn 1. Both files validate cleanly under skill-validator. The existing Rule #1 and all subsequent sections are unchanged — this is a pure prepend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../extensions/powershell.md | 44 ++++++++++++++++++ .../extensions/ruby.md | 45 +++++++++++++++++++ 2 files changed, 89 insertions(+) diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md index e0b1201d8d..39b4c6a2fd 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md @@ -2,6 +2,50 @@ Language-specific guidance for PowerShell test generation using Pester v5. +## Rule #0: Confirm the Test Target + +If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of scripts already covered by existing `*.Tests.ps1` files. + +Before planning, run these discovery commands and treat their output as ground truth: + +| Goal | Command | +|------|---------| +| List uncommitted edits + untracked files | `git status -s` | +| Untracked files only (typical for newly-added modules) | `git ls-files --others --exclude-standard` | +| Recently added scripts/modules | `git log --diff-filter=A --name-only -5 -- '*.ps1' '*.psm1' '*.psd1'` | +| Modules with no matching `*.Tests.ps1` | compare `Get-ChildItem -Recurse -Include *.psm1,*.ps1` against `*.Tests.ps1` files | + +Prefer targets that match **all** of: + +1. Untracked or recently added (`git status` / `git log --diff-filter=A`). +2. Small and pure (a few hundred lines, no external state, no `Invoke-WebRequest`/registry/filesystem side effects). +3. Located under a conventional source root (`tools/`, `src/`, `Public/`, `Private/`, or the module root next to a `.psd1`). +4. Have **no** existing matching `*.Tests.ps1` file. + +If a `.psd1` manifest's `FunctionsToExport` list references a specific module path, that module is almost certainly the target — start there. + +### Test Placement Contract + +Pester only discovers tests under the path passed to `Invoke-Pester -Path` (or the current directory when no path is given). Verification harnesses (CI, msbench, coverage tools) typically scope discovery to a single directory such as `tools/` or `tests/`. Place every test file there, matching the existing convention in the repo: + +| Layout used by the repo | Test placement | +|-------------------------|----------------| +| Co-located convention (`Module.psm1` + `Module.Tests.ps1` side-by-side) | Drop `.Tests.ps1` next to the source file (`tools/StringUtils.psm1` → `tools/StringUtils.Tests.ps1`). | +| Sibling `Tests/` directory | Mirror the source path (`src/Foo/Bar.psm1` → `Tests/Foo/Bar.Tests.ps1`). | +| Mixed / unknown | Co-locate next to the source — this is what Pester discovers by default and what most harnesses scope to. | + +A `*.Tests.ps1` file placed outside the discovery root will be invisible to both `Invoke-Pester` and the harness. + +### First-Test Sanity Loop + +After writing the **first** `*.Tests.ps1` file — before writing any others: + +1. Run `Invoke-Pester -Path -PassThru` and confirm the `TotalCount` is `> 0`. If it is `0`, Pester is not discovering your file; fix the location, filename, or `Describe`/`It` structure before continuing. +2. Run the test (`Invoke-Pester -Path -Output Detailed`); fix `Import-Module` / dot-source / `BeforeAll` errors before adding more tests. +3. Only then expand to cover the remaining functions. + +This catches placement and discovery mistakes on turn 1 instead of after dozens of failed-test iterations. + ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby.md index 6307f68351..860eb4d17f 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby.md @@ -2,6 +2,51 @@ Language-specific guidance for Ruby test generation. +## Rule #0: Confirm the Test Target + +If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of modules already covered by existing specs. + +Before planning, run these discovery commands and treat their output as ground truth: + +| Goal | Command | +|------|---------| +| List uncommitted edits + untracked files | `git status -s` | +| Untracked files only (typical for newly-added modules) | `git ls-files --others --exclude-standard` | +| Recently added files under `lib/` or `app/` | `git log --diff-filter=A --name-only -5 -- 'lib/**' 'app/**'` | +| Files referenced by `spec_helper.rb` / `rails_helper.rb` | `grep -nE "^require " spec/spec_helper.rb spec/rails_helper.rb 2>/dev/null` | +| Modules with no matching spec | compare `lib/**/*.rb` against `spec/**/*_spec.rb` paths | + +Prefer targets that match **all** of: + +1. Untracked or recently added (`git status` / `git log --diff-filter=A`). +2. Small and pure (a few hundred lines, no I/O, no global state). +3. Located under a conventional source root (`lib/`, `app/models/`, `app/services/`). +4. Have **no** existing matching `*_spec.rb` / `*_test.rb`. + +If `spec/spec_helper.rb` already `require`s one specific file (e.g. `require "string_utils"`), that file is almost certainly the target — start there. + +### Test Placement Contract + +RSpec only discovers specs under `spec/` by default, and verification harnesses (CI, msbench, coverage tools) typically scope discovery to `spec/` alone. Place every spec there, mirroring the source layout: + +| Source | Spec | +|--------|------| +| `lib/string_utils.rb` | `spec/string_utils_spec.rb` | +| `lib/foo/bar.rb` | `spec/foo/bar_spec.rb` | +| `app/models/user.rb` (Rails) | `spec/models/user_spec.rb` | + +A spec placed anywhere outside `spec/` (e.g. next to the source under `lib/`) will be invisible to `bundle exec rspec` and to the harness. The same applies to Minitest: place tests under `test/` and use `*_test.rb` naming. + +### First-Test Sanity Loop + +After writing the **first** spec — before writing any others: + +1. Run `bundle exec rspec --dry-run` and confirm the example count is `> 0`. If it is `0`, RSpec is not seeing your file; fix the location, filename, or `$LOAD_PATH` before continuing. +2. Run the spec (`bundle exec rspec spec/.rb`); fix `LoadError`, missing `require`, or constant errors before adding more tests. +3. Only then expand to cover the remaining methods. + +This catches placement and load-path mistakes on turn 1 instead of after dozens of failed-test iterations. + ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: From 3e4b0e6e67b21305afd20de868a17919e9b9cc6c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Thu, 28 May 2026 19:10:24 +0200 Subject: [PATCH 2/3] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .../skills/code-testing-extensions/extensions/powershell.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md index 39b4c6a2fd..752f04f306 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md @@ -22,7 +22,7 @@ Prefer targets that match **all** of: 3. Located under a conventional source root (`tools/`, `src/`, `Public/`, `Private/`, or the module root next to a `.psd1`). 4. Have **no** existing matching `*.Tests.ps1` file. -If a `.psd1` manifest's `FunctionsToExport` list references a specific module path, that module is almost certainly the target — start there. +If a `.psd1` manifest's `RootModule` (or `ModuleToProcess`) points at a specific `.psm1`, that module is almost certainly the target — start there. ### Test Placement Contract From a749d2f7fe715a37065e1e39c5c9ac83a216886e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Sun, 31 May 2026 20:26:58 +0200 Subject: [PATCH 3/3] Address PR review: clarify Rule #0 precedence and broaden require regex - Rephrase Rule #0's discovery preamble in ruby.md and powershell.md to call out the commands as the read-only exception to Rule #1, removing the apparent contradiction between 'before planning' and Rule #1's 'before writing any test or running any command'. - Broaden the spec_helper require grep to match leading whitespace and `require_relative`, avoiding false negatives that send the agent to the wrong target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../skills/code-testing-extensions/extensions/powershell.md | 2 +- .../skills/code-testing-extensions/extensions/ruby.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md index 752f04f306..afb3c0e37f 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md @@ -6,7 +6,7 @@ Language-specific guidance for PowerShell test generation using Pester v5. If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of scripts already covered by existing `*.Tests.ps1` files. -Before planning, run these discovery commands and treat their output as ground truth: +Run these **read-only** discovery commands first — they are the deliberate exception to Rule #1's "before writing any test or running any command" rule, and their output is the ground truth Rule #1's reading is meant to interpret. Do **not** write or execute any tests until Rule #0 and Rule #1 are both complete. | Goal | Command | |------|---------| diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby.md index 860eb4d17f..54c29fd8d5 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/ruby.md @@ -6,14 +6,14 @@ Language-specific guidance for Ruby test generation. If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of modules already covered by existing specs. -Before planning, run these discovery commands and treat their output as ground truth: +Run these **read-only** discovery commands first — they are the deliberate exception to Rule #1's "before writing any test or running any command" rule, and their output is the ground truth Rule #1's reading is meant to interpret. Do **not** write or execute any tests until Rule #0 and Rule #1 are both complete. | Goal | Command | |------|---------| | List uncommitted edits + untracked files | `git status -s` | | Untracked files only (typical for newly-added modules) | `git ls-files --others --exclude-standard` | | Recently added files under `lib/` or `app/` | `git log --diff-filter=A --name-only -5 -- 'lib/**' 'app/**'` | -| Files referenced by `spec_helper.rb` / `rails_helper.rb` | `grep -nE "^require " spec/spec_helper.rb spec/rails_helper.rb 2>/dev/null` | +| Files referenced by `spec_helper.rb` / `rails_helper.rb` | `grep -nE "^\s*require(_relative)?\s" spec/spec_helper.rb spec/rails_helper.rb 2>/dev/null` | | Modules with no matching spec | compare `lib/**/*.rb` against `spec/**/*_spec.rb` paths | Prefer targets that match **all** of: