Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions plugins/dotnet-test/skills/code-testing-extensions/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ This skill provides access to language-specific guidance files used by the code-
| File | Language | Contents |
|------|----------|----------|
| [extensions/dotnet.md](extensions/dotnet.md) | .NET (C#/F#/VB) | Build commands, test commands, project reference validation, common CS error codes, MSTest template |
| [extensions/python.md](extensions/python.md) | Python | Framework-adaptive test commands (pytest, custom runners), project layout detection, mocking guidelines, common errors |
| [extensions/typescript.md](extensions/typescript.md) | TypeScript/JavaScript | Build/test commands (Jest/Vitest/Mocha), framework detection, mocking, TS-specific considerations |
| [extensions/powershell.md](extensions/powershell.md) | PowerShell | Test commands (Pester v5), module import patterns, discovery/run pitfalls, mocking, common errors |
| [extensions/cpp.md](extensions/cpp.md) | C++ | Testing internals with friend declarations |
| [extensions/dotnet-examples.md](extensions/dotnet-examples.md) | .NET (C#/F#/VB) | Concrete pipeline examples: sample research output, plan, generated tests, fix cycles, final report |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,18 @@ If a new test project was created, register it with the solution so `dotnet test
4. Skip this if the project is already included in the solution or solution filter used for testing.
5. Prefer the researched test command. If you need to run the solution directly, use `dotnet test --solution <solution>` only for repos on .NET SDK 10+ with MTP-style syntax; otherwise use the standard positional form `dotnet test <solution>`.

## Test Framework Detection

Detect the framework from the test project's `.csproj` package references and match its conventions:

| Package Reference | Framework | Attributes | Assertion Style |
|-------------------|-----------|------------|-----------------|
| `MSTest.Sdk` or `MSTest.TestFramework` | MSTest | `[TestClass]`, `[TestMethod]`, `[DataRow]` | `Assert.AreEqual(expected, actual)` |
| `xunit` | xUnit | `[Fact]`, `[Theory]`, `[InlineData]` | `Assert.Equal(expected, actual)` |
| `NUnit` | NUnit | `[TestFixture]`, `[Test]`, `[TestCase]` | `Assert.That(actual, Is.EqualTo(expected))` |

Use the repo's existing framework — do not introduce a different one.

## MSTest Template

```csharp
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# PowerShell Extension

Language-specific guidance for PowerShell test generation using Pester v5.

## Rule #1: Investigate the Repo First

Before writing any test or running any command, read:

1. **Existing tests** — find `*.Tests.ps1` files and copy their style (structure, assertions, mock approach, import method)
2. **Module structure** — look for `.psd1` (manifest), `.psm1` (root module), `Public/`/`Private/` organization
3. **Build/test scripts** — check for `build.ps1`, `Invoke-Build` (`*.build.ps1`), `psake`, or CI scripts
4. **Shell target** — check `.psd1` for `PowerShellVersion`/`CompatiblePSEditions`, CI matrix for `pwsh` vs `powershell.exe`

Use the repo's existing test conventions. Only add Pester if the repo has no tests at all.

## Build Commands

PowerShell is interpreted — no build step. If the repo has a build script, use it. Otherwise validate with:

- **Module loads**: `Import-Module ./MyModule.psd1 -Force -ErrorAction Stop`
- **Script analyzer**: `Invoke-ScriptAnalyzer -Path ./src -Recurse` (if PSScriptAnalyzer is available)
- **Lint**: `Invoke-ScriptAnalyzer -Path path/to/file.ps1 -Fix`

## Test Commands

| Scope | Command |
|-------|---------|
| All tests | `Invoke-Pester` |
| Specific file | `Invoke-Pester -Path ./Tests/Get-Widget.Tests.ps1` |
| Filter by name | `Invoke-Pester -FullNameFilter '*Get-Widget*'` |
| Filter by tag | `Invoke-Pester -TagFilter 'Unit'` |
| Non-interactive (CI) | `Invoke-Pester -CI` |
| Detailed output | `Invoke-Pester -Output Detailed` |

- Prefer the repo's build/test script over raw `Invoke-Pester`
- Use `-Output Detailed` during fix cycles, `-Output Minimal` for final validation

## Project Layout and Imports

| Layout | Import in `BeforeAll` |
|--------|-----------------------|
| Module (`.psd1`) | `Import-Module "$PSScriptRoot/../MyModule.psd1" -Force` |
| Library script (defines functions) | `. $PSScriptRoot/Get-Widget.ps1` |
| Co-located test | `. $PSCommandPath.Replace('.Tests.ps1', '.ps1')` |
| Executable script (has `param()`) | Do **not** dot-source — invoke with `& $PSScriptRoot/script.ps1 -Param value` and assert on output/errors |

- **All imports go in `BeforeAll`** — never at script top level
- **Use `$PSScriptRoot` or `$PSCommandPath`** — never `$MyInvocation.MyCommand.Path` (returns empty in `BeforeAll`)
- Use `-Force` on `Import-Module` to pick up changes between runs

## Test File Naming

- Files: `*.Tests.ps1` — match existing convention (co-located vs `Tests/` directory)

## Pester v5 Discovery vs Run (Critical)

Pester v5 runs in **two phases**: Discovery (collects test metadata) then Run (executes tests). This is the #1 source of agent errors.

**Rules:**
- All setup code goes in `BeforeAll` or `BeforeEach` — never at script top level or loose inside `Describe`/`Context`
- Code directly inside `Describe`/`Context` (but outside `It`/`Before*`/`After*`) runs during **Discovery** — do not put setup, imports, or variable assignments there
- Data for `-ForEach` / `-TestCases` must be set in `BeforeDiscovery`, not `BeforeAll` (BeforeAll runs after discovery)
- `-Skip:$condition` evaluates at Discovery time — conditions from `BeforeAll` will be `$null`
- Use `foreach` loops for dynamic test generation only with `BeforeDiscovery` data
- Use `TestDrive:` for file-based tests instead of touching repo files — Pester cleans it up automatically

## Common Errors

| Error | Fix |
|-------|-----|
| Variable is `$null` in `It` block | Move assignment into `BeforeAll` — variables set there are visible to child `It` blocks without `$script:` |
| `-ForEach` data is empty | Move data setup from `BeforeAll` to `BeforeDiscovery` |
| `CommandNotFoundException` for Mock target | The function must exist before mocking — import the module in `BeforeAll` first |
| `$MyInvocation.MyCommand.Path` returns empty | Use `$PSCommandPath` or `$PSScriptRoot` instead |
| `Should Be` (no dash) fails | Use v5 syntax: `Should -Be` (with dash prefix) |
| `Assert-MockCalled` not recognized | Use v5 syntax: `Should -Invoke` |
| Mock has no effect | Check scope — mocks in `It` only apply to that `It`; use `BeforeAll`/`BeforeEach` for broader scope |
| `Should -Throw` doesn't catch cmdlet errors | Most cmdlet errors are non-terminating — wrap with `{ cmd -ErrorAction Stop }` or set `$ErrorActionPreference = 'Stop'` in `BeforeEach` |
| Tests pass on Windows but fail on Linux | Use `Join-Path` not string concatenation; match exact file casing; avoid Windows-only cmdlets (Registry, EventLog) |

## Mocking Rules

- Place mocks in `BeforeAll` (shared) or `BeforeEach` (reset per test)
- Mock where the command is **called from** — use `-ModuleName` to mock inside a module's scope
- Use `-ParameterFilter` for selective mocking (no `param()` block needed in v5)
- Verify calls with `Should -Invoke` — default scope inside `It` counts only that test's calls
- Use `InModuleScope` sparingly and as narrowly as possible — prefer `Mock -ModuleName` for testing via public API
- Inside mock bodies, use `$PesterBoundParameters` not `$PSBoundParameters`
- If a test needs more than 3 mocks, flag it as a design smell

## Non-Obvious Assertions

Most `Should` operators are self-explanatory. These are the ones agents get wrong:

- `Should -Throw` requires a **scriptblock**: `{ risky-op } | Should -Throw` — not a direct call
- `Should -Contain` is for **collections** — use `Should -Be` for scalar equality
- `Should -HaveParameter` validates cmdlet signatures: `Get-Command X | Should -HaveParameter 'Name' -Mandatory`
- `Should -Invoke` verifies mock calls: `Should -Invoke Get-Item -Times 1 -Exactly`

## Cross-Platform

- Prefer `pwsh` (PowerShell 7+) unless the repo explicitly targets Windows PowerShell 5.1
- Use `Join-Path` for paths — never string concatenation with `\`
- Linux/macOS file systems are **case-sensitive** — match exact casing in imports and paths
- Windows ships Pester 3.4.0 — if v5 is needed: `Install-Module Pester -Force -SkipPublisherCheck`
- Check `$PSVersionTable.PSEdition` to detect Core vs Desktop

## Skip Coverage Tools

Do not configure or run coverage tools (Pester CodeCoverage, JaCoCo export). Coverage is measured separately by the evaluation harness.
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Python Extension

Language-specific guidance for Python test generation.

## Rule #1: Investigate the Repo First

Before writing any test or running any command, discover what the repo already does:

1. **Find ALL existing test files** — search broadly: `test_*.py`, `*_test.py`, `*.uts`, `test/*.sh`, or any other test format. Do not assume pytest.
2. **Identify the test framework** — look for:
- Custom test runners (e.g. `UTscapy` for scapy, project-specific harnesses)
- Standard frameworks (`pytest`, `unittest`, `nose2`)
- Test runner scripts in `Makefile`, `tox.ini`, `nox`, `scripts/`
- Config entries in `pyproject.toml`, `setup.cfg`, `pytest.ini`, `conftest.py`
3. **Read existing tests thoroughly** — copy their exact style: file format, imports, fixtures, assertion patterns, helper utilities, setup/teardown conventions
4. **Package layout** — determine import paths from existing code, not guesswork

**Use whatever framework and conventions the repo already uses.** If the repo uses a custom test framework (custom file formats, custom runners, domain-specific test utilities), adopt it fully — do not layer pytest on top. Only introduce pytest if the repo has no tests at all.

## Environment Detection

Detect the runner from lockfiles/config and prefix all commands accordingly:

| Indicator | Prefix |
|-----------|--------|
| `poetry.lock` / `[tool.poetry]` in `pyproject.toml` | `poetry run` |
| `pdm.lock` / `[tool.pdm]` in `pyproject.toml` | `pdm run` |
| `uv.lock` / `[tool.uv]` in `pyproject.toml` | `uv run` |
| `Pipfile.lock` | `pipenv run` |
| `hatch.toml` / `[tool.hatch]` in `pyproject.toml` | `hatch run` |
| None of the above | `python -m` |

If `Makefile`, `tox.ini`, or `nox` config exists, prefer those scripts over raw commands.

## Build Commands

Python has no separate build step. Validate with the type checker if one is configured:

| Scope | Command |
|-------|---------|
| Syntax check | `<prefix> py_compile path/to/file.py` |
| Type check | `<prefix> mypy path/to/file.py` or `<prefix> pyright path/to/file.py` |

## Test Commands

If the repo uses a **custom test framework** (custom file formats, custom runner), use its native commands — do not wrap them in pytest. Examples:

| Framework | Command |
|-----------|---------|
| UTscapy (`.uts` files) | `<prefix> scapy.tools.UTscapy -f test/test_file.uts` |
| Custom runner script | `make test`, `./run_tests.sh`, `tox` |
| Repo-defined script | Whatever `scripts.test` in Makefile/tox/nox specifies |

For **pytest** projects (the most common case), use the detected `<prefix>`:

| Scope | Command |
|-------|---------|
| All tests | `<prefix> pytest` |
| Specific file | `<prefix> pytest tests/test_module.py` |
| Specific test | `<prefix> pytest tests/test_module.py::TestClass::test_method` |
| Keyword filter | `<prefix> pytest -k "keyword"` |
| Stop on first failure | `<prefix> pytest -x --tb=short` |

- Prefer `python -m pytest` over bare `pytest` to ensure the correct interpreter
- If the project uses `unittest` only (no pytest in deps), use `python -m unittest discover`

## Lint Command

Use the repo's existing lint script first (`make lint`, `tox -e lint`). Otherwise detect tools from config:

- `ruff.toml` or `[tool.ruff]` → `<prefix> ruff check --fix && <prefix> ruff format`
- `[tool.black]` → `<prefix> black`
- `.flake8` → `<prefix> flake8`

## Project Layout and Imports

| Layout | Import Style |
|--------|-------------|
| `src/package/module.py` | `from package.module import X` |
| `package/module.py` at root | `from package.module import X` |
| `module.py` at root | `from module import X` |

- **Match existing test imports exactly** — do not invent `src.` prefixes unless existing tests use them
- Check `pyproject.toml` `[tool.setuptools.package-dir]` for layout hints
- Default test placement: `tests/` mirroring source structure (`src/billing/service.py` → `tests/billing/test_service.py`)

## Test File Naming

Match the repo's existing conventions. Common patterns:

- **pytest**: Files `test_*.py` or `*_test.py`, functions `test_` prefix, classes `Test` prefix
- **Custom frameworks**: Use whatever format existing tests use (e.g. `.uts` for UTscapy, custom extensions)

If writing new tests in a repo with no tests, default to pytest conventions.

## Common Errors

| Error | Fix |
|-------|-----|
| `ModuleNotFoundError: No module named 'src'` | Import from the package name used by the repo, not from `src` |
| `ModuleNotFoundError: No module named 'X'` | Check existing imports for the correct package name; if editable install needed: `<prefix> pip install -e .` |
| `ImportError: attempted relative import` | Convert to absolute imports matching existing test patterns |
| `fixture 'X' not found` | Check `conftest.py` for existing fixtures; reuse them instead of creating new ones |
| `TypeError: missing required argument` | Read the full `__init__`/function signature; pass all required parameters |
| `async def functions are not natively supported` | Use `@pytest.mark.asyncio` only if `pytest-asyncio` is already in deps; check for `asyncio_mode = "auto"` in config |
| `SyntaxError` | Fix syntax at the indicated line |

## Mocking Rules

- Use `unittest.mock` (stdlib) — no extra dependency needed
- **Patch where the name is looked up**, not where it is defined: `@patch("mypackage.module.datetime")` not `@patch("datetime.datetime")`
- Use `Mock(spec=RealClass)` to catch attribute errors
- Use `AsyncMock` for async functions
- Prefer dependency injection over `@patch`
- If a test needs more than 3 mocks, flag it as a design smell

## Dependency Installation (Last Resort)

Only install packages after investigation confirms they are missing. Use the detected prefix:

| Manager | Install command |
|---------|----------------|
| Poetry | `poetry add --group dev pytest` |
| PDM | `pdm add -dG test pytest` |
| uv | `uv add --dev pytest` |
| pip | `python -m pip install -e ".[dev]"` |

Never run bare `pip install` in a Poetry/PDM/uv project — it bypasses the lockfile.

## Skip Coverage Tools

Do not configure or run coverage tools (coverage.py, pytest-cov). Coverage is measured separately by the evaluation harness.
Loading
Loading