diff --git a/plugins/dotnet-test/skills/code-testing-extensions/SKILL.md b/plugins/dotnet-test/skills/code-testing-extensions/SKILL.md index cd74bfee2d..5d7510917a 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/SKILL.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/SKILL.md @@ -18,6 +18,9 @@ This skill provides access to language-specific guidance files used by the code- | File | Language | Contents | |------|----------|----------| | [extensions/dotnet.md](extensions/dotnet.md) | .NET (C#/F#/VB) | Build commands, test commands, project reference validation, common CS error codes, MSTest template | +| [extensions/python.md](extensions/python.md) | Python | Framework-adaptive test commands (pytest, custom runners), project layout detection, mocking guidelines, common errors | +| [extensions/typescript.md](extensions/typescript.md) | TypeScript/JavaScript | Build/test commands (Jest/Vitest/Mocha), framework detection, mocking, TS-specific considerations | +| [extensions/powershell.md](extensions/powershell.md) | PowerShell | Test commands (Pester v5), module import patterns, discovery/run pitfalls, mocking, common errors | | [extensions/cpp.md](extensions/cpp.md) | C++ | Testing internals with friend declarations | | [extensions/dotnet-examples.md](extensions/dotnet-examples.md) | .NET (C#/F#/VB) | Concrete pipeline examples: sample research output, plan, generated tests, fix cycles, final report | diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet.md index 7c30a78174..019c76b179 100644 --- a/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet.md +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/dotnet.md @@ -71,6 +71,18 @@ If a new test project was created, register it with the solution so `dotnet test 4. Skip this if the project is already included in the solution or solution filter used for testing. 5. Prefer the researched test command. If you need to run the solution directly, use `dotnet test --solution ` only for repos on .NET SDK 10+ with MTP-style syntax; otherwise use the standard positional form `dotnet test `. +## Test Framework Detection + +Detect the framework from the test project's `.csproj` package references and match its conventions: + +| Package Reference | Framework | Attributes | Assertion Style | +|-------------------|-----------|------------|-----------------| +| `MSTest.Sdk` or `MSTest.TestFramework` | MSTest | `[TestClass]`, `[TestMethod]`, `[DataRow]` | `Assert.AreEqual(expected, actual)` | +| `xunit` | xUnit | `[Fact]`, `[Theory]`, `[InlineData]` | `Assert.Equal(expected, actual)` | +| `NUnit` | NUnit | `[TestFixture]`, `[Test]`, `[TestCase]` | `Assert.That(actual, Is.EqualTo(expected))` | + +Use the repo's existing framework — do not introduce a different one. + ## MSTest Template ```csharp diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md new file mode 100644 index 0000000000..e0b1201d8d --- /dev/null +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/powershell.md @@ -0,0 +1,110 @@ +# PowerShell Extension + +Language-specific guidance for PowerShell test generation using Pester v5. + +## Rule #1: Investigate the Repo First + +Before writing any test or running any command, read: + +1. **Existing tests** — find `*.Tests.ps1` files and copy their style (structure, assertions, mock approach, import method) +2. **Module structure** — look for `.psd1` (manifest), `.psm1` (root module), `Public/`/`Private/` organization +3. **Build/test scripts** — check for `build.ps1`, `Invoke-Build` (`*.build.ps1`), `psake`, or CI scripts +4. **Shell target** — check `.psd1` for `PowerShellVersion`/`CompatiblePSEditions`, CI matrix for `pwsh` vs `powershell.exe` + +Use the repo's existing test conventions. Only add Pester if the repo has no tests at all. + +## Build Commands + +PowerShell is interpreted — no build step. If the repo has a build script, use it. Otherwise validate with: + +- **Module loads**: `Import-Module ./MyModule.psd1 -Force -ErrorAction Stop` +- **Script analyzer**: `Invoke-ScriptAnalyzer -Path ./src -Recurse` (if PSScriptAnalyzer is available) +- **Lint**: `Invoke-ScriptAnalyzer -Path path/to/file.ps1 -Fix` + +## Test Commands + +| Scope | Command | +|-------|---------| +| All tests | `Invoke-Pester` | +| Specific file | `Invoke-Pester -Path ./Tests/Get-Widget.Tests.ps1` | +| Filter by name | `Invoke-Pester -FullNameFilter '*Get-Widget*'` | +| Filter by tag | `Invoke-Pester -TagFilter 'Unit'` | +| Non-interactive (CI) | `Invoke-Pester -CI` | +| Detailed output | `Invoke-Pester -Output Detailed` | + +- Prefer the repo's build/test script over raw `Invoke-Pester` +- Use `-Output Detailed` during fix cycles, `-Output Minimal` for final validation + +## Project Layout and Imports + +| Layout | Import in `BeforeAll` | +|--------|-----------------------| +| Module (`.psd1`) | `Import-Module "$PSScriptRoot/../MyModule.psd1" -Force` | +| Library script (defines functions) | `. $PSScriptRoot/Get-Widget.ps1` | +| Co-located test | `. $PSCommandPath.Replace('.Tests.ps1', '.ps1')` | +| Executable script (has `param()`) | Do **not** dot-source — invoke with `& $PSScriptRoot/script.ps1 -Param value` and assert on output/errors | + +- **All imports go in `BeforeAll`** — never at script top level +- **Use `$PSScriptRoot` or `$PSCommandPath`** — never `$MyInvocation.MyCommand.Path` (returns empty in `BeforeAll`) +- Use `-Force` on `Import-Module` to pick up changes between runs + +## Test File Naming + +- Files: `*.Tests.ps1` — match existing convention (co-located vs `Tests/` directory) + +## Pester v5 Discovery vs Run (Critical) + +Pester v5 runs in **two phases**: Discovery (collects test metadata) then Run (executes tests). This is the #1 source of agent errors. + +**Rules:** +- All setup code goes in `BeforeAll` or `BeforeEach` — never at script top level or loose inside `Describe`/`Context` +- Code directly inside `Describe`/`Context` (but outside `It`/`Before*`/`After*`) runs during **Discovery** — do not put setup, imports, or variable assignments there +- Data for `-ForEach` / `-TestCases` must be set in `BeforeDiscovery`, not `BeforeAll` (BeforeAll runs after discovery) +- `-Skip:$condition` evaluates at Discovery time — conditions from `BeforeAll` will be `$null` +- Use `foreach` loops for dynamic test generation only with `BeforeDiscovery` data +- Use `TestDrive:` for file-based tests instead of touching repo files — Pester cleans it up automatically + +## Common Errors + +| Error | Fix | +|-------|-----| +| Variable is `$null` in `It` block | Move assignment into `BeforeAll` — variables set there are visible to child `It` blocks without `$script:` | +| `-ForEach` data is empty | Move data setup from `BeforeAll` to `BeforeDiscovery` | +| `CommandNotFoundException` for Mock target | The function must exist before mocking — import the module in `BeforeAll` first | +| `$MyInvocation.MyCommand.Path` returns empty | Use `$PSCommandPath` or `$PSScriptRoot` instead | +| `Should Be` (no dash) fails | Use v5 syntax: `Should -Be` (with dash prefix) | +| `Assert-MockCalled` not recognized | Use v5 syntax: `Should -Invoke` | +| Mock has no effect | Check scope — mocks in `It` only apply to that `It`; use `BeforeAll`/`BeforeEach` for broader scope | +| `Should -Throw` doesn't catch cmdlet errors | Most cmdlet errors are non-terminating — wrap with `{ cmd -ErrorAction Stop }` or set `$ErrorActionPreference = 'Stop'` in `BeforeEach` | +| Tests pass on Windows but fail on Linux | Use `Join-Path` not string concatenation; match exact file casing; avoid Windows-only cmdlets (Registry, EventLog) | + +## Mocking Rules + +- Place mocks in `BeforeAll` (shared) or `BeforeEach` (reset per test) +- Mock where the command is **called from** — use `-ModuleName` to mock inside a module's scope +- Use `-ParameterFilter` for selective mocking (no `param()` block needed in v5) +- Verify calls with `Should -Invoke` — default scope inside `It` counts only that test's calls +- Use `InModuleScope` sparingly and as narrowly as possible — prefer `Mock -ModuleName` for testing via public API +- Inside mock bodies, use `$PesterBoundParameters` not `$PSBoundParameters` +- If a test needs more than 3 mocks, flag it as a design smell + +## Non-Obvious Assertions + +Most `Should` operators are self-explanatory. These are the ones agents get wrong: + +- `Should -Throw` requires a **scriptblock**: `{ risky-op } | Should -Throw` — not a direct call +- `Should -Contain` is for **collections** — use `Should -Be` for scalar equality +- `Should -HaveParameter` validates cmdlet signatures: `Get-Command X | Should -HaveParameter 'Name' -Mandatory` +- `Should -Invoke` verifies mock calls: `Should -Invoke Get-Item -Times 1 -Exactly` + +## Cross-Platform + +- Prefer `pwsh` (PowerShell 7+) unless the repo explicitly targets Windows PowerShell 5.1 +- Use `Join-Path` for paths — never string concatenation with `\` +- Linux/macOS file systems are **case-sensitive** — match exact casing in imports and paths +- Windows ships Pester 3.4.0 — if v5 is needed: `Install-Module Pester -Force -SkipPublisherCheck` +- Check `$PSVersionTable.PSEdition` to detect Core vs Desktop + +## Skip Coverage Tools + +Do not configure or run coverage tools (Pester CodeCoverage, JaCoCo export). Coverage is measured separately by the evaluation harness. diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/python.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/python.md new file mode 100644 index 0000000000..6584e520a8 --- /dev/null +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/python.md @@ -0,0 +1,132 @@ +# Python Extension + +Language-specific guidance for Python test generation. + +## Rule #1: Investigate the Repo First + +Before writing any test or running any command, discover what the repo already does: + +1. **Find ALL existing test files** — search broadly: `test_*.py`, `*_test.py`, `*.uts`, `test/*.sh`, or any other test format. Do not assume pytest. +2. **Identify the test framework** — look for: + - Custom test runners (e.g. `UTscapy` for scapy, project-specific harnesses) + - Standard frameworks (`pytest`, `unittest`, `nose2`) + - Test runner scripts in `Makefile`, `tox.ini`, `nox`, `scripts/` + - Config entries in `pyproject.toml`, `setup.cfg`, `pytest.ini`, `conftest.py` +3. **Read existing tests thoroughly** — copy their exact style: file format, imports, fixtures, assertion patterns, helper utilities, setup/teardown conventions +4. **Package layout** — determine import paths from existing code, not guesswork + +**Use whatever framework and conventions the repo already uses.** If the repo uses a custom test framework (custom file formats, custom runners, domain-specific test utilities), adopt it fully — do not layer pytest on top. Only introduce pytest if the repo has no tests at all. + +## Environment Detection + +Detect the runner from lockfiles/config and prefix all commands accordingly: + +| Indicator | Prefix | +|-----------|--------| +| `poetry.lock` / `[tool.poetry]` in `pyproject.toml` | `poetry run` | +| `pdm.lock` / `[tool.pdm]` in `pyproject.toml` | `pdm run` | +| `uv.lock` / `[tool.uv]` in `pyproject.toml` | `uv run` | +| `Pipfile.lock` | `pipenv run` | +| `hatch.toml` / `[tool.hatch]` in `pyproject.toml` | `hatch run` | +| None of the above | `python -m` | + +If `Makefile`, `tox.ini`, or `nox` config exists, prefer those scripts over raw commands. + +## Build Commands + +Python has no separate build step. Validate with the type checker if one is configured: + +| Scope | Command | +|-------|---------| +| Syntax check | ` py_compile path/to/file.py` | +| Type check | ` mypy path/to/file.py` or ` pyright path/to/file.py` | + +## Test Commands + +If the repo uses a **custom test framework** (custom file formats, custom runner), use its native commands — do not wrap them in pytest. Examples: + +| Framework | Command | +|-----------|---------| +| UTscapy (`.uts` files) | ` scapy.tools.UTscapy -f test/test_file.uts` | +| Custom runner script | `make test`, `./run_tests.sh`, `tox` | +| Repo-defined script | Whatever `scripts.test` in Makefile/tox/nox specifies | + +For **pytest** projects (the most common case), use the detected ``: + +| Scope | Command | +|-------|---------| +| All tests | ` pytest` | +| Specific file | ` pytest tests/test_module.py` | +| Specific test | ` pytest tests/test_module.py::TestClass::test_method` | +| Keyword filter | ` pytest -k "keyword"` | +| Stop on first failure | ` pytest -x --tb=short` | + +- Prefer `python -m pytest` over bare `pytest` to ensure the correct interpreter +- If the project uses `unittest` only (no pytest in deps), use `python -m unittest discover` + +## Lint Command + +Use the repo's existing lint script first (`make lint`, `tox -e lint`). Otherwise detect tools from config: + +- `ruff.toml` or `[tool.ruff]` → ` ruff check --fix && ruff format` +- `[tool.black]` → ` black` +- `.flake8` → ` flake8` + +## Project Layout and Imports + +| Layout | Import Style | +|--------|-------------| +| `src/package/module.py` | `from package.module import X` | +| `package/module.py` at root | `from package.module import X` | +| `module.py` at root | `from module import X` | + +- **Match existing test imports exactly** — do not invent `src.` prefixes unless existing tests use them +- Check `pyproject.toml` `[tool.setuptools.package-dir]` for layout hints +- Default test placement: `tests/` mirroring source structure (`src/billing/service.py` → `tests/billing/test_service.py`) + +## Test File Naming + +Match the repo's existing conventions. Common patterns: + +- **pytest**: Files `test_*.py` or `*_test.py`, functions `test_` prefix, classes `Test` prefix +- **Custom frameworks**: Use whatever format existing tests use (e.g. `.uts` for UTscapy, custom extensions) + +If writing new tests in a repo with no tests, default to pytest conventions. + +## Common Errors + +| Error | Fix | +|-------|-----| +| `ModuleNotFoundError: No module named 'src'` | Import from the package name used by the repo, not from `src` | +| `ModuleNotFoundError: No module named 'X'` | Check existing imports for the correct package name; if editable install needed: ` pip install -e .` | +| `ImportError: attempted relative import` | Convert to absolute imports matching existing test patterns | +| `fixture 'X' not found` | Check `conftest.py` for existing fixtures; reuse them instead of creating new ones | +| `TypeError: missing required argument` | Read the full `__init__`/function signature; pass all required parameters | +| `async def functions are not natively supported` | Use `@pytest.mark.asyncio` only if `pytest-asyncio` is already in deps; check for `asyncio_mode = "auto"` in config | +| `SyntaxError` | Fix syntax at the indicated line | + +## Mocking Rules + +- Use `unittest.mock` (stdlib) — no extra dependency needed +- **Patch where the name is looked up**, not where it is defined: `@patch("mypackage.module.datetime")` not `@patch("datetime.datetime")` +- Use `Mock(spec=RealClass)` to catch attribute errors +- Use `AsyncMock` for async functions +- Prefer dependency injection over `@patch` +- If a test needs more than 3 mocks, flag it as a design smell + +## Dependency Installation (Last Resort) + +Only install packages after investigation confirms they are missing. Use the detected prefix: + +| Manager | Install command | +|---------|----------------| +| Poetry | `poetry add --group dev pytest` | +| PDM | `pdm add -dG test pytest` | +| uv | `uv add --dev pytest` | +| pip | `python -m pip install -e ".[dev]"` | + +Never run bare `pip install` in a Poetry/PDM/uv project — it bypasses the lockfile. + +## Skip Coverage Tools + +Do not configure or run coverage tools (coverage.py, pytest-cov). Coverage is measured separately by the evaluation harness. diff --git a/plugins/dotnet-test/skills/code-testing-extensions/extensions/typescript.md b/plugins/dotnet-test/skills/code-testing-extensions/extensions/typescript.md new file mode 100644 index 0000000000..de26a70ab9 --- /dev/null +++ b/plugins/dotnet-test/skills/code-testing-extensions/extensions/typescript.md @@ -0,0 +1,136 @@ +# TypeScript Extension + +Language-specific guidance for TypeScript (and JavaScript) test generation. + +## Rule #1: Investigate the Repo First + +Before writing any test or running any command, read: + +1. **Existing tests** — find `*.test.ts` / `*.spec.ts` files and copy their style (imports, describe/it vs test, assertion patterns, mock approach) +2. **`package.json`** — `scripts.test`, `devDependencies`, `type` field +3. **Config files** — `tsconfig.json`, `jest.config.*`, `vitest.config.*`, `eslint.config.*` + +Use the repo's existing test runner and conventions — do not switch frameworks. If multiple runners are configured, follow whichever `scripts.test` invokes. Only introduce a framework if the repo has no tests at all. + +## Package Manager Detection + +Detect the package manager from lockfiles and use it consistently for **all** commands: + +| Indicator | Manager | Run script | Execute binary | +|-----------|---------|------------|----------------| +| `pnpm-lock.yaml` | pnpm | `pnpm test` | `pnpm exec ` | +| `yarn.lock` | Yarn | `yarn test` | `yarn ` | +| `bun.lockb` / `bun.lock` | Bun | `bun test` | `bunx ` | +| `package-lock.json` or none | npm | `npm test` | `npx ` | + +Use `` below as shorthand for the detected exec command. + +## Build Commands + +| Scope | Command | +|-------|---------| +| Type check | ` tsc --noEmit` or the repo's `typecheck` script | +| Build (if configured) | The repo's `build` script | + +Many projects don't need an explicit build step — the test runner handles transpilation. + +## Test Commands + +Detect the runner from `devDependencies` and `scripts.test`. Always prefer the repo's test script first. + +| Runner | Run once | Filter by file | Filter by name | +|--------|----------|----------------|----------------| +| **Jest** | ` jest` | ` jest path/to/file` | ` jest -t "name"` | +| **Vitest** | ` vitest run` | ` vitest run path/to/file` | ` vitest run -t "name"` | +| **Mocha** | ` mocha` | (use config or positional args) | ` mocha --grep "name"` | + +- **Always use `vitest run`** (not bare `vitest`) — bare `vitest` starts watch mode +- **Never use `--watch`** — the agent must not start interactive/watch mode +- For Jest: `--bail` to stop on first failure, `--verbose` for detail +- Mocha `--grep` filters by **test name**, not file path + +## Lint Command + +Use the repo's lint script first. Otherwise detect from `devDependencies` and config: + +- `eslint.config.*` or `.eslintrc.*` → ` eslint --fix path/to/file.ts` +- `prettier` → ` prettier --write path/to/file.ts` +- `biome.json` → ` biome check --write path/to/file.ts` + +## Project Layout and Imports + +| Layout | Import Style | +|--------|-------------| +| Colocated (`src/module.test.ts`) | `import { X } from './module'` | +| `__tests__/` dir | `import { X } from '../module'` | +| Top-level `tests/` | `import { X } from '../src/module'` | + +- **Match existing test imports** — copy path style from neighboring tests +- If `tsconfig.json` has `paths` aliases (e.g., `@/`), use them in tests too +- For monorepos: import from the package name, not relative cross-package paths +- For monorepo workspaces (Nx, Turborepo, Lerna): run tests via the workspace tool (`nx test `, `turbo test`), not from a random package directory + +## Test File Naming + +- Match existing convention — check for `.test.ts` vs `.spec.ts` +- Jest/Vitest default: `*.test.ts`, `*.spec.ts`, or files inside `__tests__/` +- Place test files to mirror the existing project pattern + +## Common Errors + +| Error | Fix | +|-------|-----| +| `Cannot find module 'X'` | Check existing imports for correct paths; verify `tsconfig.json` `paths`; check `moduleNameMapper` (Jest) or `resolve.alias` (Vitest) | +| `TS2305: has no exported member` | Verify the exact export name from the source file | +| `TS2345: type not assignable` | Match the expected type; use type assertion only for mock objects | +| `SyntaxError: Unexpected token` / `Jest encountered an unexpected token` | Verify TS transform config (`ts-jest`, `@swc/jest`, or Vitest handles natively) | +| `ReferenceError: describe is not defined` | Vitest: import from `vitest` or set `globals: true` in config; Jest: ensure tests run under Jest not bare `node` | +| `Cannot use import statement outside a module` / `ERR_REQUIRE_ESM` | ESM/CJS mismatch — align runner config with the project's module system (see ESM section); do **not** blindly set `"type": "module"` | +| `ReferenceError: document is not defined` | Set test environment: `testEnvironment: 'jsdom'` (Jest) or `environment: 'jsdom'` (Vitest) | +| `jest.mock() ... out-of-scope variables` | Keep `jest.mock()` at top level; don't reference variables declared after the mock call (Jest hoists mocks) | +| `Cannot find module '@/...'` | Mirror the project's alias config in the test runner's module resolution | +| `Warning: not wrapped in act(...)` | Await async UI updates using the repo's existing pattern (`waitFor`, `act`) | + +## ESM vs CommonJS + +Check these signals to determine the project's module system: + +- `"type": "module"` in `package.json` → ESM +- `"module": "ESNext"` or `"NodeNext"` in `tsconfig.json` → ESM output (but not sufficient alone) +- `.mjs`/`.mts` extensions → ESM files + +If the test runner fails with ESM errors, align the runner's config with the project's module system. **Do not change `package.json` `type` field** — align the test runner to match whatever the project uses: + +- **Jest**: `--experimental-vm-modules` + `ts-jest` with `useESM: true`, or `@swc/jest` +- **Vitest**: handles ESM natively +- **Mocha**: `--loader ts-node/esm` + +## Mocking Rules + +- Prefer dependency injection over module mocking +- Use typed mocks: `jest.Mocked`, `vi.mocked(obj)`, or `Partial` with `as T` +- Jest: `jest.mock()` is hoisted — keep at top level, don't close over local variables +- Vitest: `vi.mock()` follows the same hoisting rules +- If a test needs more than 3–4 mocks, flag it as a design smell +- Mock reset: rely on `clearMocks`/`restoreMocks` config if present; otherwise reset in `beforeEach` + +## Framework-Specific Notes + +- **React/Preact**: use `@testing-library/react`, wrap with necessary providers (router, query client, theme) matching existing test setup +- **Express/Koa**: use `supertest` for HTTP testing if the repo already uses it +- **NestJS**: build testing module with `Test.createTestingModule` — don't instantiate controllers directly + +## Dependency Installation (Last Resort) + +Only install packages after investigation confirms they are missing. Use the detected package manager: + +``` + add --save-dev jest ts-jest @types/jest + add --save-dev vitest +``` + +Never install test infrastructure that conflicts with what the repo already uses. + +## Skip Coverage Tools + +Do not configure or run coverage tools (istanbul, c8, `vitest --coverage`). Coverage is measured separately by the evaluation harness.