From b703a2bc0e54117df7e73064cb710bdbbbd119e9 Mon Sep 17 00:00:00 2001 From: nohwnd Date: Wed, 4 Mar 2026 11:41:56 +0100 Subject: [PATCH 1/6] Add skill writing skill --- .github/skills/creating-skills/SKILL.md | 166 ++++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 .github/skills/creating-skills/SKILL.md diff --git a/.github/skills/creating-skills/SKILL.md b/.github/skills/creating-skills/SKILL.md new file mode 100644 index 0000000000..2ed42681bc --- /dev/null +++ b/.github/skills/creating-skills/SKILL.md @@ -0,0 +1,166 @@ +--- +name: creating-skills +description: Create custom agent capabilities when discovering novel tools, receiving task-agnostic tips from reviewers, or after researching specialized workflows not covered in existing instructions. Teaches structure, YAML optimization for LLM discoverability, and token efficiency. +--- + +# Creating GitHub Copilot Agent Skills + +This skill teaches you how to create effective GitHub Copilot Agent Skills for this repository. + +## Pre-Check: Avoid Duplication + +**STOP** and check before creating a new skill: + +1. **Does it exist already?** + - List all skills: `ls -la .github/skills/` + - Read existing skill frontmatter and content + - If semantically similar skill exists, STOP + +2. **Should an existing skill be expanded?** + - If frontmatter semantically matches your use case → Update existing skill's description + - Add keywords to improve discoverability rather than creating duplicate + +3. **Should existing skill content change?** + - If frontmatter matches but content incomplete → Add section to existing skill + - Enhance with additional examples, procedures, or troubleshooting + - Update frontmatter only if significantly broadening scope + +**Only create new skill if:** +- No semantic overlap with existing skills +- Addresses distinct problem domain +- Has unique triggering conditions + +## Skill Structure + +### Directory Placement + +Skills should be placed in `.github/skills/` directory: +- **Project skills** (repository-specific): `.github/skills/skill-name/` + +Each skill must have its own subdirectory with a lowercase, hyphenated name that matches the `name` field in the frontmatter. + +### File Requirements + +Every skill directory must contain a `SKILL.md` file (case-sensitive) with: + +1. **YAML Frontmatter** (required): + +2. **Markdown Body** with clear instructions, examples, procedures, guidelines, and references + +### Additional Resources + +Skills can include: +- Scripts (e.g., `.sh`, `.fsx`, `.ps1`) +- Example files +- Templates +- Reference documentation + +## YAML Frontmatter Best Practices + +The frontmatter is critical for skill discoverability and token efficiency: + +### Required Fields + +- **name** (string): Unique identifier, lowercase with hyphens + - Must match the directory name + - Should be descriptive but concise + - Example: `hypothesis-driven-debugging`, `github-actions-failure-debugging` + +- **description** (string): When and why to use this skill + - Should be 1-2 sentences + - Include trigger keywords that help the AI recognize when to load the skill + - Example: "Guide for debugging failing GitHub Actions workflows. Use this when asked to debug failing GitHub Actions workflows." + - **SEO-like optimization for LLMs**: Include key terms that would appear in user requests + +### Optional Fields + +- **license** (string): License for the skill (e.g., MIT, Apache-2.0) + +### Description Guidelines + +The description is crucial for skill discoverability. Think of it like SEO for LLMs: + +✅ **Good descriptions** (specific, actionable, keyword-rich): +- "Guide for debugging failing GitHub Actions workflows. Use this when asked to debug failing GitHub Actions workflows." +- "Systematic approach to investigating F# compiler performance issues using traces, dumps, and benchmarks." +- "Step-by-step process for analyzing test failures using hypothesis-driven debugging." + +❌ **Poor descriptions** (vague, generic): +- "Helps with debugging" +- "Tool for testing" +- "Useful utility" + +### Token Efficiency + +Skills should be concise to avoid wasting context tokens: +- Keep instructions focused and relevant +- Use bullet points and numbered lists +- Avoid redundant information +- Reference external resources rather than duplicating content +- The agent will only load skills when relevant, so clear descriptions help prevent unnecessary loading + +## Skill Content Best Practices + +### Structure + +1. **Title and Overview**: Brief introduction +2. **When to Use**: Clear triggering conditions +3. **Prerequisites**: Required tools, setup, or knowledge +4. **Step-by-Step Instructions**: Numbered procedures +5. **Examples**: Concrete use cases +6. **Troubleshooting**: Common issues +7. **References**: Links to related documentation + +### Writing Style + +- Use imperative mood ("Run the test", not "You should run the test") +- Be specific and actionable +- Include command examples with expected output +- Use code blocks with language identifiers +- Highlight warnings and critical information +- Reference tools and APIs that the agent has access to + +### Examples + +Always include concrete examples: +- Command invocations with flags and arguments +- Expected output and how to interpret it +- Common variations and edge cases +- Links to real-world usage in the repository + +## Testing Your Skill + +After creating a skill: + +1. Verify the file structure: + ```bash + ls -la .github/skills/your-skill-name/ + # Should show SKILL.md and any additional resources + ``` + +2. Validate YAML frontmatter: + - Ensure proper YAML syntax + - Required fields are present + - Name matches directory name + +3. Test skill invocation: + - Ask Copilot a question that should trigger the skill + - Verify the skill is loaded (check response for skill-specific guidance) + - Ensure instructions are clear and actionable + +4. Iterate based on usage: + - Monitor how often the skill is used + - Refine description for better discoverability + - Update instructions based on feedback + +## Examples from This Repository + +See existing skills in `.github/skills/` for reference: +- `hypothesis-driven-debugging`: Systematic failure investigation +- Additional skills may be added over time + +## References + +- [GitHub Copilot Agent Skills Documentation](https://docs.github.com/en/copilot/concepts/agents/about-agent-skills) +- [Agent Skills Open Standard](https://github.com/agentskills/agentskills) +- [Community Skills Collection](https://github.com/github/awesome-copilot) \ No newline at end of file From a935a8e335d362afdf3f841fcdc9f441ebf871f8 Mon Sep 17 00:00:00 2001 From: nohwnd Date: Wed, 4 Mar 2026 12:45:29 +0100 Subject: [PATCH 2/6] Skill writing skill --- .github/skills/vstest-build-test/SKILL.md | 86 +++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 .github/skills/vstest-build-test/SKILL.md diff --git a/.github/skills/vstest-build-test/SKILL.md b/.github/skills/vstest-build-test/SKILL.md new file mode 100644 index 0000000000..e452d502d9 --- /dev/null +++ b/.github/skills/vstest-build-test/SKILL.md @@ -0,0 +1,86 @@ +--- +name: vstest-build-test +description: Build, test, and validate changes in the vstest repository. Use when building vstest projects, running unit tests, smoke tests, or acceptance tests, or when deploying locally built vstest.console for manual testing. +--- + +# Building and Testing vstest + +## Build + +### Full Build (Recommended) + +For projects with many cross-project dependencies (e.g., HtmlLogger, TrxLogger, vstest.console): + +```powershell +./build.cmd -pack +``` + +This produces NuGet packages under `artifacts/packages/Debug/Shipping/`. + +### Single Project Build + +For isolated projects with few dependencies: + +```powershell +./build.cmd -project +``` + +> **Warning:** This does NOT work for projects like HtmlLogger that have many transitive dependencies. Use `-pack` instead. + +### Release Configuration + +```powershell +./build.cmd -c Release -pack +``` + +## Test + +### Unit Tests (Default) + +```powershell +./test.cmd +``` + +### Specific Test Assembly + +Use `-p` to filter by assembly name pattern: + +```powershell +./test.cmd -p htmllogger # HTML logger tests +./test.cmd -p trxlogger # TRX logger tests +./test.cmd -p datacollector # Data collector tests +./test.cmd -p smoke # Smoke tests +``` + +### Specific Test by Name + +```powershell +./test.cmd -bl -c release /p:TestRunnerAdditionalArguments="'--filter TestName'" -Integration +``` + +## Manual Validation with vstest.console + +After `./build.cmd -pack`, validate vstest.console changes by unzipping the built package: + +1. Locate the package: `artifacts/packages/Debug/Shipping/Microsoft.TestPlatform.-dev.nupkg` +2. Unzip it (`.nupkg` files are ZIP archives) +3. Run the local vstest.console against a test project + +### Alternative: Direct Artifact Paths + +- **xplat (netcoreapp):** `artifacts//netcoreapp1.0/vstest.console.dll` +- **Windows desktop:** `artifacts//net46/win7-x64/vstest.console.exe` + +## Test Categories + +| Category | Speed | What it tests | Filter | +|---|---|---|---| +| Unit tests | Fast | Individual units | `./test.cmd` (default) | +| Smoke tests | Slow | P0 end-to-end scenarios | `./test.cmd -p smoke` | +| Acceptance tests | Slowest | Extensive coverage | `-Integration` flag | + +## Troubleshooting + +- If build fails asking for .NET 4.6 targeting pack, install it from [Microsoft Downloads](https://www.microsoft.com/download/details.aspx?id=48136) +- Enable verbose diagnostics: see `docs/diagnose.md` +- For debugging, add `Debugger.Launch` at process entry points (testhost.exe, vstest.console.exe) From 20b08d7d83a5fcf3c5175832171c9f9ef98e617e Mon Sep 17 00:00:00 2001 From: Copilot Date: Wed, 4 Mar 2026 11:57:44 +0000 Subject: [PATCH 3/6] Make vstest-build-test skill cross-platform with OS mismatch detection Add pre-build environment setup section that detects when .dotnet contains binaries for the wrong OS and cleans .dotnet, .packages, and artifacts for a fresh bootstrap. Update all build/test commands to show both Linux/macOS and Windows equivalents with a quick-reference table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/vstest-build-test/SKILL.md | 99 ++++++++++++++++++----- 1 file changed, 79 insertions(+), 20 deletions(-) diff --git a/.github/skills/vstest-build-test/SKILL.md b/.github/skills/vstest-build-test/SKILL.md index e452d502d9..1b21c0e574 100644 --- a/.github/skills/vstest-build-test/SKILL.md +++ b/.github/skills/vstest-build-test/SKILL.md @@ -5,13 +5,60 @@ description: Build, test, and validate changes in the vstest repository. Use whe # Building and Testing vstest +## Pre-Build: Environment Setup + +Before building, verify the `.dotnet` toolchain matches the current OS. The repo bootstraps its own .NET SDK into `.dotnet/`. + +### Detect OS vs .dotnet Mismatch + +Run this check **before every first build in a session**: + +```bash +# Determine current OS +OS=$(uname -s) # "Linux", "Darwin" (macOS), or contains "MINGW"/"MSYS" (Windows/Git Bash) + +if [ -d ".dotnet" ]; then + if [ "$OS" = "Linux" ] || [ "$OS" = "Darwin" ]; then + # On Linux/macOS the dotnet binary must be an ELF/Mach-O executable, not .exe + if [ -f ".dotnet/dotnet.exe" ] && [ ! -f ".dotnet/dotnet" ]; then + echo "MISMATCH: .dotnet contains Windows binaries but OS is $OS" + rm -rf .dotnet .packages artifacts + echo "Cleaned .dotnet, .packages, and artifacts for fresh bootstrap" + fi + else + # On Windows the dotnet binary should be dotnet.exe + if [ -f ".dotnet/dotnet" ] && [ ! -f ".dotnet/dotnet.exe" ]; then + echo "MISMATCH: .dotnet contains Linux/macOS binaries but OS is Windows" + rm -rf .dotnet .packages artifacts + echo "Cleaned .dotnet, .packages, and artifacts for fresh bootstrap" + fi + fi +fi +``` + +After cleanup (or if `.dotnet` doesn't exist), the build script automatically downloads the correct SDK version from `global.json`. + ## Build +### Platform Commands + +| Action | Windows | Linux / macOS | +|---|---|---| +| Restore + Build | `./build.cmd` | `./build.sh` | +| Restore only | `./restore.cmd` | `./restore.sh` | +| Build + Pack | `./build.cmd -pack` | `./build.sh --pack` | +| Release config | `./build.cmd -c Release -pack` | `./build.sh -c Release --pack` | +| Single project | `./build.cmd -project ` | `./build.sh --projects ` | + ### Full Build (Recommended) For projects with many cross-project dependencies (e.g., HtmlLogger, TrxLogger, vstest.console): -```powershell +```bash +# Linux / macOS +./build.sh --pack + +# Windows ./build.cmd -pack ``` @@ -21,23 +68,25 @@ This produces NuGet packages under `artifacts/packages/Debug/Shipping/`. For isolated projects with few dependencies: -```powershell +```bash +# Linux / macOS +./build.sh --projects + +# Windows ./build.cmd -project ``` -> **Warning:** This does NOT work for projects like HtmlLogger that have many transitive dependencies. Use `-pack` instead. - -### Release Configuration - -```powershell -./build.cmd -c Release -pack -``` +> **Warning:** This does NOT work for projects like HtmlLogger that have many transitive dependencies. Use `--pack` / `-pack` instead. ## Test ### Unit Tests (Default) -```powershell +```bash +# Linux / macOS +./test.sh + +# Windows ./test.cmd ``` @@ -45,22 +94,31 @@ For isolated projects with few dependencies: Use `-p` to filter by assembly name pattern: -```powershell -./test.cmd -p htmllogger # HTML logger tests -./test.cmd -p trxlogger # TRX logger tests -./test.cmd -p datacollector # Data collector tests -./test.cmd -p smoke # Smoke tests +```bash +# Linux / macOS +./test.sh -p htmllogger # HTML logger tests +./test.sh -p trxlogger # TRX logger tests +./test.sh -p datacollector # Data collector tests +./test.sh -p smoke # Smoke tests + +# Windows +./test.cmd -p htmllogger +./test.cmd -p smoke ``` ### Specific Test by Name -```powershell +```bash +# Windows ./test.cmd -bl -c release /p:TestRunnerAdditionalArguments="'--filter TestName'" -Integration + +# Linux / macOS +./test.sh -bl -c release /p:TestRunnerAdditionalArguments="'--filter TestName'" --integrationTest ``` ## Manual Validation with vstest.console -After `./build.cmd -pack`, validate vstest.console changes by unzipping the built package: +After building with `--pack` / `-pack`, validate vstest.console changes by unzipping the built package: 1. Locate the package: `artifacts/packages/Debug/Shipping/Microsoft.TestPlatform.-dev.nupkg` 2. Unzip it (`.nupkg` files are ZIP archives) @@ -75,12 +133,13 @@ After `./build.cmd -pack`, validate vstest.console changes by unzipping the buil | Category | Speed | What it tests | Filter | |---|---|---|---| -| Unit tests | Fast | Individual units | `./test.cmd` (default) | -| Smoke tests | Slow | P0 end-to-end scenarios | `./test.cmd -p smoke` | -| Acceptance tests | Slowest | Extensive coverage | `-Integration` flag | +| Unit tests | Fast | Individual units | `./test.sh` / `./test.cmd` (default) | +| Smoke tests | Slow | P0 end-to-end scenarios | `-p smoke` | +| Acceptance tests | Slowest | Extensive coverage | `--integrationTest` / `-Integration` flag | ## Troubleshooting +- **OS mismatch errors:** If you see SDK load failures, run the mismatch detection script above to clean and re-bootstrap. - If build fails asking for .NET 4.6 targeting pack, install it from [Microsoft Downloads](https://www.microsoft.com/download/details.aspx?id=48136) - Enable verbose diagnostics: see `docs/diagnose.md` - For debugging, add `Debugger.Launch` at process entry points (testhost.exe, vstest.console.exe) From 7bc6553b9c0e6a45ad6a9a0f2659392bde24ceb2 Mon Sep 17 00:00:00 2001 From: Copilot Date: Wed, 4 Mar 2026 13:26:30 +0100 Subject: [PATCH 4/6] Improve validate-skills skill with Windows support and ENV_ISSUE classification - Add PowerShell OS detection alongside bash - Add guidance for extracting OS-specific commands from tables - Add placeholder substitution and cross-platform adaptation rules - Use SQL tracking table instead of markdown table - Distinguish ENV_ISSUE from ERROR in result classification - Improve Fix or Flag section with actionable steps per status Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/validate-skills/SKILL.md | 98 +++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 .github/skills/validate-skills/SKILL.md diff --git a/.github/skills/validate-skills/SKILL.md b/.github/skills/validate-skills/SKILL.md new file mode 100644 index 0000000000..07a5a9ff09 --- /dev/null +++ b/.github/skills/validate-skills/SKILL.md @@ -0,0 +1,98 @@ +--- +name: validate-skills +description: Validate that commands documented in skill files actually work. Use when creating, updating, or reviewing skills to ensure all documented commands exit with code 0. +--- + +# Validating Skills + +Verify every executable command in a skill runs successfully on the current OS. + +## When to Use + +- After creating or updating a skill that contains executable commands +- During skill review to catch stale or broken instructions +- When switching OS (e.g. Windows → Linux) to confirm cross-platform commands + +## Procedure + +### 1. Detect Current OS + +Determine which platform commands to extract: + +```powershell +# PowerShell (Windows) +$os = "Windows" +``` + +```bash +# Bash (Linux / macOS) +OS=$(uname -s) # "Linux" or "Darwin" +``` + +### 2. Extract Commands + +Parse the target skill's `SKILL.md` and list every shell command for the detected OS: + +- Many skills document commands in tables with **Windows** and **Linux / macOS** columns. Pick the column matching your OS. +- If a command contains comments like `# Windows` or `# Linux / macOS`, only run the one for your OS. +- **Placeholder substitution:** Replace obvious placeholders (e.g. ``, ``) with real values from the repo. If no sensible value exists, skip the command. +- **Adapt cross-platform commands:** Commands like `ls -la` should be adapted to PowerShell equivalents (`Get-ChildItem`) on Windows when no native Windows command is documented. + +### 3. Track Results + +Use the SQL tool to create a tracking table: + +```sql +CREATE TABLE skill_commands ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + skill TEXT NOT NULL, + command TEXT NOT NULL, + expected_exit INTEGER DEFAULT 0, + actual_exit INTEGER, + status TEXT DEFAULT 'pending', + notes TEXT +); +``` + +### 4. Run Each Command + +For every extracted command: + +1. Run it from the repo root +2. Record the exit code +3. Classify the result: + - **Exit 0 → PASS** + - **Non-zero + environment issue (missing SDK, no internet) → ENV_ISSUE** + - **Non-zero + command/docs wrong → ERROR** + +### 5. Safety Rules + +> **CRITICAL:** Never run unfiltered integration/acceptance tests. They take hours. +> - `test.sh --integrationTest` or `test.cmd -Integration` **MUST** include a `--filter` or `-p` flag. +> - `test.sh -p smoke` is acceptable (scoped to smoke tests), but expect it to be slow. + +### 6. Report + +After all commands finish, print a summary: + +``` +=== Skill Validation Report === +Skill: +OS: +Commands tested: N +PASS: X +ENV_ISSUE: Y (list with reasons) +ERROR: Z (list failed commands with exit codes) +``` + +### 7. Fix or Flag + +- **ERROR (documentation bug):** Update the skill's `SKILL.md` to fix the command. +- **ENV_ISSUE:** Add a troubleshooting note to the skill if the environment prerequisite is not already documented. +- **PASS:** No action needed. + +## Ordering Tips + +- Run restore/build before tests (tests depend on build output) +- Run the cheapest commands first to fail fast +- Batch independent test commands in parallel when possible From 2bd9d56b27b73f5c341b2c9fe624be49cde817da Mon Sep 17 00:00:00 2001 From: Copilot Date: Wed, 4 Mar 2026 14:38:31 +0100 Subject: [PATCH 5/6] Fix build exit code propagation and skill documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - eng/build.ps1: Forward exit code from Arcade build script using \xit \0\ to ensure non-zero exit codes are propagated to the caller - eng/common/tools.ps1: Add diagnostic log message before exiting so the exit code is visible in build output - .github/skills/vstest-build-test/SKILL.md: Fix Windows test filter flag from \-p\ to \-projects\ since \-p\ is ambiguous in PowerShell and does not work as expected Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/vstest-build-test/SKILL.md | 6 +++--- eng/build.ps1 | 4 +++- eng/common/tools.ps1 | 1 + 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/.github/skills/vstest-build-test/SKILL.md b/.github/skills/vstest-build-test/SKILL.md index 1b21c0e574..3c061b2c28 100644 --- a/.github/skills/vstest-build-test/SKILL.md +++ b/.github/skills/vstest-build-test/SKILL.md @@ -101,9 +101,9 @@ Use `-p` to filter by assembly name pattern: ./test.sh -p datacollector # Data collector tests ./test.sh -p smoke # Smoke tests -# Windows -./test.cmd -p htmllogger -./test.cmd -p smoke +# Windows (-p is ambiguous in PowerShell; use -projects) +./test.cmd -projects htmllogger +./test.cmd -projects smoke ``` ### Specific Test by Name diff --git a/eng/build.ps1 b/eng/build.ps1 index 3f17a9cf7d..0d4d2b915e 100644 --- a/eng/build.ps1 +++ b/eng/build.ps1 @@ -38,4 +38,6 @@ Param( # Call the build script provided by Arcade -& $PSScriptRoot/common/build.ps1 @PSBoundParameters \ No newline at end of file +& $PSScriptRoot/common/build.ps1 @PSBoundParameters +# Forward exit code of the parent script +exit $LastExitCode diff --git a/eng/common/tools.ps1 b/eng/common/tools.ps1 index 01296ee601..8b3ae7cd43 100644 --- a/eng/common/tools.ps1 +++ b/eng/common/tools.ps1 @@ -747,6 +747,7 @@ function ExitWithExitCode([int] $exitCode) { if ($ci -and $prepareMachine) { Stop-Processes } + Write-Host "exiting with exit code: $exitCode" exit $exitCode } From eec01436311e67baf892fdefa1f5fc8e9d131364 Mon Sep 17 00:00:00 2001 From: nohwnd Date: Wed, 4 Mar 2026 15:52:40 +0100 Subject: [PATCH 6/6] Remove exit code message --- eng/common/tools.ps1 | 1 - 1 file changed, 1 deletion(-) diff --git a/eng/common/tools.ps1 b/eng/common/tools.ps1 index 8b3ae7cd43..01296ee601 100644 --- a/eng/common/tools.ps1 +++ b/eng/common/tools.ps1 @@ -747,7 +747,6 @@ function ExitWithExitCode([int] $exitCode) { if ($ci -and $prepareMachine) { Stop-Processes } - Write-Host "exiting with exit code: $exitCode" exit $exitCode }