diff --git a/.github/skills/creating-skills/SKILL.md b/.github/skills/creating-skills/SKILL.md new file mode 100644 index 0000000000..2ed42681bc --- /dev/null +++ b/.github/skills/creating-skills/SKILL.md @@ -0,0 +1,166 @@ +--- +name: creating-skills +description: Create custom agent capabilities when discovering novel tools, receiving task-agnostic tips from reviewers, or after researching specialized workflows not covered in existing instructions. Teaches structure, YAML optimization for LLM discoverability, and token efficiency. +--- + +# Creating GitHub Copilot Agent Skills + +This skill teaches you how to create effective GitHub Copilot Agent Skills for this repository. + +## Pre-Check: Avoid Duplication + +**STOP** and check before creating a new skill: + +1. **Does it exist already?** + - List all skills: `ls -la .github/skills/` + - Read existing skill frontmatter and content + - If semantically similar skill exists, STOP + +2. **Should an existing skill be expanded?** + - If frontmatter semantically matches your use case → Update existing skill's description + - Add keywords to improve discoverability rather than creating duplicate + +3. **Should existing skill content change?** + - If frontmatter matches but content incomplete → Add section to existing skill + - Enhance with additional examples, procedures, or troubleshooting + - Update frontmatter only if significantly broadening scope + +**Only create new skill if:** +- No semantic overlap with existing skills +- Addresses distinct problem domain +- Has unique triggering conditions + +## Skill Structure + +### Directory Placement + +Skills should be placed in `.github/skills/` directory: +- **Project skills** (repository-specific): `.github/skills/skill-name/` + +Each skill must have its own subdirectory with a lowercase, hyphenated name that matches the `name` field in the frontmatter. + +### File Requirements + +Every skill directory must contain a `SKILL.md` file (case-sensitive) with: + +1. **YAML Frontmatter** (required): + +2. **Markdown Body** with clear instructions, examples, procedures, guidelines, and references + +### Additional Resources + +Skills can include: +- Scripts (e.g., `.sh`, `.fsx`, `.ps1`) +- Example files +- Templates +- Reference documentation + +## YAML Frontmatter Best Practices + +The frontmatter is critical for skill discoverability and token efficiency: + +### Required Fields + +- **name** (string): Unique identifier, lowercase with hyphens + - Must match the directory name + - Should be descriptive but concise + - Example: `hypothesis-driven-debugging`, `github-actions-failure-debugging` + +- **description** (string): When and why to use this skill + - Should be 1-2 sentences + - Include trigger keywords that help the AI recognize when to load the skill + - Example: "Guide for debugging failing GitHub Actions workflows. Use this when asked to debug failing GitHub Actions workflows." + - **SEO-like optimization for LLMs**: Include key terms that would appear in user requests + +### Optional Fields + +- **license** (string): License for the skill (e.g., MIT, Apache-2.0) + +### Description Guidelines + +The description is crucial for skill discoverability. Think of it like SEO for LLMs: + +✅ **Good descriptions** (specific, actionable, keyword-rich): +- "Guide for debugging failing GitHub Actions workflows. Use this when asked to debug failing GitHub Actions workflows." +- "Systematic approach to investigating F# compiler performance issues using traces, dumps, and benchmarks." +- "Step-by-step process for analyzing test failures using hypothesis-driven debugging." + +❌ **Poor descriptions** (vague, generic): +- "Helps with debugging" +- "Tool for testing" +- "Useful utility" + +### Token Efficiency + +Skills should be concise to avoid wasting context tokens: +- Keep instructions focused and relevant +- Use bullet points and numbered lists +- Avoid redundant information +- Reference external resources rather than duplicating content +- The agent will only load skills when relevant, so clear descriptions help prevent unnecessary loading + +## Skill Content Best Practices + +### Structure + +1. **Title and Overview**: Brief introduction +2. **When to Use**: Clear triggering conditions +3. **Prerequisites**: Required tools, setup, or knowledge +4. **Step-by-Step Instructions**: Numbered procedures +5. **Examples**: Concrete use cases +6. **Troubleshooting**: Common issues +7. **References**: Links to related documentation + +### Writing Style + +- Use imperative mood ("Run the test", not "You should run the test") +- Be specific and actionable +- Include command examples with expected output +- Use code blocks with language identifiers +- Highlight warnings and critical information +- Reference tools and APIs that the agent has access to + +### Examples + +Always include concrete examples: +- Command invocations with flags and arguments +- Expected output and how to interpret it +- Common variations and edge cases +- Links to real-world usage in the repository + +## Testing Your Skill + +After creating a skill: + +1. Verify the file structure: + ```bash + ls -la .github/skills/your-skill-name/ + # Should show SKILL.md and any additional resources + ``` + +2. Validate YAML frontmatter: + - Ensure proper YAML syntax + - Required fields are present + - Name matches directory name + +3. Test skill invocation: + - Ask Copilot a question that should trigger the skill + - Verify the skill is loaded (check response for skill-specific guidance) + - Ensure instructions are clear and actionable + +4. Iterate based on usage: + - Monitor how often the skill is used + - Refine description for better discoverability + - Update instructions based on feedback + +## Examples from This Repository + +See existing skills in `.github/skills/` for reference: +- `hypothesis-driven-debugging`: Systematic failure investigation +- Additional skills may be added over time + +## References + +- [GitHub Copilot Agent Skills Documentation](https://docs.github.com/en/copilot/concepts/agents/about-agent-skills) +- [Agent Skills Open Standard](https://github.com/agentskills/agentskills) +- [Community Skills Collection](https://github.com/github/awesome-copilot) \ No newline at end of file diff --git a/.github/skills/validate-skills/SKILL.md b/.github/skills/validate-skills/SKILL.md new file mode 100644 index 0000000000..07a5a9ff09 --- /dev/null +++ b/.github/skills/validate-skills/SKILL.md @@ -0,0 +1,98 @@ +--- +name: validate-skills +description: Validate that commands documented in skill files actually work. Use when creating, updating, or reviewing skills to ensure all documented commands exit with code 0. +--- + +# Validating Skills + +Verify every executable command in a skill runs successfully on the current OS. + +## When to Use + +- After creating or updating a skill that contains executable commands +- During skill review to catch stale or broken instructions +- When switching OS (e.g. Windows → Linux) to confirm cross-platform commands + +## Procedure + +### 1. Detect Current OS + +Determine which platform commands to extract: + +```powershell +# PowerShell (Windows) +$os = "Windows" +``` + +```bash +# Bash (Linux / macOS) +OS=$(uname -s) # "Linux" or "Darwin" +``` + +### 2. Extract Commands + +Parse the target skill's `SKILL.md` and list every shell command for the detected OS: + +- Many skills document commands in tables with **Windows** and **Linux / macOS** columns. Pick the column matching your OS. +- If a command contains comments like `# Windows` or `# Linux / macOS`, only run the one for your OS. +- **Placeholder substitution:** Replace obvious placeholders (e.g. ``, ``) with real values from the repo. If no sensible value exists, skip the command. +- **Adapt cross-platform commands:** Commands like `ls -la` should be adapted to PowerShell equivalents (`Get-ChildItem`) on Windows when no native Windows command is documented. + +### 3. Track Results + +Use the SQL tool to create a tracking table: + +```sql +CREATE TABLE skill_commands ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + skill TEXT NOT NULL, + command TEXT NOT NULL, + expected_exit INTEGER DEFAULT 0, + actual_exit INTEGER, + status TEXT DEFAULT 'pending', + notes TEXT +); +``` + +### 4. Run Each Command + +For every extracted command: + +1. Run it from the repo root +2. Record the exit code +3. Classify the result: + - **Exit 0 → PASS** + - **Non-zero + environment issue (missing SDK, no internet) → ENV_ISSUE** + - **Non-zero + command/docs wrong → ERROR** + +### 5. Safety Rules + +> **CRITICAL:** Never run unfiltered integration/acceptance tests. They take hours. +> - `test.sh --integrationTest` or `test.cmd -Integration` **MUST** include a `--filter` or `-p` flag. +> - `test.sh -p smoke` is acceptable (scoped to smoke tests), but expect it to be slow. + +### 6. Report + +After all commands finish, print a summary: + +``` +=== Skill Validation Report === +Skill: +OS: +Commands tested: N +PASS: X +ENV_ISSUE: Y (list with reasons) +ERROR: Z (list failed commands with exit codes) +``` + +### 7. Fix or Flag + +- **ERROR (documentation bug):** Update the skill's `SKILL.md` to fix the command. +- **ENV_ISSUE:** Add a troubleshooting note to the skill if the environment prerequisite is not already documented. +- **PASS:** No action needed. + +## Ordering Tips + +- Run restore/build before tests (tests depend on build output) +- Run the cheapest commands first to fail fast +- Batch independent test commands in parallel when possible diff --git a/.github/skills/vstest-build-test/SKILL.md b/.github/skills/vstest-build-test/SKILL.md new file mode 100644 index 0000000000..3c061b2c28 --- /dev/null +++ b/.github/skills/vstest-build-test/SKILL.md @@ -0,0 +1,145 @@ +--- +name: vstest-build-test +description: Build, test, and validate changes in the vstest repository. Use when building vstest projects, running unit tests, smoke tests, or acceptance tests, or when deploying locally built vstest.console for manual testing. +--- + +# Building and Testing vstest + +## Pre-Build: Environment Setup + +Before building, verify the `.dotnet` toolchain matches the current OS. The repo bootstraps its own .NET SDK into `.dotnet/`. + +### Detect OS vs .dotnet Mismatch + +Run this check **before every first build in a session**: + +```bash +# Determine current OS +OS=$(uname -s) # "Linux", "Darwin" (macOS), or contains "MINGW"/"MSYS" (Windows/Git Bash) + +if [ -d ".dotnet" ]; then + if [ "$OS" = "Linux" ] || [ "$OS" = "Darwin" ]; then + # On Linux/macOS the dotnet binary must be an ELF/Mach-O executable, not .exe + if [ -f ".dotnet/dotnet.exe" ] && [ ! -f ".dotnet/dotnet" ]; then + echo "MISMATCH: .dotnet contains Windows binaries but OS is $OS" + rm -rf .dotnet .packages artifacts + echo "Cleaned .dotnet, .packages, and artifacts for fresh bootstrap" + fi + else + # On Windows the dotnet binary should be dotnet.exe + if [ -f ".dotnet/dotnet" ] && [ ! -f ".dotnet/dotnet.exe" ]; then + echo "MISMATCH: .dotnet contains Linux/macOS binaries but OS is Windows" + rm -rf .dotnet .packages artifacts + echo "Cleaned .dotnet, .packages, and artifacts for fresh bootstrap" + fi + fi +fi +``` + +After cleanup (or if `.dotnet` doesn't exist), the build script automatically downloads the correct SDK version from `global.json`. + +## Build + +### Platform Commands + +| Action | Windows | Linux / macOS | +|---|---|---| +| Restore + Build | `./build.cmd` | `./build.sh` | +| Restore only | `./restore.cmd` | `./restore.sh` | +| Build + Pack | `./build.cmd -pack` | `./build.sh --pack` | +| Release config | `./build.cmd -c Release -pack` | `./build.sh -c Release --pack` | +| Single project | `./build.cmd -project ` | `./build.sh --projects ` | + +### Full Build (Recommended) + +For projects with many cross-project dependencies (e.g., HtmlLogger, TrxLogger, vstest.console): + +```bash +# Linux / macOS +./build.sh --pack + +# Windows +./build.cmd -pack +``` + +This produces NuGet packages under `artifacts/packages/Debug/Shipping/`. + +### Single Project Build + +For isolated projects with few dependencies: + +```bash +# Linux / macOS +./build.sh --projects + +# Windows +./build.cmd -project +``` + +> **Warning:** This does NOT work for projects like HtmlLogger that have many transitive dependencies. Use `--pack` / `-pack` instead. + +## Test + +### Unit Tests (Default) + +```bash +# Linux / macOS +./test.sh + +# Windows +./test.cmd +``` + +### Specific Test Assembly + +Use `-p` to filter by assembly name pattern: + +```bash +# Linux / macOS +./test.sh -p htmllogger # HTML logger tests +./test.sh -p trxlogger # TRX logger tests +./test.sh -p datacollector # Data collector tests +./test.sh -p smoke # Smoke tests + +# Windows (-p is ambiguous in PowerShell; use -projects) +./test.cmd -projects htmllogger +./test.cmd -projects smoke +``` + +### Specific Test by Name + +```bash +# Windows +./test.cmd -bl -c release /p:TestRunnerAdditionalArguments="'--filter TestName'" -Integration + +# Linux / macOS +./test.sh -bl -c release /p:TestRunnerAdditionalArguments="'--filter TestName'" --integrationTest +``` + +## Manual Validation with vstest.console + +After building with `--pack` / `-pack`, validate vstest.console changes by unzipping the built package: + +1. Locate the package: `artifacts/packages/Debug/Shipping/Microsoft.TestPlatform.-dev.nupkg` +2. Unzip it (`.nupkg` files are ZIP archives) +3. Run the local vstest.console against a test project + +### Alternative: Direct Artifact Paths + +- **xplat (netcoreapp):** `artifacts//netcoreapp1.0/vstest.console.dll` +- **Windows desktop:** `artifacts//net46/win7-x64/vstest.console.exe` + +## Test Categories + +| Category | Speed | What it tests | Filter | +|---|---|---|---| +| Unit tests | Fast | Individual units | `./test.sh` / `./test.cmd` (default) | +| Smoke tests | Slow | P0 end-to-end scenarios | `-p smoke` | +| Acceptance tests | Slowest | Extensive coverage | `--integrationTest` / `-Integration` flag | + +## Troubleshooting + +- **OS mismatch errors:** If you see SDK load failures, run the mismatch detection script above to clean and re-bootstrap. +- If build fails asking for .NET 4.6 targeting pack, install it from [Microsoft Downloads](https://www.microsoft.com/download/details.aspx?id=48136) +- Enable verbose diagnostics: see `docs/diagnose.md` +- For debugging, add `Debugger.Launch` at process entry points (testhost.exe, vstest.console.exe) diff --git a/eng/build.ps1 b/eng/build.ps1 index 3f17a9cf7d..0d4d2b915e 100644 --- a/eng/build.ps1 +++ b/eng/build.ps1 @@ -38,4 +38,6 @@ Param( # Call the build script provided by Arcade -& $PSScriptRoot/common/build.ps1 @PSBoundParameters \ No newline at end of file +& $PSScriptRoot/common/build.ps1 @PSBoundParameters +# Forward exit code of the parent script +exit $LastExitCode