-
Notifications
You must be signed in to change notification settings - Fork 355
Add skill writing skill, skill validating skill, and build and test skill #15434
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
b703a2b
Add skill writing skill
nohwnd a935a8e
Skill writing skill
nohwnd 20b08d7
Make vstest-build-test skill cross-platform with OS mismatch detection
invalid-email-address 7bc6553
Improve validate-skills skill with Windows support and ENV_ISSUE clas…
invalid-email-address 2bd9d56
Fix build exit code propagation and skill documentation
invalid-email-address eec0143
Remove exit code message
nohwnd File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,166 @@ | ||
| --- | ||
| name: creating-skills | ||
| description: Create custom agent capabilities when discovering novel tools, receiving task-agnostic tips from reviewers, or after researching specialized workflows not covered in existing instructions. Teaches structure, YAML optimization for LLM discoverability, and token efficiency. | ||
| --- | ||
|
|
||
| # Creating GitHub Copilot Agent Skills | ||
|
|
||
| This skill teaches you how to create effective GitHub Copilot Agent Skills for this repository. | ||
|
|
||
| ## Pre-Check: Avoid Duplication | ||
|
|
||
| **STOP** and check before creating a new skill: | ||
|
|
||
| 1. **Does it exist already?** | ||
| - List all skills: `ls -la .github/skills/` | ||
| - Read existing skill frontmatter and content | ||
| - If semantically similar skill exists, STOP | ||
|
|
||
| 2. **Should an existing skill be expanded?** | ||
| - If frontmatter semantically matches your use case → Update existing skill's description | ||
| - Add keywords to improve discoverability rather than creating duplicate | ||
|
|
||
| 3. **Should existing skill content change?** | ||
| - If frontmatter matches but content incomplete → Add section to existing skill | ||
| - Enhance with additional examples, procedures, or troubleshooting | ||
| - Update frontmatter only if significantly broadening scope | ||
|
|
||
| **Only create new skill if:** | ||
| - No semantic overlap with existing skills | ||
| - Addresses distinct problem domain | ||
| - Has unique triggering conditions | ||
|
|
||
| ## Skill Structure | ||
|
|
||
| ### Directory Placement | ||
|
|
||
| Skills should be placed in `.github/skills/` directory: | ||
| - **Project skills** (repository-specific): `.github/skills/skill-name/` | ||
|
|
||
| Each skill must have its own subdirectory with a lowercase, hyphenated name that matches the `name` field in the frontmatter. | ||
|
|
||
| ### File Requirements | ||
|
|
||
| Every skill directory must contain a `SKILL.md` file (case-sensitive) with: | ||
|
|
||
| 1. **YAML Frontmatter** (required): | ||
|
|
||
| 2. **Markdown Body** with clear instructions, examples, procedures, guidelines, and references | ||
|
|
||
| ### Additional Resources | ||
|
|
||
| Skills can include: | ||
| - Scripts (e.g., `.sh`, `.fsx`, `.ps1`) | ||
| - Example files | ||
| - Templates | ||
| - Reference documentation | ||
|
|
||
| ## YAML Frontmatter Best Practices | ||
|
|
||
| The frontmatter is critical for skill discoverability and token efficiency: | ||
|
|
||
| ### Required Fields | ||
|
|
||
| - **name** (string): Unique identifier, lowercase with hyphens | ||
| - Must match the directory name | ||
| - Should be descriptive but concise | ||
| - Example: `hypothesis-driven-debugging`, `github-actions-failure-debugging` | ||
|
|
||
| - **description** (string): When and why to use this skill | ||
| - Should be 1-2 sentences | ||
| - Include trigger keywords that help the AI recognize when to load the skill | ||
| - Example: "Guide for debugging failing GitHub Actions workflows. Use this when asked to debug failing GitHub Actions workflows." | ||
| - **SEO-like optimization for LLMs**: Include key terms that would appear in user requests | ||
|
|
||
| ### Optional Fields | ||
|
|
||
| - **license** (string): License for the skill (e.g., MIT, Apache-2.0) | ||
|
|
||
| ### Description Guidelines | ||
|
|
||
| The description is crucial for skill discoverability. Think of it like SEO for LLMs: | ||
|
|
||
| ✅ **Good descriptions** (specific, actionable, keyword-rich): | ||
| - "Guide for debugging failing GitHub Actions workflows. Use this when asked to debug failing GitHub Actions workflows." | ||
| - "Systematic approach to investigating F# compiler performance issues using traces, dumps, and benchmarks." | ||
| - "Step-by-step process for analyzing test failures using hypothesis-driven debugging." | ||
|
|
||
| ❌ **Poor descriptions** (vague, generic): | ||
| - "Helps with debugging" | ||
| - "Tool for testing" | ||
| - "Useful utility" | ||
|
|
||
| ### Token Efficiency | ||
|
|
||
| Skills should be concise to avoid wasting context tokens: | ||
| - Keep instructions focused and relevant | ||
| - Use bullet points and numbered lists | ||
| - Avoid redundant information | ||
| - Reference external resources rather than duplicating content | ||
| - The agent will only load skills when relevant, so clear descriptions help prevent unnecessary loading | ||
|
|
||
| ## Skill Content Best Practices | ||
|
|
||
| ### Structure | ||
|
|
||
| 1. **Title and Overview**: Brief introduction | ||
| 2. **When to Use**: Clear triggering conditions | ||
| 3. **Prerequisites**: Required tools, setup, or knowledge | ||
| 4. **Step-by-Step Instructions**: Numbered procedures | ||
| 5. **Examples**: Concrete use cases | ||
| 6. **Troubleshooting**: Common issues | ||
| 7. **References**: Links to related documentation | ||
|
|
||
| ### Writing Style | ||
|
|
||
| - Use imperative mood ("Run the test", not "You should run the test") | ||
| - Be specific and actionable | ||
| - Include command examples with expected output | ||
| - Use code blocks with language identifiers | ||
| - Highlight warnings and critical information | ||
| - Reference tools and APIs that the agent has access to | ||
|
|
||
| ### Examples | ||
|
|
||
| Always include concrete examples: | ||
| - Command invocations with flags and arguments | ||
| - Expected output and how to interpret it | ||
| - Common variations and edge cases | ||
| - Links to real-world usage in the repository | ||
|
|
||
| ## Testing Your Skill | ||
|
|
||
| After creating a skill: | ||
|
|
||
| 1. Verify the file structure: | ||
| ```bash | ||
| ls -la .github/skills/your-skill-name/ | ||
| # Should show SKILL.md and any additional resources | ||
| ``` | ||
|
|
||
| 2. Validate YAML frontmatter: | ||
| - Ensure proper YAML syntax | ||
| - Required fields are present | ||
| - Name matches directory name | ||
|
|
||
| 3. Test skill invocation: | ||
| - Ask Copilot a question that should trigger the skill | ||
| - Verify the skill is loaded (check response for skill-specific guidance) | ||
| - Ensure instructions are clear and actionable | ||
|
|
||
| 4. Iterate based on usage: | ||
| - Monitor how often the skill is used | ||
| - Refine description for better discoverability | ||
| - Update instructions based on feedback | ||
|
|
||
| ## Examples from This Repository | ||
|
|
||
| See existing skills in `.github/skills/` for reference: | ||
| - `hypothesis-driven-debugging`: Systematic failure investigation | ||
| - Additional skills may be added over time | ||
|
|
||
| ## References | ||
|
|
||
| - [GitHub Copilot Agent Skills Documentation](https://docs.github.com/en/copilot/concepts/agents/about-agent-skills) | ||
| - [Agent Skills Open Standard](https://github.com/agentskills/agentskills) | ||
| - [Community Skills Collection](https://github.com/github/awesome-copilot) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| --- | ||
| name: validate-skills | ||
| description: Validate that commands documented in skill files actually work. Use when creating, updating, or reviewing skills to ensure all documented commands exit with code 0. | ||
| --- | ||
|
|
||
| # Validating Skills | ||
|
|
||
| Verify every executable command in a skill runs successfully on the current OS. | ||
|
|
||
| ## When to Use | ||
|
|
||
| - After creating or updating a skill that contains executable commands | ||
| - During skill review to catch stale or broken instructions | ||
| - When switching OS (e.g. Windows → Linux) to confirm cross-platform commands | ||
|
|
||
| ## Procedure | ||
|
|
||
| ### 1. Detect Current OS | ||
|
|
||
| Determine which platform commands to extract: | ||
|
|
||
| ```powershell | ||
| # PowerShell (Windows) | ||
| $os = "Windows" | ||
| ``` | ||
|
|
||
| ```bash | ||
| # Bash (Linux / macOS) | ||
| OS=$(uname -s) # "Linux" or "Darwin" | ||
| ``` | ||
|
|
||
| ### 2. Extract Commands | ||
|
|
||
| Parse the target skill's `SKILL.md` and list every shell command for the detected OS: | ||
|
|
||
| - Many skills document commands in tables with **Windows** and **Linux / macOS** columns. Pick the column matching your OS. | ||
| - If a command contains comments like `# Windows` or `# Linux / macOS`, only run the one for your OS. | ||
| - **Placeholder substitution:** Replace obvious placeholders (e.g. `<path-to-csproj>`, `<skill-name>`) with real values from the repo. If no sensible value exists, skip the command. | ||
| - **Adapt cross-platform commands:** Commands like `ls -la` should be adapted to PowerShell equivalents (`Get-ChildItem`) on Windows when no native Windows command is documented. | ||
|
|
||
| ### 3. Track Results | ||
|
|
||
| Use the SQL tool to create a tracking table: | ||
|
nohwnd marked this conversation as resolved.
|
||
|
|
||
| ```sql | ||
| CREATE TABLE skill_commands ( | ||
| id INTEGER PRIMARY KEY AUTOINCREMENT, | ||
| skill TEXT NOT NULL, | ||
| command TEXT NOT NULL, | ||
| expected_exit INTEGER DEFAULT 0, | ||
| actual_exit INTEGER, | ||
| status TEXT DEFAULT 'pending', | ||
| notes TEXT | ||
| ); | ||
| ``` | ||
|
|
||
| ### 4. Run Each Command | ||
|
|
||
| For every extracted command: | ||
|
|
||
| 1. Run it from the repo root | ||
| 2. Record the exit code | ||
| 3. Classify the result: | ||
| - **Exit 0 → PASS** | ||
| - **Non-zero + environment issue (missing SDK, no internet) → ENV_ISSUE** | ||
| - **Non-zero + command/docs wrong → ERROR** | ||
|
|
||
| ### 5. Safety Rules | ||
|
|
||
| > **CRITICAL:** Never run unfiltered integration/acceptance tests. They take hours. | ||
| > - `test.sh --integrationTest` or `test.cmd -Integration` **MUST** include a `--filter` or `-p` flag. | ||
| > - `test.sh -p smoke` is acceptable (scoped to smoke tests), but expect it to be slow. | ||
|
|
||
| ### 6. Report | ||
|
|
||
| After all commands finish, print a summary: | ||
|
|
||
| ``` | ||
| === Skill Validation Report === | ||
| Skill: <skill-name> | ||
| OS: <Linux|Darwin|Windows> | ||
| Commands tested: N | ||
| PASS: X | ||
| ENV_ISSUE: Y (list with reasons) | ||
| ERROR: Z (list failed commands with exit codes) | ||
| ``` | ||
|
|
||
| ### 7. Fix or Flag | ||
|
|
||
| - **ERROR (documentation bug):** Update the skill's `SKILL.md` to fix the command. | ||
| - **ENV_ISSUE:** Add a troubleshooting note to the skill if the environment prerequisite is not already documented. | ||
| - **PASS:** No action needed. | ||
|
|
||
| ## Ordering Tips | ||
|
|
||
| - Run restore/build before tests (tests depend on build output) | ||
| - Run the cheapest commands first to fail fast | ||
| - Batch independent test commands in parallel when possible | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.