diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 4bf9175521..4030621d80 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -22,6 +22,20 @@ /plugins/dotnet/skills/nuget-trusted-publishing/ @lewing @kartheekp-ms /tests/dotnet/nuget-trusted-publishing/ @lewing @kartheekp-ms +/plugins/dotnet/agents/optimizing-dotnet-performance.agent.md @dotnet/appmodel + +/plugins/dotnet-ai/skills/mcp-csharp-create/ @leslierichardson95 @artl93 +/tests/dotnet-ai/mcp-csharp-create/ @leslierichardson95 @artl93 + +/plugins/dotnet-ai/skills/mcp-csharp-debug/ @leslierichardson95 @artl93 +/tests/dotnet-ai/mcp-csharp-debug/ @leslierichardson95 @artl93 + +/plugins/dotnet-ai/skills/mcp-csharp-publish/ @leslierichardson95 @artl93 +/tests/dotnet-ai/mcp-csharp-publish/ @leslierichardson95 @artl93 + +/plugins/dotnet-ai/skills/mcp-csharp-test/ @leslierichardson95 @artl93 +/tests/dotnet-ai/mcp-csharp-test/ @leslierichardson95 @artl93 + # dotnet-upgrade (migrating and upgrading .NET projects) /plugins/dotnet-upgrade/skills/thread-abort-migration/ @dotnet/appmodel /tests/dotnet-upgrade/thread-abort-migration/ @dotnet/appmodel diff --git a/.github/workflows/evaluation.yml b/.github/workflows/evaluation.yml index 304df67b2c..c7e9da03d7 100644 --- a/.github/workflows/evaluation.yml +++ b/.github/workflows/evaluation.yml @@ -642,6 +642,14 @@ jobs: cat summary-body.md echo "" echo "[Full results]($RUN_URL)" + # If any skill failed, add a copy-paste prompt for AI-assisted investigation + if grep -q '❌' summary-body.md; then + RUN_ID="${{ github.run_id }}" + echo "" + echo "> **To investigate failures**, paste this to your AI coding agent:" + echo ">" + echo "> _Download eval artifacts with \`gh run download ${RUN_ID} --repo ${{ github.repository }} --dir /tmp/eval-results\`, then fetch https://raw.githubusercontent.com/${{ github.repository }}/main/eng/skill-validator/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first._" + fi } > consolidated-comment.md cat consolidated-comment.md >> $GITHUB_STEP_SUMMARY diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bfb2b839ab..6ff7fda26d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -288,6 +288,8 @@ dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate - Tests run automatically on pull requests that modify files under `plugins/`. The evaluation workflow discovers changed plugins and runs the skill-validator for each one. Results are posted as a PR comment and uploaded as build artifacts. +If a scenario fails or regresses, see [Investigating Results](eng/skill-validator/InvestigatingResults.md) for how to download artifacts, interpret `results.json`, and diagnose common failure patterns. + ## Writing style - Be concise and specific. diff --git a/eng/known-domains.txt b/eng/known-domains.txt index 7420f26092..486544091c 100644 --- a/eng/known-domains.txt +++ b/eng/known-domains.txt @@ -6,7 +6,8 @@ # alongside the skill content. # # Format: one domain per line. Lines starting with # are comments. -# For GitHub, use github.com// to scope to a specific repo. +# Entries containing a slash match URLs with that exact prefix followed by +# '/', '?', '#', or end-of-URL. Entries without a slash match the domain and all its subdomains. # Standards dot.net @@ -19,6 +20,7 @@ aka.ms download.sysinternals.com msdl.microsoft.com nuget.org +nugettest.org dotnetcli.blob.core.windows.net # Platforms @@ -45,7 +47,18 @@ github.com/microsoft/perfview github.com/microsoftdocs/visualstudio-docs github.com/NuGet/docs.microsoft.com-nuget +# MCP ecosystem +code.visualstudio.com +github.com/modelcontextprotocol +github.com/open-telemetry/semantic-conventions + +# Example paths used in skill documentation +github.com/yourusername +github.com/username + # Community +fluentassertions.com +npmjs.com/package/@modelcontextprotocol ollama.com stackoverflow.com xunit.net diff --git a/eng/skill-validator/InvestigatingResults.md b/eng/skill-validator/InvestigatingResults.md new file mode 100644 index 0000000000..3303743284 --- /dev/null +++ b/eng/skill-validator/InvestigatingResults.md @@ -0,0 +1,265 @@ +# Investigating Evaluation Results + +This guide is intended primarily for AI agents investigating skill evaluation failures, though humans will find it useful too. It documents the `results.json` schema, common failure patterns, and recommended fixes. + +## Using this guide with an AI agent + +This document is designed to be read by AI coding agents. When a skill evaluation has failures, the PR comment includes a ready-to-use prompt — just copy and paste it to your AI agent. The agent will download the artifacts, read this guide, analyze the results, and suggest fixes. + +If you need to run the investigation manually, follow the [Quick start](#quick-start) below. + +## Quick start + +1. **Download the results artifact:** `gh run download --repo dotnet/skills --dir ` +2. **Read `summary.md` first** for a quick overview of which scenarios passed/failed +3. **Read `results.json`** for the full metrics, agent output, assertions, and judge reasoning +4. **Identify the failure pattern** using the categories below — most failures match multiple patterns; fix them in priority order (timeouts first, then activation, then quality/rubric issues) +5. **Apply the fix** and re-run with `/evaluate` + +## Finding the artifacts + +### Via CLI (recommended for AI agents) + +Extract the workflow run ID from the **Full results** link in the PR eval comment (e.g., `https://github.com/dotnet/skills/actions/runs/23520818616` → `23520818616`), then: + +```bash +gh run download --repo dotnet/skills --dir /tmp/eval-results +``` + +This downloads all artifacts into subdirectories, each containing `results.json` and `summary.md`. + +### Via browser + +From the PR comment, click the **Full results** link to open the GitHub Actions workflow run. Then: + +1. Click on any job (e.g., `evaluate (mcp-csharp-debug)`) +2. Expand the **Upload results** step +3. Find the `Artifact download URL` in the log output +4. Download and extract + +Alternatively, scroll to the bottom of the workflow run summary page and download from the **Artifacts** section. + +## Understanding `results.json` + +Each file contains a top-level object with: + +| Field | Description | +|-------|-------------| +| `model` | Model used for agent runs | +| `judgeModel` | Model used for judging | +| `timestamp` | When the run started | +| `verdicts[]` | Array of per-skill results | + +### Verdict structure + +Each verdict contains: + +| Field | Description | +|-------|-------------| +| `skillName` | Name of the skill being evaluated | +| `passed` | Overall pass/fail | +| `scenarios[]` | Array of per-scenario comparisons | +| `overfittingResult` | Overfitting analysis (if enabled) | + +### Scenario structure + +Each scenario contains three runs and their comparison: + +| Field | Description | +|-------|-------------| +| `scenarioName` | Human-readable scenario name | +| `baseline` | Run without the skill | +| `skilledIsolated` | Run with only this skill loaded | +| `skilledPlugin` | Run with the full plugin loaded | +| `timedOut` | Whether any run hit the timeout | +| `isolatedImprovementScore` | Weighted improvement (isolated vs baseline) | +| `pluginImprovementScore` | Weighted improvement (plugin vs baseline) | +| `isolatedBreakdown` | Per-metric contribution to the score (see below) | +| `pluginBreakdown` | Per-metric contribution to the score (see below) | +| `pairwiseResult` | Judge's rubric-by-rubric comparison | +| `perRunScores` | Individual run scores (shows variance) | + +### Breakdown fields + +The `isolatedBreakdown` and `pluginBreakdown` objects show how each metric contributed to the improvement score. Each field is a raw delta (not yet weighted). The final score is computed as a weighted sum: + +| Field | Weight | Range | Meaning | +|-------|--------|-------|---------| +| `qualityImprovement` | 0.40 | [-1, 1] | Rubric-based quality delta | +| `overallJudgmentImprovement` | 0.30 | [-1, 1] | Holistic judge assessment delta | +| `taskCompletionImprovement` | 0.15 | {-1, 0, 1} | Did assertions pass? | +| `tokenReduction` | 0.05 | [-1, 1] | Positive = fewer tokens (more efficient) | +| `errorReduction` | 0.05 | [-1, 1] | Positive = fewer errors | +| `toolCallReduction` | 0.025 | [-1, 1] | Positive = fewer tool calls | +| `timeReduction` | 0.025 | [-1, 1] | Positive = faster | + +A `tokenReduction` of -1.0 means the skilled run used ≥2× the baseline's tokens. This is common when a skill is loaded (the skill content itself consumes tokens) but is only -0.05 in the final score, so it rarely causes failure on its own. + +### Run metrics + +Each of `baseline`, `skilledIsolated`, and `skilledPlugin` contains a `metrics` object: + +| Field | Description | +|-------|-------------| +| `timedOut` | Whether this run hit the timeout | +| `wallTimeMs` | Total wall-clock time | +| `taskCompleted` | Whether assertions passed | +| `tokenEstimate` | Total tokens used | +| `turnCount` | Number of agent turns | +| `toolCallCount` | Number of tool calls | +| `toolCallBreakdown` | Tool call counts by tool name | +| `errorCount` | Number of errors during the run | +| `assertionResults[]` | Per-assertion pass/fail with messages | +| `agentOutput` | The agent's final text output | + +## Common failure patterns + +### 1. Timeout with empty output + +**Symptoms:** +- `timedOut: true` +- `agentOutput` is empty or just `\n\n` +- All assertions fail +- `toolCallBreakdown` shows `bash` usage + +**Cause:** The model spent its entire time budget running shell commands (e.g., `dotnet new`, `dotnet add package`, exploring NuGet contents) and never produced user-facing text. + +**Fixes:** +- **Increase `timeout`** in `eval.yaml` — 180s is often not enough for scenarios that involve code generation. Try 360s. +- **Restructure the prompt** to discourage bash exploration (e.g., "Show me the code" rather than "Create a project") +- **Add `reject_tools: ["bash"]`** if the scenario should be answerable without shell commands + +### 2. Baseline already bad + +**Symptoms:** +- Baseline scores are very low (1.0–2.0/5) +- Skilled scores are also low +- Quality improvement shows 0 or negative + +**Cause:** The question is too hard for the model even without the skill. The skill can't fix what the model can't do. + +**Fixes:** +- Simplify the scenario prompt +- Verify the baseline is working by examining `baseline.metrics.agentOutput` +- Consider whether the scenario is testing the right thing + +### 3. High variance across runs + +**Symptoms:** +- `perRunScores` contains both positive and negative values (e.g., `[0.07, -0.85, 0.04]`) +- A spread greater than ~0.3 between min and max scores suggests problematic variance +- Results flip between passing and failing across eval runs +- Isolated and plugin scores disagree + +**Cause:** LLM non-determinism. The model takes different strategies on different runs. + +**Fixes:** +- **Increase `--runs`** for more statistical stability (5 is the default; consider 7–10 for noisy scenarios) +- **Tighten the prompt** to reduce the space of valid strategies +- **Add `setup.files`** to give the model concrete files to work with rather than letting it scaffold from scratch + +### 4. Quality unchanged but weighted score negative + +**Symptoms:** +- Footnote says "Quality unchanged but weighted score is -X% due to: judgment, tokens, tool calls" +- The skilled output is roughly as good as baseline + +**Cause:** The skill adds token overhead (the skill content itself uses tokens) but doesn't improve quality enough to offset it. + +**Fixes:** +- **Improve the skill content** to produce clearly better output for this scenario +- **Reduce skill size** — shorter skills have less token overhead +- **Check if the rubric matches** what the skill actually teaches + +### 5. Skill not activated + +**Symptoms:** +- Skills Loaded column shows `⚠️ NOT ACTIVATED` +- Skilled run has near-zero tokens (e.g., <100), 0 turns, 0 tools +- The `turnCount` being 0 is the clearest signal — a small token count with 0 turns indicates the skill was loaded but the agent never ran + +**Cause:** The agent runtime didn't select the skill for this prompt. The skill's frontmatter `description` didn't match. + +**Fixes:** +- Update the skill's `description` in SKILL.md frontmatter to better match the scenario prompt +- Make sure the description includes keywords from the scenario +- Check the scenario itself has sufficient information that the agent can reason that it needs the skill. (It should not cheat and suggest the skill.) + +### 6. Rubric penalizes valid alternatives + +**Symptoms:** +- Pairwise judge picks baseline over skill +- Both outputs are correct but use different approaches +- `pairwiseResult.rubricResults` shows the rubric criterion is too narrow + +**Cause:** The rubric item favors one specific approach (e.g., step-by-step UI walkthrough) over an equally valid alternative (e.g., single CLI command). + +**Fixes:** +- **Broaden the rubric** to explicitly accept multiple valid approaches +- Example: Instead of `"Shows step-by-step UI configuration"`, use `"Explains how to connect — either as a single CLI command or via the UI configuration"` + +### 7. Judge regressions on close calls + +**Symptoms:** +- `overallJudgmentImprovement` is -0.4 even though quality scores are similar +- Pairwise judge is inconsistent between position-swapped runs + +**Cause:** When outputs are nearly equal, the judge's position bias can dominate. The position-swap mitigation defaults to "tie" on inconsistency, but the weighted scoring still penalizes. + +**Fixes:** +- This is usually noise — re-run the eval to see if it persists +- If it consistently happens, improve the skill to produce clearly differentiated output + +## When multiple patterns apply + +Most failing scenarios match 2–3 patterns simultaneously (e.g., timeout + token overhead + high variance). Fix them in this priority order: + +1. **Timeouts (#1)** — if the model can't finish, nothing else matters. Increase timeout first. +2. **Skill not activated (#5)** — if the skill never loaded, fix the description before tuning anything else. +3. **Baseline already bad (#2)** — if the baseline scores ≤2.0/5, the scenario may need simplification regardless of the skill. +4. **High variance (#3)** — if `perRunScores` are unstable, a single eval run is unreliable. Re-run before concluding the skill is broken. +5. **Rubric/judgment issues (#6, #7)** — once the runs are stable, tune the rubric. +6. **Token overhead (#4)** — only optimize if quality is already good but the weighted score is marginally negative. + +## Analyzing results with an AI agent + +The `results.json` file is designed to be machine-readable. An AI agent can: + +1. **Parse the JSON** and extract metrics for each scenario +2. **Compare baseline vs skilled** metrics to identify regressions +3. **Read `agentOutput`** to see what the model actually produced +4. **Check `assertionResults`** to see which assertions failed +5. **Read `pairwiseResult.rubricResults`** for the judge's per-criterion reasoning +6. **Examine `perRunScores`** to assess variance +7. **Look at `toolCallBreakdown`** to understand what the model spent time on +8. **Cross-reference `isolatedBreakdown`** to see which metrics drove the score + +### Example analysis script + +```python +import json + +def analyze(path): + with open(path) as f: + data = json.load(f) + for verdict in data['verdicts']: + for scenario in verdict['scenarios']: + name = scenario['scenarioName'] + bl = scenario['baseline']['metrics'] + sk = scenario['skilledIsolated']['metrics'] + print(f"--- {name} ---") + print(f" Baseline: timedOut={bl['timedOut']}, output={len(bl.get('agentOutput',''))} chars") + print(f" Skilled: timedOut={sk['timedOut']}, output={len(sk.get('agentOutput',''))} chars") + print(f" Improvement: {scenario.get('isolatedImprovementScore', 0):.1%}") + for a in bl.get('assertionResults', []): + status = 'PASS' if a['passed'] else 'FAIL' + print(f" Baseline assertion [{status}]: {a['message']}") + +analyze('results.json') +``` + +## See also + +- [skill-validator README](README.md) — CLI usage, eval file format, scoring weights +- [Overfitting detection](OverfittingDetection.md) — how overfitting scores are computed +- [CONTRIBUTING.md](../../CONTRIBUTING.md) — writing eval files and running tests locally diff --git a/eng/skill-validator/README.md b/eng/skill-validator/README.md index 2b96ff5ba3..b1e30527ca 100644 --- a/eng/skill-validator/README.md +++ b/eng/skill-validator/README.md @@ -155,6 +155,8 @@ Results are displayed in the console with color-coded scores and metric deltas. - `junit` — `results.xml` with JUnit XML test results - `markdown` — `summary.md` with a results table, plus per-skill directories with per-scenario judge reports +See [Investigating Results](InvestigatingResults.md) for how to diagnose poor scores, download artifacts, and interpret `results.json`. + ### Consolidating results across matrix jobs When evaluating multiple plugins in parallel CI matrix jobs, use the `consolidate` subcommand to merge individual `results.json` files into a single markdown summary: diff --git a/eng/skill-validator/src/Evaluate/Reporter.cs b/eng/skill-validator/src/Evaluate/Reporter.cs index 59f06679c0..7e08a37b60 100644 --- a/eng/skill-validator/src/Evaluate/Reporter.cs +++ b/eng/skill-validator/src/Evaluate/Reporter.cs @@ -609,6 +609,10 @@ public static string GenerateMarkdownSummary( sb.AppendLine($"\nModel: {model ?? "unknown"} | Judge: {judgeModel ?? "unknown"}"); + bool anyFailure = verdicts.Any(v => !v.Passed); + if (anyFailure) + sb.AppendLine("\n> 📖 See [InvestigatingResults.md](https://github.com/dotnet/skills/blob/main/eng/skill-validator/InvestigatingResults.md) for how to diagnose failures — or use the copy-paste prompt below."); + return sb.ToString(); } diff --git a/plugins/dotnet-ai/skills/mcp-csharp-create/SKILL.md b/plugins/dotnet-ai/skills/mcp-csharp-create/SKILL.md new file mode 100644 index 0000000000..8c471c935c --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-create/SKILL.md @@ -0,0 +1,265 @@ +--- +name: mcp-csharp-create +description: > + Create MCP servers using the C# SDK and .NET project templates. Covers scaffolding, + tool/prompt/resource implementation, and transport configuration for stdio and HTTP. + USE FOR: creating new MCP server projects, scaffolding with dotnet new mcpserver, adding + MCP tools/prompts/resources, choosing stdio vs HTTP transport, configuring MCP hosting in + Program.cs, setting up ASP.NET Core MCP endpoints with MapMcp. + DO NOT USE FOR: debugging or running existing servers (use mcp-csharp-debug), writing tests + (use mcp-csharp-test), publishing or deploying (use mcp-csharp-publish), building MCP + clients, non-.NET MCP servers. +--- + +# C# MCP Server Creation + +Create Model Context Protocol servers using the official C# SDK (`ModelContextProtocol` NuGet package) and the `dotnet new mcpserver` project template. Servers expose tools, prompts, and resources that LLMs can discover and invoke via the MCP protocol. + +## When to Use + +- Starting a new MCP server project from scratch +- Adding tools, prompts, or resources to an existing MCP server +- Choosing between stdio (`--transport local`) and HTTP (`--transport remote`) transport +- Setting up ASP.NET Core hosting for an HTTP MCP server +- Wrapping an external API or service as MCP tools + +## Stop Signals + +- **Server already exists and needs debugging?** → Use `mcp-csharp-debug` +- **Need tests or evaluations?** → Use `mcp-csharp-test` +- **Ready to publish?** → Use `mcp-csharp-publish` +- **Building an MCP client, not a server** → This skill is server-side only + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Transport type | Yes | `stdio` (local/CLI) or `http` (remote/web). Ask user if not specified — default to stdio | +| Project name | Yes | PascalCase name for the project (e.g., `WeatherMcpServer`) | +| .NET SDK version | Recommended | .NET 10.0+ required. Check with `dotnet --version` | +| Service/API to wrap | Recommended | External API or service the tools will interact with | + +## Workflow + +> **Commit strategy:** Commit after completing each step so scaffolding and implementation are separately reviewable. + +### Step 1: Verify prerequisites + +1. Confirm .NET 10+ SDK: `dotnet --version` (install from https://dotnet.microsoft.com if < 10.0) + +2. Check if the MCP server template is already installed: + ```bash + dotnet new list mcpserver + ``` + If "No templates found" → install: `dotnet new install Microsoft.McpServer.ProjectTemplates` + +### Step 2: Choose transport + +| Choose **stdio** if… | Choose **HTTP** if… | +|----------------------|---------------------| +| Local CLI tool or IDE plugin | Cloud/web service deployment | +| Single user at a time | Multiple simultaneous clients | +| Running as subprocess (VS Code, GitHub Copilot) | Cross-network access needed | +| Simpler setup, no network config | Containerized deployment (Docker/Azure) | + +**Default:** stdio — simpler, works for most local development. Users can add HTTP later. + +### Step 3: Scaffold the project + +**stdio server:** +```bash +dotnet new mcpserver -n +``` +If the template times out or is unavailable, use `dotnet new console -n ` and add `dotnet add package ModelContextProtocol`. + +**HTTP server:** +```bash +dotnet new web -n +cd +dotnet add package ModelContextProtocol.AspNetCore +``` +This is the recommended approach — faster and more reliable than the template. The template also supports HTTP via `dotnet new mcpserver -n --transport remote`, but `dotnet new web` gives you more control over the project structure. + +**Template flags reference:** `--transport local` (stdio, default), `--transport remote` (ASP.NET Core HTTP), `--aot`, `--self-contained`. + +### Step 4: Implement tools + +Tools are the primary way MCP servers expose functionality. Add a class with `[McpServerToolType]` and methods with `[McpServerTool]`: + +```csharp +using ModelContextProtocol.Server; +using System.ComponentModel; + +[McpServerToolType] +public static class MyTools +{ + [McpServerTool, Description("Brief description of what the tool does.")] + public static async Task DoSomething( + [Description("What this parameter controls")] string input, + CancellationToken cancellationToken = default) + { + // Implementation + return $"Result: {input}"; + } +} +``` + +**Critical rules:** +- Every tool method **must** have a `[Description]` attribute — LLMs use this to decide when to call the tool +- Every parameter **must** have a `[Description]` attribute +- Accept `CancellationToken` in all async tools +- Use `[McpServerTool(Name = "custom_name")]` only if the default method name is unclear + +**DI injection patterns** — the SDK supports two styles: + +1. **Method parameter injection (static class):** DI services appear as method parameters. The SDK resolves them automatically — they do not appear in the tool schema. + +2. **Constructor injection (non-static class):** Use when tools need shared state or multiple services: +```csharp +[McpServerToolType] +public class ApiTools(HttpClient httpClient, ILogger logger) +{ + [McpServerTool, Description("Fetch a resource by ID.")] + public async Task FetchResource( + [Description("Resource identifier")] string id, + CancellationToken cancellationToken = default) + { + logger.LogInformation("Fetching {Id}", id); + return await httpClient.GetStringAsync($"/api/{id}", cancellationToken); + } +} +``` +Register services in Program.cs: +```csharp +var builder = Host.CreateApplicationBuilder(args); +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); + +builder.Services.AddHttpClient(); // registers IHttpClientFactory + HttpClient +// ILogger is registered by default — no extra setup needed. + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithToolsFromAssembly(); // discovers non-static [McpServerToolType] classes + +await builder.Build().RunAsync(); +``` + +**For the full attribute reference, return types, DI injection, and builder API patterns**, see [references/api-patterns.md](references/api-patterns.md). + +### Step 5: Add prompts and resources (optional) + +**Prompts** — reusable LLM interaction templates: +```csharp +[McpServerPromptType] +public static class MyPrompts +{ + [McpServerPrompt, Description("Summarize content into one sentence.")] + public static ChatMessage Summarize( + [Description("Content to summarize")] string content) => + new(ChatRole.User, $"Summarize this into one sentence: {content}"); +} +``` + +**Resources** — data the LLM can read: +```csharp +[McpServerResourceType] +public static class MyResources +{ + [McpServerResource(UriTemplate = "config://app", Name = "App Config", + MimeType = "application/json"), Description("Application configuration")] + public static string GetConfig() => JsonSerializer.Serialize(AppConfig.Current); +} +``` + +### Step 6: Configure Program.cs + +**stdio transport:** +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using ModelContextProtocol.Server; + +var builder = Host.CreateApplicationBuilder(args); +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); // CRITICAL: stderr only + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithToolsFromAssembly(); + +await builder.Build().RunAsync(); +``` + +**HTTP transport:** +```csharp +using ModelContextProtocol.Server; + +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddMcpServer() + .WithHttpTransport() + .WithToolsFromAssembly(); + +// Register services your tools need via DI +// builder.Services.AddHttpClient(); +// builder.Services.AddSingleton(); + +var app = builder.Build(); +app.MapMcp(); // exposes MCP endpoint at /mcp (Streamable HTTP) +app.MapGet("/health", () => "ok"); // health check for container orchestrators +app.Run(); +``` + +**Key HTTP details:** `MapMcp()` defaults to `/mcp` path. For containers, set `ASPNETCORE_URLS=http://+:8080` and `EXPOSE 8080`. The MCP HTTP protocol uses Streamable HTTP — no special client config needed beyond the URL. + +**For transport configuration details** (stateless mode, auth, path prefix, `HttpContextAccessor`), see [references/transport-config.md](references/transport-config.md). + +### Step 7: Verify the server starts + +```bash +cd +dotnet build +dotnet run +``` + +For stdio: the process starts and waits for JSON-RPC input on stdin. +For HTTP: the server listens on the configured port. + +## Validation + +- [ ] Project builds with no errors (`dotnet build`) +- [ ] All tool classes have `[McpServerToolType]` attribute +- [ ] All tool methods have `[McpServerTool]` and `[Description]` attributes +- [ ] All parameters have `[Description]` attributes +- [ ] stdio: logging directed to stderr, not stdout +- [ ] HTTP: `app.MapMcp()` is called in Program.cs +- [ ] Server starts successfully with `dotnet run` + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| stdio server outputs garbage or hangs | Logging to stdout corrupts JSON-RPC protocol. Set `LogToStandardErrorThreshold = LogLevel.Trace` | +| Tool not discovered by LLM clients | Missing `[McpServerToolType]` on the class or `[McpServerTool]` on the method. Verify `.WithToolsFromAssembly()` in Program.cs | +| LLM doesn't understand when to use a tool | Add clear `[Description]` attributes on both the method and all parameters | +| `WithToolsFromAssembly()` fails in AOT | Reflection-based discovery is incompatible with Native AOT. Use `.WithTools()` instead | +| Parameters not appearing in tool schema | `CancellationToken`, `IMcpServer`, and DI services are injected automatically — they do not appear in the schema. Only parameters with `[Description]` are exposed | +| HTTP server returns 404 | `app.MapMcp()` must be called. Check the request path matches the configured route | + +## Related Skills + +- `mcp-csharp-debug` — Run, debug, and test with MCP Inspector +- `mcp-csharp-test` — Unit tests, integration tests, evaluations +- `mcp-csharp-publish` — NuGet, Docker, Azure deployment + +## Reference Files + +- [references/api-patterns.md](references/api-patterns.md) — Complete attribute reference, return types, DI injection, builder API, dynamic tools, experimental APIs. **Load when:** implementing tools, prompts, or resources beyond the basic patterns shown above. +- [references/transport-config.md](references/transport-config.md) — Detailed transport configuration: stateless HTTP mode, OAuth/auth, custom path prefix, `HttpContextAccessor`, OpenTelemetry observability. **Load when:** configuring advanced transport options or authentication. + +## More Info + +- [C# MCP SDK](https://github.com/modelcontextprotocol/csharp-sdk) — Official SDK repository +- [Build an MCP server (.NET)](https://learn.microsoft.com/dotnet/ai/quickstarts/build-mcp-server) — Microsoft quickstart +- [MCP Specification](https://modelcontextprotocol.io/specification/) — Protocol specification diff --git a/plugins/dotnet-ai/skills/mcp-csharp-create/references/api-patterns.md b/plugins/dotnet-ai/skills/mcp-csharp-create/references/api-patterns.md new file mode 100644 index 0000000000..f01e357c71 --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-create/references/api-patterns.md @@ -0,0 +1,148 @@ +# C# MCP SDK API Patterns + +Complete reference for MCP server implementation patterns using the C# SDK. + +## Attribute Reference + +### Tool Attributes + +| Attribute | Target | Key Properties | +|-----------|--------|----------------| +| `[McpServerToolType]` | Class | Marks class as containing tool methods | +| `[McpServerTool]` | Method | `Name`, `Title`, `Destructive`, `Idempotent`, `OpenWorld`, `ReadOnly` | +| `[Description("...")]` | Method/Parameter | From `System.ComponentModel` — provides LLM-visible descriptions | +| `[McpMeta("key", value)]` | Any | Adds `_meta` entries to the MCP protocol response | + +### Prompt Attributes + +| Attribute | Target | Key Properties | +|-----------|--------|----------------| +| `[McpServerPromptType]` | Class | Marks class as containing prompt methods | +| `[McpServerPrompt]` | Method | `Name`, `Title` | + +### Resource Attributes + +| Attribute | Target | Key Properties | +|-----------|--------|----------------| +| `[McpServerResourceType]` | Class | Marks class as containing resource methods | +| `[McpServerResource]` | Method | `UriTemplate`, `Name`, `Title`, `MimeType` | + +## Tool Return Types + +Tools can return any of these types (or their `Task`/`ValueTask` async variants): + +| Return Type | Behavior | +|-------------|----------| +| `string` | Wrapped as `TextContentBlock` | +| `TextContentBlock` | Text content with optional annotations | +| `ImageContentBlock` | Base64-encoded image data | +| `AudioContentBlock` | Base64-encoded audio data | +| `EmbeddedResourceBlock` | Resource reference | +| `CallToolResult` | Full control over content blocks and `isError` flag | +| `IEnumerable` | Multiple content blocks | + +## Injected Parameters + +These types are automatically injected by the framework and **do not appear** in the tool's JSON schema: + +| Type | Purpose | +|------|---------| +| `CancellationToken` | Cooperative cancellation | +| `IMcpServer` / `McpServer` | Access to server instance for notifications, logging | +| `RequestContext` | Full request context, progress tokens | +| `IProgress` | Report progress back to the client | +| Any DI-registered service | Constructor or method parameter injection | + +Example with DI and progress: +```csharp +[McpServerToolType] +public class MyTools(IHttpClientFactory httpFactory) +{ + [McpServerTool, Description("Fetches data from API")] + public async Task FetchData( + [Description("Resource identifier")] string resourceId, + IProgress progress, + CancellationToken cancellationToken) + { + progress.Report(new() { Progress = 0, Total = 100 }); + var client = httpFactory.CreateClient(); + var result = await client.GetStringAsync($"/api/{resourceId}", cancellationToken); + progress.Report(new() { Progress = 100, Total = 100 }); + return result; + } +} +``` + +## Builder API + +The fluent builder API configures the MCP server via dependency injection: + +```csharp +services.AddMcpServer() + // Transports (choose one) + .WithStdioServerTransport() // stdio: Generic Host + .WithHttpTransport() // HTTP: ASP.NET Core + + // Register primitives (attribute-based) + .WithTools() // Specific class + .WithToolsFromAssembly() // All [McpServerToolType] in entry assembly + .WithPrompts() + .WithPromptsFromAssembly() + .WithResources() + .WithResourcesFromAssembly() + + // Register primitives (handler-based) + .WithListToolsHandler(async (ctx, ct) => { ... }) + .WithCallToolHandler(async (ctx, ct) => { ... }) + + // Middleware + .WithRequestFilters(filters => { ... }); +``` + +> **AOT warning:** `.WithToolsFromAssembly()` uses reflection and is not compatible with Native AOT. Use `.WithTools()` for AOT scenarios. + +## Dynamic Tool Creation + +Create tools at runtime without attribute-decorated classes: + +```csharp +var tool = McpServerTool.Create( + (int count, string prefix) => Enumerable.Range(1, count).Select(i => $"{prefix}-{i}"), + new McpServerToolCreateOptions { Name = "generate_ids", Description = "Generate sequential IDs" }); +``` + +## McpServerOptions + +Configure server behavior via `McpServerOptions`: + +```csharp +services.AddMcpServer(options => +{ + options.ServerInfo = new() { Name = "MyServer", Version = "1.0.0" }; + options.ServerInstructions = "You are connected to MyService. Use tools to query data."; + options.Capabilities = new() + { + Tools = new() { ListChanged = true }, + Resources = new() { Subscribe = true, ListChanged = true } + }; +}); +``` + +Key properties: `ServerInfo`, `Capabilities`, `ServerInstructions`, `InitializationTimeout`, `ToolCollection`, `ResourceCollection`, `PromptCollection`. + +## Experimental APIs + +| Diagnostic ID | Feature | Suppression | +|---------------|---------|-------------| +| `MCPEXP001` | Tasks feature | `#pragma warning disable MCPEXP001` | +| `MCPEXP002` | Subclassing `McpServer`/`McpClient` | `#pragma warning disable MCPEXP002` | + +Suppress project-wide: `MCPEXP001;MCPEXP002` in `.csproj`. + +## NuGet Packages + +| Package | When to Use | +|---------|-------------| +| `ModelContextProtocol` | **Default.** Hosting, DI, attribute-based discovery | +| `ModelContextProtocol.AspNetCore` | HTTP servers with ASP.NET Core (`MapMcp()`) | +| `ModelContextProtocol.Core` | Minimum dependencies — low-level client/server APIs only | diff --git a/plugins/dotnet-ai/skills/mcp-csharp-create/references/transport-config.md b/plugins/dotnet-ai/skills/mcp-csharp-create/references/transport-config.md new file mode 100644 index 0000000000..70181831f9 --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-create/references/transport-config.md @@ -0,0 +1,141 @@ +# Transport Configuration + +Detailed configuration for stdio and HTTP transports in C# MCP servers. + +## Stdio Transport (Generic Host) + +Uses `Microsoft.Extensions.Hosting` for the application lifecycle: + +```csharp +var builder = Host.CreateApplicationBuilder(args); + +// CRITICAL: All logging must go to stderr — stdout is reserved for JSON-RPC +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithToolsFromAssembly(); + +await builder.Build().RunAsync(); +``` + +### Stdio Key Points + +- `stdout` carries JSON-RPC messages — **never** write anything else to stdout +- All logging, diagnostics, and debug output must use stderr +- The process runs as a subprocess of the MCP client +- No network configuration needed + +## HTTP Transport (ASP.NET Core) + +Uses ASP.NET Core with Streamable HTTP (default) or SSE (legacy): + +```csharp +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddMcpServer() + .WithHttpTransport() + .WithToolsFromAssembly(); + +var app = builder.Build(); +app.MapMcp(); +app.Run(); +``` + +### Custom Path Prefix + +```csharp +app.MapMcp("/custom-mcp-path"); +``` + +### Stateless Mode + +Disables session state — each request is independent: + +```csharp +builder.Services.AddMcpServer() + .WithHttpTransport(options => options.Stateless = true); +``` + +### Idle Timeout + +Configure session cleanup: + +```csharp +builder.Services.AddMcpServer() + .WithHttpTransport(options => options.IdleTimeout = TimeSpan.FromMinutes(30)); +``` + +### Port Configuration + +```csharp +app.Run("http://localhost:3001"); +// or via launchSettings.json / ASPNETCORE_URLS environment variable +``` + +## Authentication and Authorization + +### JWT Bearer Auth + +```csharp +builder.Services.AddAuthentication() + .AddJwtBearer(options => + { + options.Authority = "https://your-auth-server"; + options.Audience = "mcp-server"; + }); + +builder.Services.AddAuthorization(); + +var app = builder.Build(); +app.UseAuthentication(); +app.UseAuthorization(); +app.MapMcp().RequireAuthorization(); +``` + +### Accessing HttpContext in Tools + +Register `HttpContextAccessor` to access HTTP request details from tool methods: + +```csharp +builder.Services.AddHttpContextAccessor(); + +[McpServerToolType] +public class AuthAwareTools(IHttpContextAccessor httpContextAccessor) +{ + [McpServerTool, Description("Returns the authenticated user's ID")] + public string GetCurrentUser() + { + var user = httpContextAccessor.HttpContext?.User; + return user?.Identity?.Name ?? "anonymous"; + } +} +``` + +### OAuth 2.0 with Dynamic Client Registration + +The SDK includes a full OAuth sample in `samples/ProtectedMcpServer/` covering: +- JWT bearer token validation +- OAuth 2.0 authorization flows +- Dynamic Client Registration (RFC 7591) + +## OpenTelemetry Observability + +Built-in distributed tracing and metrics: + +| Component | Name | +|-----------|------| +| `ActivitySource` | `Experimental.ModelContextProtocol` | +| `Meter` | `Experimental.ModelContextProtocol` | + +```csharp +builder.Services.AddOpenTelemetry() + .WithTracing(tracing => tracing + .AddSource("Experimental.ModelContextProtocol") + .AddAspNetCoreInstrumentation()) + .WithMetrics(metrics => metrics + .AddMeter("Experimental.ModelContextProtocol")); +``` + +Trace context propagated via `_meta.traceparent` across client/server boundaries. +Metrics follow [MCP semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/mcp.md#metrics). diff --git a/plugins/dotnet-ai/skills/mcp-csharp-debug/SKILL.md b/plugins/dotnet-ai/skills/mcp-csharp-debug/SKILL.md new file mode 100644 index 0000000000..0b6e806f44 --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-debug/SKILL.md @@ -0,0 +1,236 @@ +--- +name: mcp-csharp-debug +description: > + Run and debug C# MCP servers locally. Covers IDE configuration, MCP Inspector testing, + GitHub Copilot Agent Mode integration, logging setup, and troubleshooting. + USE FOR: running MCP servers locally with dotnet run, configuring VS Code or Visual Studio + for MCP debugging, testing tools with MCP Inspector, testing with GitHub Copilot Agent Mode, + diagnosing tool registration issues, setting up mcp.json configuration, debugging MCP + protocol messages, configuring logging for stdio and HTTP servers. + DO NOT USE FOR: creating new MCP servers (use mcp-csharp-create), writing automated tests + (use mcp-csharp-test), publishing or deploying to production (use mcp-csharp-publish). +--- + +# C# MCP Server Debugging + +Run, debug, and interactively test C# MCP servers. Covers local execution, IDE debugging with breakpoints, MCP Inspector for protocol-level testing, and GitHub Copilot Agent Mode integration. + +## When to Use + +- Running an MCP server locally for the first time +- Configuring VS Code or Visual Studio to debug an MCP server +- Testing tools interactively with MCP Inspector +- Verifying tools appear in GitHub Copilot Agent Mode +- Diagnosing issues: tools not discovered, protocol errors, server crashes +- Setting up `mcp.json` or `.mcp.json` configuration + +## Stop Signals + +- **No project yet?** → Use `mcp-csharp-create` first +- **Need automated tests?** → Use `mcp-csharp-test` +- **Production deployment issue?** → Use `mcp-csharp-publish` + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Project path | Yes | Path to the `.csproj` file or project directory | +| Transport type | Recommended | `stdio` or `http` — detect from `.csproj` if not specified | +| IDE | Recommended | VS Code or Visual Studio — detect from environment if not specified | + +**Agent behavior:** Detect transport type by checking the `.csproj` for a `PackageReference` to `ModelContextProtocol.AspNetCore`. If present → HTTP, otherwise → stdio. + +## Workflow + +### Step 1: Run the server locally + +**stdio transport:** +```bash +cd +dotnet run +``` +The process starts and waits for JSON-RPC messages on stdin. No output on stdout means it's working correctly. + +**HTTP transport:** +```bash +cd +dotnet run +# Server listens on http://localhost:3001 (or configured port) +``` + +### Step 2: Generate MCP configuration + +Detect the IDE and transport, then create the appropriate config file. + +**For VS Code** — create `.vscode/mcp.json`: + +stdio: +```json +{ + "servers": { + "": { + "type": "stdio", + "command": "dotnet", + "args": ["run", "--project", ""] + } + } +} +``` + +HTTP: +```json +{ + "servers": { + "": { + "type": "http", + "url": "http://localhost:3001" + } + } +} +``` + +**For Visual Studio** — create `.mcp.json` at solution root (same JSON structure). + +**For detailed IDE-specific configuration** (launch.json, environment variables, secrets), see [references/ide-config.md](references/ide-config.md). + +### Step 3: Test with MCP Inspector + +The MCP Inspector provides a UI for testing tools, viewing schemas, and inspecting protocol messages. + +**stdio server:** +```bash +npx @modelcontextprotocol/inspector dotnet run --project +``` + +**HTTP server:** +1. Start your server: `dotnet run` +2. Run Inspector: `npx @modelcontextprotocol/inspector` +3. Connect to `http://localhost:3001` + +**Inspector capabilities:** +- List all registered tools, prompts, and resources +- Call tools with custom parameters and see results +- View request/response JSON-RPC messages +- Inspect tool schemas and descriptions + +**For detailed Inspector usage and troubleshooting**, see [references/mcp-inspector.md](references/mcp-inspector.md). + +### Step 4: Test with GitHub Copilot Agent Mode + +1. Open GitHub Copilot Chat → switch to **Agent** mode +2. Click **Select Tools** (wrench icon) → verify your server and tools are listed +3. Test with a prompt that should trigger your tool +4. Approve tool execution when prompted + +**If tools don't appear — troubleshoot tool discovery:** + +1. **Rebuild first** — stale builds are the #1 cause: + ```bash + dotnet build + ``` + Then restart the MCP server (click Stop → Start in VS Code, or restart `dotnet run`). + +2. **Check `[McpServerToolType]` on the class:** + ```csharp + [McpServerToolType] // ← Required on the class + public class MyTools { ... } + ``` + +3. **Check `[McpServerTool]` on each tool method:** + - The method must be `public`. + - It can be `static` or instance. For instance methods, ensure the containing type is discoverable/registered (for example via `WithTools()` or `WithToolsFromAssembly()`) so DI can construct it. + ```csharp + [McpServerTool, Description("Does something")] + public string DoSomething(string input) => input; + ``` + +4. **Verify tool registration in Program.cs** — use one of: + ```csharp + .WithTools() // register specific class + .WithToolsFromAssembly() // scan entire assembly for [McpServerToolType] + ``` + +5. **Check `mcp.json`** points to the correct project path + +6. If still not appearing, reference the tool explicitly: `Using #tool_name, do X` + +### Step 5: Set up breakpoint debugging + +1. Set breakpoints in your tool methods +2. Launch with the debugger: + - **VS Code:** F5 (requires `launch.json` — see [references/ide-config.md](references/ide-config.md)) + - **Visual Studio:** F5 or right-click project → Debug → Start +3. Trigger the tool (via Inspector, Copilot, or test client) +4. Execution pauses at breakpoints + +**Critical:** Build in Debug configuration. Breakpoints won't hit in Release builds. + +### Step 6: Configure logging + +**Critical for stdio transport:** Any output to stdout (including `Console.WriteLine`) **corrupts the MCP JSON-RPC protocol** and causes garbled responses or crashes. All logging and diagnostic output must go to stderr. + +**stdio transport** — log to stderr only: +```csharp +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); +``` + +**HTTP transport** — standard console logging: +```csharp +builder.Logging.ClearProviders(); +builder.Logging.AddConsole(); +builder.Logging.SetMinimumLevel( + builder.Environment.IsDevelopment() ? LogLevel.Debug : LogLevel.Information); +``` + +**In tool methods** — inject `ILogger` via constructor: +```csharp +[McpServerToolType] +public class MyTools(ILogger logger) +{ + [McpServerTool, Description("Processes data")] + public string ProcessData(string input) + { + logger.LogDebug("Processing: {Input}", input); + return DoProcessing(input); + } +} +``` + +## Validation + +- [ ] Server starts without errors via `dotnet run` +- [ ] MCP Inspector connects and lists all expected tools +- [ ] Tool calls via Inspector return expected results +- [ ] Breakpoints hit when debugging in IDE +- [ ] Tools appear in GitHub Copilot Agent Mode tool list +- [ ] stdio: no logging output on stdout (stderr only) + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Tools not appearing in Copilot or Inspector | **Rebuild first:** `dotnet build`, then restart the server. If still missing, verify `[McpServerToolType]` on class, `[McpServerTool]` on methods, and `WithTools()` or `WithToolsFromAssembly()` in Program.cs | +| stdio server produces garbled output | `Console.WriteLine()` or logging is writing to stdout. All output **must** go to stderr. Set `LogToStandardErrorThreshold = LogLevel.Trace` on the console logger | +| "Command not found" when starting server | .NET 10+ SDK not installed. Check with `dotnet --version` | +| HTTP server returns 404 at MCP endpoint | Missing `app.MapMcp()` in Program.cs | +| Breakpoints not hit | Building in Release mode. Rebuild in Debug: `dotnet build -c Debug`, then restart | +| Environment variables not passed to server | Add `"env"` section to `mcp.json`. For secrets in VS Code, use `"${input:var_id}"` syntax | +| MCP Inspector can't connect to HTTP server | Server not running, or wrong port. Check `dotnet run` output for the listening URL | +| Stale tools after code changes | Always `dotnet build` and restart the server after changing tool methods or attributes | + +## Related Skills + +- `mcp-csharp-create` — Create a new MCP server project +- `mcp-csharp-test` — Automated tests and evaluations +- `mcp-csharp-publish` — NuGet, Docker, Azure deployment + +## Reference Files + +- [references/mcp-inspector.md](references/mcp-inspector.md) — Detailed MCP Inspector usage: installation, connecting to servers, feature walkthrough, troubleshooting. **Load when:** user needs detailed Inspector guidance or is having connection issues. +- [references/ide-config.md](references/ide-config.md) — Complete VS Code and Visual Studio configuration: mcp.json templates, launch.json, environment variables, conditional breakpoints. **Load when:** setting up IDE debugging or configuring environment-specific settings. + +## More Info + +- [MCP Inspector](https://www.npmjs.com/package/@modelcontextprotocol/inspector) — Interactive debugging tool for MCP servers +- [VS Code MCP documentation](https://code.visualstudio.com/docs/copilot/chat/mcp-servers) — Configuring MCP servers in VS Code diff --git a/plugins/dotnet-ai/skills/mcp-csharp-debug/references/ide-config.md b/plugins/dotnet-ai/skills/mcp-csharp-debug/references/ide-config.md new file mode 100644 index 0000000000..b807327cf2 --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-debug/references/ide-config.md @@ -0,0 +1,153 @@ +# IDE Configuration + +Complete configuration for debugging C# MCP servers in VS Code and Visual Studio. + +## VS Code Configuration + +### mcp.json (MCP Server Registration) + +Create `.vscode/mcp.json` to register your server with VS Code and GitHub Copilot: + +**stdio transport:** +```json +{ + "servers": { + "MyMcpServer": { + "type": "stdio", + "command": "dotnet", + "args": [ + "run", + "--project", + "MyMcpServer/MyMcpServer.csproj" + ], + "env": { + "API_KEY": "${input:api_key}" + } + } + }, + "inputs": [ + { + "type": "promptString", + "id": "api_key", + "description": "API key for the service", + "password": true + } + ] +} +``` + +**HTTP transport:** +```json +{ + "servers": { + "MyMcpServer": { + "type": "http", + "url": "http://localhost:3001", + "headers": {} + } + } +} +``` + +### launch.json (Debugger Configuration) + +Create `.vscode/launch.json` for F5 debugging: + +```json +{ + "version": "0.2.0", + "configurations": [ + { + "name": "Debug MCP Server", + "type": "coreclr", + "request": "launch", + "program": "${workspaceFolder}/MyMcpServer/bin/Debug/net10.0/MyMcpServer.dll", + "args": [], + "cwd": "${workspaceFolder}/MyMcpServer", + "console": "integratedTerminal", + "stopAtEntry": false, + "env": { + "DOTNET_ENVIRONMENT": "Development", + "API_KEY": "your-dev-api-key" + } + } + ] +} +``` + +### Attach to Running Process + +For debugging a server started by VS Code's MCP integration: + +```json +{ + "name": "Attach to MCP Server", + "type": "coreclr", + "request": "attach", + "processName": "MyMcpServer" +} +``` + +### Environment Variable Patterns + +| Pattern | Usage | +|---------|-------| +| `"API_KEY": "literal-value"` | Direct value (dev only) | +| `"API_KEY": "${input:api_key}"` | Prompt user for value | +| `"API_KEY": "${env:API_KEY}"` | Read from system environment | + +## Visual Studio Configuration + +### 1. Register MCP Server + +1. Open **GitHub Copilot Chat** (top right icon) +2. Click **Select Tools** (wrench icon) +3. Click **+** to add a custom MCP server +4. Configure: + - **Destination**: Solution or Global + - **Server ID**: Your server name + - **Type**: stdio or HTTP + - For stdio: **Command**: `dotnet run --project path/to/project.csproj` + - For HTTP: **URL**: `http://localhost:3001` + +This creates a `.mcp.json` file in your solution root or global config. + +### 2. Debug with F5 + +1. Right-click your MCP server project → **Set as Startup Project** +2. Open **Properties** → **Debug** → **General** → **Open debug launch profiles UI** +3. Add environment variables as needed +4. Set breakpoints in tool methods +5. Press **F5** to start debugging + +### 3. Conditional Breakpoints + +Right-click a breakpoint → **Conditions**: +- **Condition**: `query == "test"` — break only for specific input +- **Hit Count**: `>= 5` — break after N invocations +- **Filter**: `ProcessName == "MyMcpServer"` — filter by process + +## Auto-Detect and Generate mcp.json + +Script to auto-detect transport and generate config: + +```powershell +$proj = (Get-ChildItem *.csproj | Select-Object -First 1) +$name = $proj.BaseName +$isHttp = (Get-Content $proj.FullName -Raw) -match 'ModelContextProtocol\.AspNetCore' +$configDir = if (Test-Path .vscode) { ".vscode" } else { "." } +$configPath = Join-Path $configDir "mcp.json" + +if ($isHttp) { + $port = "3001" + $programCs = Get-Content "Program.cs" -Raw -ErrorAction SilentlyContinue + if ($programCs -match 'localhost:(\d+)') { $port = $matches[1] } + $config = @{ servers = @{ $name = @{ type = "http"; url = "http://localhost:$port" } } } +} else { + $config = @{ servers = @{ $name = @{ type = "stdio"; command = "dotnet"; args = @("run", "--project", $proj.Name) } } } +} + +New-Item -ItemType Directory -Path $configDir -Force | Out-Null +$config | ConvertTo-Json -Depth 5 | Out-File $configPath -Encoding utf8 +Write-Host "Created $configPath for $( if ($isHttp) {'HTTP'} else {'stdio'} ) server" +``` diff --git a/plugins/dotnet-ai/skills/mcp-csharp-debug/references/mcp-inspector.md b/plugins/dotnet-ai/skills/mcp-csharp-debug/references/mcp-inspector.md new file mode 100644 index 0000000000..0f3ef75918 --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-debug/references/mcp-inspector.md @@ -0,0 +1,101 @@ +# MCP Inspector + +Interactive debugging tool for testing MCP servers. Provides a web UI for listing tools, calling them with custom parameters, and inspecting protocol messages. + +## Installation + +Requires Node.js (npm/npx). No global install needed: + +```bash +npx @modelcontextprotocol/inspector +``` + +## Connecting to a Server + +### stdio Server + +Pass the server command directly: +```bash +npx @modelcontextprotocol/inspector dotnet run --project +``` + +The Inspector launches the server process and communicates via stdin/stdout. + +### HTTP Server + +1. Start the server separately: + ```bash + cd + dotnet run + ``` + +2. Launch Inspector and connect to the server URL: + ```bash + npx @modelcontextprotocol/inspector + ``` + +3. In the Inspector UI, enter the server URL (e.g., `http://localhost:3001`) + +### File-Based Server (.NET 10+ only) + +For single-file servers using `#:package` directives: +```bash +npx @modelcontextprotocol/inspector ./Program.cs +``` + +## Features + +### Tool Testing + +1. Click **Tools** tab to see all registered tools +2. View each tool's JSON schema (parameters, types, descriptions) +3. Enter parameter values and click **Call** to execute +4. View the return value and any error details + +### Prompt Testing + +1. Click **Prompts** tab to see registered prompts +2. Fill in prompt arguments +3. View the generated messages + +### Resource Browsing + +1. Click **Resources** tab to list registered resources +2. Read resource contents directly in the UI + +### Protocol Inspection + +- View raw JSON-RPC request/response messages +- Inspect headers and metadata +- See timing information for each request + +## Troubleshooting + +### Inspector won't start + +- Verify Node.js is installed: `node --version` +- Try clearing npx cache: `npx clear-npx-cache` then retry + +### Can't connect to stdio server + +- Verify the server builds: `dotnet build` (fix errors first) +- Check that the server doesn't write to stdout (logging must go to stderr for stdio transport) +- Try running the server directly first: `dotnet run` — if it hangs waiting for input, that's correct + +### Can't connect to HTTP server + +- Verify the server is running and the port is correct +- Check firewall/proxy settings +- Try `curl http://localhost:/` to verify basic connectivity + +### Tools not appearing + +- Verify `[McpServerToolType]` and `[McpServerTool]` attributes are present +- Check `.WithToolsFromAssembly()` or `.WithTools()` in Program.cs +- Rebuild the project and retry + +### Tool call returns error + +- Check the Inspector's protocol view for the full error message +- Common issues: missing required parameters, serialization errors, unhandled exceptions in tool code +- Add logging to the tool method and check stderr output (stdio) or console output (HTTP) diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md new file mode 100644 index 0000000000..76abca1ebd --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md @@ -0,0 +1,275 @@ +--- +name: mcp-csharp-publish +description: > + Publish and deploy C# MCP servers. Covers NuGet packaging for stdio servers, Docker + containerization for HTTP servers, Azure Container Apps and App Service deployment, + and publishing to the official MCP Registry. + USE FOR: packaging stdio MCP servers as NuGet tools, creating Dockerfiles for HTTP MCP + servers, deploying to Azure Container Apps or App Service, publishing to the MCP Registry + at registry.modelcontextprotocol.io, configuring server.json for MCP package metadata, + setting up CI/CD for MCP server publishing. + DO NOT USE FOR: publishing general NuGet libraries (not MCP-specific), general Docker + guidance unrelated to MCP, creating new servers (use mcp-csharp-create), debugging + (use mcp-csharp-debug), writing tests (use mcp-csharp-test). +--- + +# C# MCP Server Publishing + +Publish and deploy MCP servers to their target platforms. stdio servers are distributed as NuGet tool packages. HTTP servers are containerized and deployed to Azure or other container hosts. Both can optionally be listed in the official MCP Registry. + +## When to Use + +- Packaging a stdio MCP server for NuGet distribution +- Creating a Docker container for an HTTP MCP server +- Deploying to Azure Container Apps or App Service +- Publishing to the official MCP Registry for discoverability +- Setting up `server.json` metadata for the MCP Registry + +## Stop Signals + +- **Server not tested yet?** → Use `mcp-csharp-test` first +- **Server not working locally?** → Use `mcp-csharp-debug` +- **No server project yet?** → Use `mcp-csharp-create` +- **Publishing a non-MCP NuGet package?** → Use `nuget-trusted-publishing` instead + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Transport type | Yes | `stdio` → NuGet path, `http` → Docker/Azure path | +| Target destination | Yes | NuGet.org, Docker registry, Azure Container Apps, Azure App Service, MCP Registry | +| Project path | Yes | Path to the `.csproj` file | +| Package ID / server name | Required for publishing | NuGet `PackageId` or MCP Registry name | + +## Workflow + +### Step 1: Choose the publishing path + +| Transport | Primary Destination | Users Run With | +|-----------|-------------------|----------------| +| **stdio** | NuGet.org | `dnx YourPackage@version` | +| **HTTP** | Docker → Azure | Container URL | + +Both paths can optionally publish to the MCP Registry for discoverability. + +### Step 2a: NuGet publishing (stdio servers) + +1. **Configure `.csproj`** with package properties: +```xml + + true + mymcpserver + YourUsername.MyMcpServer + 1.0.0 + Your Name + MCP server for interacting with MyService + MIT + mcp;modelcontextprotocol;ai;llm + README.md + + + + + +``` + +2. **Build and pack:** +```bash +dotnet build -c Release +dotnet pack -c Release +``` + +3. **Test locally before publishing:** +```bash +dotnet tool install --global --add-source bin/Release/ YourUsername.MyMcpServer +mymcpserver --help # verify it runs +dotnet tool uninstall --global YourUsername.MyMcpServer +``` + +4. **Push to NuGet.org:** +```bash +dotnet nuget push bin/Release/*.nupkg \ + --api-key YOUR_NUGET_API_KEY \ + --source https://api.nuget.org/v3/index.json +``` + +5. **Verify** — users configure in `mcp.json`: +```json +{ + "servers": { + "MyMcpServer": { + "type": "stdio", + "command": "dnx", + "args": ["YourUsername.MyMcpServer@1.0.0", "--yes"] + } + } +} +``` + +**For detailed NuGet packaging and trusted publishing setup**, see [references/nuget-packaging.md](references/nuget-packaging.md). + +### Step 2b: Docker containerization (HTTP servers) + +1. **Create Dockerfile:** +```dockerfile +FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build +WORKDIR /src +COPY *.csproj ./ +RUN dotnet restore +COPY . ./ +RUN dotnet publish -c Release -o /app + +FROM mcr.microsoft.com/dotnet/aspnet:10.0 +WORKDIR /app +COPY --from=build /app . + +# Non-root user for security +RUN adduser --disabled-password --gecos '' appuser +USER appuser + +ENV ASPNETCORE_URLS=http://+:8080 +EXPOSE 8080 +HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \ + CMD curl -f http://localhost:8080/health || exit 1 +ENTRYPOINT ["dotnet", "MyMcpServer.dll"] +``` + +2. **Build and test locally:** +```bash +docker build -t mymcpserver:latest . +docker run -d -p 3001:8080 -e API_KEY=test-key --name mymcpserver mymcpserver:latest +curl http://localhost:3001/health +``` + +3. **Push to container registry:** +```bash +# Docker Hub +docker tag mymcpserver:latest /:1.0.0 +docker push /:1.0.0 + +# Azure Container Registry +az acr login --name yourregistry +docker tag mymcpserver:latest .azurecr.io/:1.0.0 +docker push .azurecr.io/:1.0.0 +``` + +### Step 3: Deploy to Azure (HTTP servers) + +**Azure Container Apps** (recommended — serverless with auto-scaling): +```bash +az containerapp create \ + --name mymcpserver \ + --resource-group mygroup \ + --environment myenvironment \ + --image .azurecr.io/:1.0.0 \ + --target-port 8080 \ + --ingress external \ + --min-replicas 0 \ + --max-replicas 10 \ + --secrets api-key=my-actual-api-key \ + --env-vars API_KEY=secretref:api-key +``` + +**Azure App Service** (traditional web hosting): +```bash +az webapp create \ + --name mymcpserver \ + --resource-group mygroup \ + --plan myplan \ + --deployment-container-image-name .azurecr.io/:1.0.0 +``` + +**For detailed Azure deployment**, see [references/docker-azure.md](references/docker-azure.md). + +### Step 4: Publish to MCP Registry (optional) + +List your server in the official MCP Registry for discoverability. + +1. **Install `mcp-publisher`:** +```bash +# macOS/Linux +brew install mcp-publisher + +# Or download from https://github.com/modelcontextprotocol/registry/releases +``` + +2. **Create `.mcp/server.json`** (or run `mcp-publisher init` to generate interactively): +```json +{ + "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", + "name": "io.github.username/servername", + "description": "Your server description", + "version": "1.0.0", + "packages": [{ + "registryType": "nuget", + "registryBaseUrl": "https://api.nuget.org", + "identifier": "YourUsername.MyMcpServer", + "version": "1.0.0", + "transport": { "type": "stdio" } + }], + "repository": { + "url": "https://github.com/username/repo", + "source": "github" + } +} +``` + +> **Version consistency (critical):** The root `version`, `packages[].version`, and `` in `.csproj` **must all match**. A mismatch causes registry validation failures or users downloading the wrong version. + +3. **Authenticate and publish:** +```bash +mcp-publisher login github # name must be io.github./... for GitHub auth +mcp-publisher publish +``` + +4. **Verify:** +```bash +curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github./" +``` + +**For Registry details** (namespace conventions, environment variables, CI/CD automation), see [references/mcp-registry.md](references/mcp-registry.md). + +### Step 5: Security checklist + +- [ ] No hardcoded secrets — use environment variables or Key Vault +- [ ] HTTPS enabled for HTTP transport in production +- [ ] Health check endpoint implemented +- [ ] Input validation on all tool parameters +- [ ] Rate limiting considered for HTTP servers + +## Validation + +- [ ] **NuGet:** Package installs and runs via `dnx PackageId@version` +- [ ] **Docker:** Container starts and health check passes +- [ ] **Azure:** Server is reachable and tools respond +- [ ] **MCP Registry:** Server appears at `registry.modelcontextprotocol.io` +- [ ] MCP client can connect and call tools on the deployed server + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| NuGet package doesn't run as a tool | Missing `true` in `.csproj` | +| Version mismatch between `.csproj` and `server.json` | Keep ``, `server.json` root `version`, and `packages[].version` in sync | +| Docker container exits immediately | Check entrypoint DLL name matches project output. Run `docker logs mymcpserver` for errors | +| Azure Container App returns 502 | Target port mismatch. Ensure `--target-port` matches `ASPNETCORE_URLS` port in the container | +| MCP Registry rejects publish | Name must follow namespace convention: `io.github./` for GitHub auth | +| API keys leaked in Docker image | Use multi-stage builds. Never `COPY` `.env` files. Pass secrets via `--env-vars` at runtime | + +## Related Skills + +- `mcp-csharp-create` — Create a new MCP server project +- `mcp-csharp-debug` — Running and interactive debugging +- `mcp-csharp-test` — Automated tests and evaluations + +## Reference Files + +- [references/nuget-packaging.md](references/nuget-packaging.md) — Complete NuGet `.csproj` configuration, `server.json` for MCP, NuGet.org push, testing with `dnx`, version management. **Load when:** publishing a stdio server to NuGet. +- [references/docker-azure.md](references/docker-azure.md) — Production Dockerfile patterns, ACR setup, Azure Container Apps full configuration, App Service with Key Vault, secrets management. **Load when:** deploying an HTTP server to Docker or Azure. +- [references/mcp-registry.md](references/mcp-registry.md) — `mcp-publisher` CLI installation, `server.json` schema, namespace conventions (GitHub vs DNS auth), CI/CD automation. **Load when:** publishing to the official MCP Registry. + +## More Info + +- [NuGet publishing](https://learn.microsoft.com/nuget/nuget-org/publish-a-package) — NuGet.org publishing guide +- [Azure Container Apps](https://learn.microsoft.com/azure/container-apps/) — Serverless container hosting +- [MCP Registry](https://registry.modelcontextprotocol.io) — Official MCP server registry diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/docker-azure.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/docker-azure.md new file mode 100644 index 0000000000..d7bc172f12 --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/docker-azure.md @@ -0,0 +1,187 @@ +# Docker and Azure Deployment + +Production deployment patterns for HTTP MCP servers. + +## Production Dockerfile + +```dockerfile +# Build stage +FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build +WORKDIR /src + +# Restore first (layer caching) +COPY *.csproj ./ +RUN dotnet restore + +# Build and publish +COPY . ./ +RUN dotnet publish -c Release -o /app --no-restore + +# Runtime stage +FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime +WORKDIR /app +COPY --from=build /app . + +# Non-root user (security) +RUN adduser --disabled-password --gecos '' appuser +USER appuser + +# Configure +ENV ASPNETCORE_URLS=http://+:8080 +EXPOSE 8080 + +# Health check +HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \ + CMD curl -f http://localhost:8080/health || exit 1 + +ENTRYPOINT ["dotnet", "MyMcpServer.dll"] +``` + +### Key Practices + +- **Multi-stage build** — keeps the final image small (no SDK, only runtime) +- **Restore-first layer** — `COPY *.csproj` then `dotnet restore` before `COPY .` for better layer caching +- **Non-root user** — run as unprivileged user in production +- **Health check** — enables orchestrator health monitoring +- **Never copy secrets** — no `.env` files, no `appsettings.Production.json` with secrets + +## Azure Container Registry (ACR) + +```bash +# Create ACR (one-time) +az acr create --name yourregistry --resource-group mygroup --sku Basic + +# Login +az acr login --name yourregistry + +# Build and push +docker build -t .azurecr.io/:1.0.0 . +docker push .azurecr.io/:1.0.0 + +# Or build directly in ACR (no local Docker needed) +az acr build --registry yourregistry --image mymcpserver:1.0.0 . +``` + +## Azure Container Apps + +Serverless container hosting with auto-scaling. Best for MCP servers that need to scale to zero when idle. + +### Full Setup + +```bash +# Create environment (one-time) +az containerapp env create \ + --name myenvironment \ + --resource-group mygroup \ + --location eastus + +# Create the container app +az containerapp create \ + --name mymcpserver \ + --resource-group mygroup \ + --environment myenvironment \ + --image .azurecr.io/:1.0.0 \ + --registry-server .azurecr.io \ + --target-port 8080 \ + --ingress external \ + --min-replicas 0 \ + --max-replicas 10 \ + --cpu 0.5 \ + --memory 1.0Gi \ + --secrets api-key="your-secret-value" \ + --env-vars API_KEY=secretref:api-key + +# Get the URL +az containerapp show --name mymcpserver --resource-group mygroup \ + --query properties.configuration.ingress.fqdn -o tsv +``` + +### Update Deployment + +```bash +az containerapp update \ + --name mymcpserver \ + --resource-group mygroup \ + --image .azurecr.io/:1.1.0 +``` + +### Scaling Configuration + +```bash +# Scale based on HTTP requests +az containerapp update \ + --name mymcpserver \ + --resource-group mygroup \ + --scale-rule-name http-rule \ + --scale-rule-type http \ + --scale-rule-http-concurrency 50 +``` + +## Azure App Service + +Traditional web hosting with more control over infrastructure. + +```bash +# Create App Service plan +az appservice plan create \ + --name myplan \ + --resource-group mygroup \ + --sku B1 \ + --is-linux + +# Create web app with container +az webapp create \ + --name mymcpserver \ + --resource-group mygroup \ + --plan myplan \ + --deployment-container-image-name .azurecr.io/:1.0.0 + +# Configure secrets via Key Vault +az webapp config appsettings set \ + --name mymcpserver \ + --resource-group mygroup \ + --settings API_KEY=@Microsoft.KeyVault(VaultName=myvault;SecretName=api-key) +``` + +## Secrets Management + +### Azure Container Apps + +```bash +# Add a secret +az containerapp secret set --name mymcpserver --resource-group mygroup \ + --secrets api-key="value" + +# Reference in environment variables +az containerapp update --name mymcpserver --resource-group mygroup \ + --set-env-vars API_KEY=secretref:api-key +``` + +### Azure Key Vault (App Service) + +```bash +# Create Key Vault +az keyvault create --name myvault --resource-group mygroup + +# Add secret +az keyvault secret set --vault-name myvault --name api-key --value "your-secret" + +# Grant access to App Service identity +az webapp identity assign --name mymcpserver --resource-group mygroup +az keyvault set-policy --name myvault \ + --object-id \ + --secret-permissions get list +``` + +### In Application Code + +```csharp +// Read from environment (works with both approaches) +var apiKey = Environment.GetEnvironmentVariable("API_KEY") + ?? throw new InvalidOperationException("API_KEY environment variable required"); + +// Or use Azure Key Vault directly +builder.Configuration.AddAzureKeyVault( + new Uri($"https://{vaultName}.vault.azure.net/"), + new DefaultAzureCredential()); +``` diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/mcp-registry.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/mcp-registry.md new file mode 100644 index 0000000000..ce50cb2284 --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/mcp-registry.md @@ -0,0 +1,140 @@ +# MCP Registry + +Publish your MCP server to the official registry at [registry.modelcontextprotocol.io](https://registry.modelcontextprotocol.io) for discoverability. + +## When to Publish + +| Publish if… | Skip if… | +|-------------|----------| +| Server is for public/community use | Server is internal/private | +| You want discoverability in MCP clients | Still developing/testing | +| You want to appear in the official registry | No need for public discovery | + +## Prerequisites + +1. Package published to NuGet.org (for stdio) or container registry (for HTTP) +2. GitHub repository with the server source code +3. `mcp-publisher` CLI installed + +## Install mcp-publisher + +```bash +# macOS/Linux (Homebrew) +brew install mcp-publisher + +# Or download binary from releases +# https://github.com/modelcontextprotocol/registry/releases +``` + +## server.json Schema + +Place at `.mcp/server.json` in your repository root: + +```json +{ + "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", + "name": "io.github./", + "description": "One-line description of your server", + "version": "1.0.0", + "packages": [ + { + "registryType": "nuget", + "registryBaseUrl": "https://api.nuget.org", + "identifier": "YourUsername.MyMcpServer", + "version": "1.0.0", + "transport": { + "type": "stdio" + }, + "packageArguments": [], + "environmentVariables": [ + { + "name": "API_KEY", + "value": "{api_key}", + "variables": { + "api_key": { + "description": "API key for MyService authentication", + "isRequired": true, + "isSecret": true + } + } + } + ] + } + ], + "repository": { + "url": "https://github.com/username/repo", + "source": "github" + } +} +``` + +## Namespace Conventions + +The `name` field must follow a namespace convention based on your authentication method: + +| Auth Method | Name Format | Example | +|-------------|-------------|---------| +| GitHub | `io.github.{github-username}/{server-name}` | `io.github.jsmith/weather-server` | +| DNS | `{reverse-domain}/{server-name}` | `com.mycompany/weather-server` | + +## Publish Workflow + +```bash +# 1. Initialize server.json (interactive, if not already created) +mcp-publisher init + +# 2. Authenticate with GitHub +mcp-publisher login github + +# 3. Publish to the registry +mcp-publisher publish + +# 4. Verify publication +curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github./" +``` + +## CI/CD Automation + +### GitHub Actions + +```yaml +name: Publish to MCP Registry +on: + release: + types: [published] + +jobs: + publish: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install mcp-publisher + run: | + curl -sSL https://github.com/modelcontextprotocol/registry/releases/latest/download/mcp-publisher-linux-amd64 -o mcp-publisher + chmod +x mcp-publisher + + - name: Publish to registry + run: ./mcp-publisher publish + env: + MCP_REGISTRY_TOKEN: ${{ secrets.MCP_REGISTRY_TOKEN }} +``` + +## Version Consistency + +Always keep these three versions in sync: + +1. `` in `.csproj` +2. `version` at root of `server.json` +3. `packages[].version` in `server.json` + +A mismatch between any of these will cause registry validation to fail or users to get the wrong version. + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| "Invalid name format" | Use `io.github./` format | +| "Package not found" | Package must be published to NuGet.org first | +| "Version mismatch" | Sync `.csproj` version with both `server.json` version fields | +| "Authentication failed" | Re-run `mcp-publisher login github` | diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md new file mode 100644 index 0000000000..739f485320 --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md @@ -0,0 +1,144 @@ +# NuGet Packaging + +Detailed guide for publishing stdio MCP servers as NuGet tool packages. + +## Complete .csproj Configuration + +```xml + + + Exe + net10.0 + enable + enable + + + true + mymcpserver + + + YourUsername.MyMcpServer + 1.0.0 + Your Name + MCP server for interacting with MyService API + + + https://github.com/yourusername/mymcpserver + https://github.com/yourusername/mymcpserver + MIT + mcp;modelcontextprotocol;ai;llm + README.md + + + win-x64;linux-x64;osx-x64;osx-arm64 + + + + + + +``` + +### Key Properties + +| Property | Required | Purpose | +|----------|----------|---------| +| `PackAsTool` | Yes | Makes the package installable as a dotnet tool | +| `ToolCommandName` | Recommended | CLI command name. Defaults to assembly name if omitted | +| `PackageId` | Yes | Unique identifier on NuGet.org | +| `Version` | Yes | SemVer version (e.g., `1.0.0`, `2.0.0-preview.1`) | +| `PackageTags` | Recommended | Include `mcp` and `modelcontextprotocol` for discoverability | + +## Build, Pack, and Push + +```bash +# Build +dotnet build -c Release + +# Create NuGet package +dotnet pack -c Release +# Output: bin/Release/YourUsername.MyMcpServer.1.0.0.nupkg + +# Test package locally +dotnet tool install --global --add-source bin/Release/ YourUsername.MyMcpServer +mymcpserver --help +dotnet tool uninstall --global YourUsername.MyMcpServer + +# Push to NuGet.org +dotnet nuget push bin/Release/*.nupkg \ + --api-key YOUR_NUGET_API_KEY \ + --source https://api.nuget.org/v3/index.json + +# Or push to NuGet test environment first +dotnet nuget push bin/Release/*.nupkg \ + --api-key YOUR_NUGET_API_KEY \ + --source https://apiint.nugettest.org/v3/index.json +``` + +## User Configuration + +After publishing, users configure their MCP client to run the tool: + +```json +{ + "servers": { + "MyMcpServer": { + "type": "stdio", + "command": "dnx", + "args": ["YourUsername.MyMcpServer@1.0.0", "--yes"], + "env": { + "API_KEY": "${input:api_key}" + } + } + } +} +``` + +The `dnx` tool runner (a `dotnet execute`-style runner for NuGet packages) downloads and runs the package automatically. For more details, see the .NET package execution docs: https://learn.microsoft.com/dotnet/core/tools/dotnet-execute + +## server.json for MCP Registry Integration + +If you plan to publish to the MCP Registry, include `.mcp/server.json` in your repo: + +```json +{ + "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", + "name": "io.github.yourusername/mymcpserver", + "description": "MCP server for interacting with MyService API", + "version": "1.0.0", + "packages": [ + { + "registryType": "nuget", + "registryBaseUrl": "https://api.nuget.org", + "identifier": "YourUsername.MyMcpServer", + "version": "1.0.0", + "transport": { + "type": "stdio" + }, + "environmentVariables": [ + { + "name": "API_KEY", + "value": "{api_key}", + "variables": { + "api_key": { + "description": "API key for MyService", + "isRequired": true, + "isSecret": true + } + } + } + ] + } + ], + "repository": { + "url": "https://github.com/yourusername/mymcpserver", + "source": "github" + } +} +``` + +**Version consistency:** Keep `` in `.csproj`, root `version` in `server.json`, and `packages[].version` in sync. A mismatch will cause MCP Registry validation to fail. + +## Trusted Publishing (OIDC) + +For CI/CD, use NuGet trusted publishing instead of long-lived API keys. See `nuget-trusted-publishing` skill for the full setup guide. diff --git a/plugins/dotnet-ai/skills/mcp-csharp-test/SKILL.md b/plugins/dotnet-ai/skills/mcp-csharp-test/SKILL.md new file mode 100644 index 0000000000..a373cd109d --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-test/SKILL.md @@ -0,0 +1,180 @@ +--- +name: mcp-csharp-test +description: > + Test C# MCP servers at multiple levels: unit tests for individual tools and integration + tests using the MCP client SDK. + USE FOR: unit testing MCP tool methods, integration testing with in-memory MCP + client/server, end-to-end testing via MCP protocol, + testing HTTP MCP servers with WebApplicationFactory, mocking dependencies in tool tests. + DO NOT USE FOR: testing MCP clients (this is server testing only), load or performance + testing, testing non-.NET MCP servers, debugging server issues (use mcp-csharp-debug). +--- + +# C# MCP Server Testing + +Test MCP servers at two levels: unit tests for individual tool methods, and integration tests that exercise the full MCP protocol in-memory. + +## When to Use + +- Adding automated tests to an MCP server +- Testing individual tool methods with mocked dependencies +- Writing integration tests that validate tool listing and invocation via MCP protocol +- Setting up CI test pipelines for MCP servers + +## Stop Signals + +- **No server yet?** → Use `mcp-csharp-create` first +- **Server not running?** → Use `mcp-csharp-debug` +- **Just need manual/interactive testing?** → Use `mcp-csharp-debug` for MCP Inspector + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| MCP server project path | Yes | Path to the server `.csproj` being tested | +| Test framework | Recommended | Default: xUnit. Also supports NUnit or MSTest | +| Transport type | Recommended | Determines integration test approach (stdio vs HTTP) | + +## Workflow + +### Step 1: Create the test project + +```bash +dotnet new xunit -n .Tests +cd .Tests +dotnet add reference ..//.csproj +dotnet add package ModelContextProtocol +dotnet add package Moq +dotnet add package FluentAssertions +``` + +### Step 2: Write unit tests for tool methods + +Test tool methods directly — fastest and most isolated: + +```csharp +public class MyToolTests +{ + [Fact] + public void Echo_ReturnsFormattedMessage() + { + var result = MyTools.Echo("Hello"); + result.Should().Be("Echo: Hello"); + } + + [Theory] + [InlineData("")] + [InlineData(" ")] + public void Echo_HandlesEdgeCases(string input) + { + var result = MyTools.Echo(input); + result.Should().StartWith("Echo:"); + } +} +``` + +For tools with DI dependencies, mock the dependency: +```csharp +public class ApiToolTests +{ + [Fact] + public async Task FetchData_ReturnsApiResponse() + { + var handler = new MockHttpMessageHandler("""{"id": 1}"""); + var httpClient = new HttpClient(handler); + + var result = await ApiTools.FetchData(httpClient, "resource-1"); + result.Should().Contain("id"); + } +} +``` + +### Step 3: Write integration tests with MCP client + +Test the full MCP protocol using a client-server connection: + +```csharp +using ModelContextProtocol.Client; + +public class ServerIntegrationTests : IAsyncLifetime +{ + private McpClient _client = null!; + + public async Task InitializeAsync() + { + var transport = new StdioClientTransport(new StdioClientTransportOptions + { + Name = "TestClient", + Command = "dotnet", + Arguments = ["run", "--project", "..//.csproj"] + }); + _client = await McpClient.CreateAsync(transport); + } + + public async Task DisposeAsync() => await _client.DisposeAsync(); + + [Fact] + public async Task Server_ListsExpectedTools() + { + var tools = await _client.ListToolsAsync(); + tools.Should().Contain(t => t.Name == "echo"); + } + + [Fact] + public async Task Tool_ReturnsExpectedResult() + { + var result = await _client.CallToolAsync("echo", + new Dictionary { ["message"] = "Test" }); + var text = result.Content.OfType().First().Text; + text.Should().Contain("Test"); + } +} +``` + +**For the SDK's `ClientServerTestBase` (in-memory testing) and HTTP testing with `WebApplicationFactory`**, see [references/test-patterns.md](references/test-patterns.md). + +### Step 4: Run tests + +```bash +# Run all tests +dotnet test + +# Run a specific test class +dotnet test --filter "FullyQualifiedName~MyToolTests" + +# Run with coverage +dotnet test --collect:"XPlat Code Coverage" +``` + +## Validation + +- [ ] Unit tests cover all tool methods, including edge cases +- [ ] Integration tests verify tool listing via `ListToolsAsync()` +- [ ] Integration tests verify tool invocation via `CallToolAsync()` +- [ ] All tests pass: `dotnet test` +- [ ] Tests run in CI without manual setup + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Integration test hangs on `CreateAsync` | Server fails to start. Verify `dotnet build` succeeds first. For stdio, ensure no stdout logging | +| `StdioClientTransport` not finding project | Use the correct relative path to `.csproj` from the test project directory | +| Tests pass locally but fail in CI | Run `dotnet build` before test execution. Use `--no-build` only after an explicit build step | +| Mocking `HttpClient` is awkward | Mock `HttpMessageHandler`, not `HttpClient` directly. See [references/test-patterns.md](references/test-patterns.md) | +| Full test suite runs are slow | Use `--filter` for development. Run the full suite only for CI verification | + +## Related Skills + +- `mcp-csharp-create` — Create a new MCP server project +- `mcp-csharp-debug` — Running and interactive debugging +- `mcp-csharp-publish` — NuGet, Docker, Azure deployment + +## Reference Files + +- [references/test-patterns.md](references/test-patterns.md) — Complete test code examples: `ClientServerTestBase` in-memory pattern, `WebApplicationFactory` for HTTP, `MockHttpMessageHandler` helper, test categorization, coverage reporting. **Load when:** writing integration tests or need detailed mock patterns. + +## More Info + +- [xUnit documentation](https://xunit.net/docs/getting-started/netcore/cmdline) — Getting started with xUnit for .NET +- [FluentAssertions](https://fluentassertions.com/) — Readable assertion library for .NET diff --git a/plugins/dotnet-ai/skills/mcp-csharp-test/references/test-patterns.md b/plugins/dotnet-ai/skills/mcp-csharp-test/references/test-patterns.md new file mode 100644 index 0000000000..2f3816ccf4 --- /dev/null +++ b/plugins/dotnet-ai/skills/mcp-csharp-test/references/test-patterns.md @@ -0,0 +1,175 @@ +# Test Patterns + +Complete code patterns for testing C# MCP servers at every level. + +## MockHttpMessageHandler Helper + +Reusable mock for tools that use `HttpClient`: + +```csharp +public class MockHttpMessageHandler : HttpMessageHandler +{ + private readonly string _response; + private readonly HttpStatusCode _statusCode; + + public MockHttpMessageHandler( + string response = "", + HttpStatusCode statusCode = HttpStatusCode.OK) + { + _response = response; + _statusCode = statusCode; + } + + protected override Task SendAsync( + HttpRequestMessage request, + CancellationToken cancellationToken) => + Task.FromResult(new HttpResponseMessage + { + StatusCode = _statusCode, + Content = new StringContent(_response) + }); +} +``` + +## ClientServerTestBase (In-Memory Testing) + +The SDK provides `ClientServerTestBase` for zero-network integration tests using `System.IO.Pipelines`: + +```csharp +using ModelContextProtocol.Tests; // from SDK test utilities + +public class MyToolTests : ClientServerTestBase +{ + public MyToolTests(ITestOutputHelper output) : base(output) { } + + protected override void ConfigureServices( + ServiceCollection services, IMcpServerBuilder builder) + { + builder.WithTools(); + // Register any DI services your tools need + services.AddSingleton(); + } + + [Fact] + public async Task MyTool_ReturnsExpected() + { + await using var client = await CreateMcpClientForServer(); + var result = await client.CallToolAsync("my_tool", + new() { ["input"] = "test" }, + cancellationToken: TestContext.Current.CancellationToken); + Assert.NotNull(result); + } +} +``` + +**Key advantages:** +- In-memory transport — no process spawning, no network +- Full DI support — inject fakes/mocks for external dependencies +- Runs in milliseconds + +## HTTP Testing with WebApplicationFactory + +Test HTTP MCP servers using ASP.NET Core's test infrastructure: + +```csharp +using Microsoft.AspNetCore.Mvc.Testing; + +public class HttpServerTests : IClassFixture> +{ + private readonly WebApplicationFactory _factory; + + public HttpServerTests(WebApplicationFactory factory) + { + _factory = factory; + } + + [Fact] + public async Task McpEndpoint_AcceptsInitialize() + { + var client = _factory.CreateClient(); + var request = new + { + jsonrpc = "2.0", + id = 1, + method = "initialize", + @params = new + { + protocolVersion = "2024-11-05", + capabilities = new { }, + clientInfo = new { name = "test", version = "1.0" } + } + }; + + var response = await client.PostAsJsonAsync("/mcp", request); + response.EnsureSuccessStatusCode(); + } + + [Fact] + public async Task HealthEndpoint_ReturnsOk() + { + var client = _factory.CreateClient(); + var response = await client.GetAsync("/health"); + response.EnsureSuccessStatusCode(); + } +} +``` + +**Note:** Requires `` or a public `Program` class. + +## Input Validation Tests + +```csharp +public class ValidationTests +{ + [Theory] + [InlineData(0)] + [InlineData(-1)] + [InlineData(101)] + public void Search_ClampsInvalidLimit(int invalidLimit) + { + var result = SearchTools.Search("query", limit: invalidLimit); + result.Should().NotBeNull(); + } + + [Fact] + public void Search_HandlesSpecialCharacters() + { + var result = SearchTools.Search("'; DROP TABLE users; --"); + result.Should().NotContain("DROP TABLE"); + } +} +``` + +## Test Categories + +Organize tests with traits for selective execution: + +```csharp +[Trait("Category", "Unit")] +public class UnitTests { ... } + +[Trait("Category", "Integration")] +public class IntegrationTests { ... } +``` + +Run by category: +```bash +dotnet test --filter "Category=Unit" +dotnet test --filter "Category=Integration" +``` + +## Coverage Reporting + +```bash +# Add coverage collector +dotnet add package coverlet.collector + +# Run with coverage +dotnet test --collect:"XPlat Code Coverage" + +# Generate HTML report +dotnet tool install --global dotnet-reportgenerator-globaltool +reportgenerator \ + -reports:TestResults/**/coverage.cobertura.xml \ + -targetdir:coveragereport +``` diff --git a/tests/dotnet-ai/mcp-csharp-create/eval.yaml b/tests/dotnet-ai/mcp-csharp-create/eval.yaml new file mode 100644 index 0000000000..1a563f413b --- /dev/null +++ b/tests/dotnet-ai/mcp-csharp-create/eval.yaml @@ -0,0 +1,56 @@ +scenarios: + - name: "Implement MCP tools with proper attributes and DI" + prompt: | + I have a new C# MCP server project. I need to implement a tool class that + wraps a REST API using HttpClient. The tools should follow MCP SDK conventions + with proper attributes for LLM discovery. Show me the tool class, Program.cs + with stdio transport, and explain how DI works for MCP tools. + assertions: + - type: "output_matches" + pattern: "\\[McpServerTool[\\],\\(]" + - type: "output_matches" + pattern: "(AddHttpClient|HttpClient)" + rubric: + - "Shows a tool class with [McpServerTool] attributes" + - "Injects HttpClient via DI (constructor injection or method parameter injection)" + - "Includes [Description] attributes on both the method and all parameters" + - "Configures Program.cs with AddMcpServer, WithStdioServerTransport, and logging to stderr" + timeout: 360 + - name: "Create an HTTP MCP server with tools and resources" + prompt: | + I need to create a C# MCP server that uses HTTP transport for deployment + as a web service. It should expose both tools and resources (for example, + a resource that returns configuration data). Show me how to set up the + HTTP transport with MapMcp and implement a resource. + assertions: + - type: "output_contains" + value: "MapMcp" + - type: "output_matches" + pattern: "(WithHttpTransport|ModelContextProtocol\\.AspNetCore)" + - type: "output_matches" + pattern: "(McpServerResource|McpServerResourceType)" + rubric: + - "Configures HTTP transport with WithHttpTransport() and references ModelContextProtocol.AspNetCore" + - "Includes MapMcp() in the endpoint configuration" + - "Shows a resource class with [McpServerResourceType] and [McpServerResource] attributes including UriTemplate" + - "Shows how to register tools and resources with WithTools or similar DI registration" + timeout: 360 + - name: "Create an MCP server with tools, prompts, and proper logging" + prompt: | + I'm building a C# MCP server using stdio transport. I need to add a tool + that calls an external API using HttpClient (injected via DI), and a prompt + template for summarization. Make sure logging doesn't interfere with the + stdio JSON-RPC protocol. + assertions: + - type: "output_matches" + pattern: "\\[McpServerTool[\\]T,\\(]" + - type: "output_matches" + pattern: "(McpServerPrompt|McpServerPromptType)" + - type: "output_matches" + pattern: "(LogToStandardErrorThreshold|stderr)" + rubric: + - "Shows a tool class with [McpServerTool] and HttpClient injected via DI" + - "Shows a prompt class with [McpServerPromptType] and [McpServerPrompt] returning ChatMessage" + - "Configures logging to stderr (LogToStandardErrorThreshold) to avoid corrupting stdio transport" + - "Adds [Description] attributes on tools, parameters, and prompts for LLM discoverability" + timeout: 360 diff --git a/tests/dotnet-ai/mcp-csharp-debug/eval.yaml b/tests/dotnet-ai/mcp-csharp-debug/eval.yaml new file mode 100644 index 0000000000..cf4b1fdaa6 --- /dev/null +++ b/tests/dotnet-ai/mcp-csharp-debug/eval.yaml @@ -0,0 +1,49 @@ +scenarios: + - name: "Debug an MCP server with MCP Inspector" + prompt: | + I have a C# MCP server that uses stdio transport. I want to test it + interactively with the MCP Inspector to see if my tools are working. + How do I connect and debug? + assertions: + - type: "output_contains" + value: "npx @modelcontextprotocol/inspector" + - type: "output_matches" + pattern: "(dotnet run|--project)" + rubric: + - "Shows how to launch MCP Inspector with npx @modelcontextprotocol/inspector" + - "Explains how to connect it to the stdio server using dotnet run — either as a single CLI command (npx @modelcontextprotocol/inspector dotnet run ...) or via the Inspector UI configuration (Transport Type, Command, Arguments)" + - "Mentions that the Inspector UI shows tools, prompts, and resources" + - "Notes that server logging must go to stderr, not stdout" + timeout: 120 + - name: "Configure VS Code to use an MCP server" + prompt: | + I built a C# MCP server at src/MyMcpServer/MyMcpServer.csproj. + I want to use it with GitHub Copilot in VS Code agent mode. + How do I configure mcp.json? + assertions: + - type: "output_contains" + value: "mcp.json" + - type: "output_matches" + pattern: "(dotnet|run|--project)" + rubric: + - "Shows the mcp.json configuration with type stdio" + - "Uses dotnet run --project with the correct project path" + - "Places the config in .vscode/mcp.json for workspace or user settings" + - "Mentions testing the configuration by asking Copilot to use the tools" + timeout: 120 + - name: "Debug a failing MCP server tool" + prompt: | + My C# MCP server is returning errors when I call one of its tools from + VS Code Copilot. The tool works fine when I run it manually as a console + app. How do I debug this? + assertions: + - type: "output_matches" + pattern: "(stderr|Console\\.Error|logging)" + - type: "output_matches" + pattern: "(breakpoint|attach|debugger)" + rubric: + - "Explains that stdout is reserved for MCP protocol in stdio mode" + - "Recommends checking that all logging goes to stderr" + - "Shows how to further diagnose the issue — such as attaching a debugger to the running process, using Debugger.Launch(), or configuring launch.json to debug the server" + - "Suggests using MCP Inspector or VS Code debug config to test or step through the server" + timeout: 120 diff --git a/tests/dotnet-ai/mcp-csharp-publish/eval.yaml b/tests/dotnet-ai/mcp-csharp-publish/eval.yaml new file mode 100644 index 0000000000..c783fcfda5 --- /dev/null +++ b/tests/dotnet-ai/mcp-csharp-publish/eval.yaml @@ -0,0 +1,55 @@ +scenarios: + - name: "Publish an MCP server as a NuGet tool package" + prompt: | + I have a C# MCP server using stdio transport at src/MyMcpServer/. + I want to publish it to NuGet.org so users can install it as a + dotnet tool. What do I need to configure? + assertions: + - type: "output_contains" + value: "PackAsTool" + - type: "output_contains" + value: "dotnet pack" + - type: "output_matches" + pattern: "(dotnet nuget push|nuget\\.org)" + rubric: + - "Configures PackAsTool and ToolCommandName in the csproj" + - "Shows how to build, pack, and push to NuGet.org" + - "Includes local testing with dotnet tool install --global" + - "Mentions the dnx tool runner for MCP client configuration" + timeout: 120 + - name: "Deploy an HTTP MCP server to Azure Container Apps" + prompt: | + I have an HTTP-based C# MCP server that I want to deploy to Azure. + It uses environment variables for API keys. What's the best way + to containerize and deploy it? + assertions: + - type: "output_matches" + pattern: "(Dockerfile|docker)" + - type: "output_matches" + pattern: "(Container Apps|containerapp)" + - type: "output_matches" + pattern: "(secret|Secret|KEY)" + rubric: + - "Provides a multi-stage Dockerfile with a non-root user" + - "Shows Azure Container Apps deployment commands" + - "Configures secrets using Container Apps secrets, not environment variables in plain text" + - "Includes a health check endpoint" + timeout: 120 + - name: "Publish to the MCP Registry" + prompt: | + I already published my C# MCP server to NuGet.org. Now I want to + register it in the official MCP Registry so it's discoverable. + How do I do this? + assertions: + - type: "output_contains" + value: "server.json" + - type: "output_contains" + value: "mcp-publisher" + - type: "output_matches" + pattern: "io\\.github\\." + rubric: + - "Shows the server.json schema with the correct format" + - "Uses the mcp-publisher CLI for publishing" + - "Explains the naming convention (io.github./)" + - "Emphasizes version consistency between csproj and server.json" + timeout: 120 diff --git a/tests/dotnet-ai/mcp-csharp-test/eval.yaml b/tests/dotnet-ai/mcp-csharp-test/eval.yaml new file mode 100644 index 0000000000..f2d04372e7 --- /dev/null +++ b/tests/dotnet-ai/mcp-csharp-test/eval.yaml @@ -0,0 +1,51 @@ +scenarios: + - name: "Write unit and integration tests for an MCP server" + prompt: | + I have a C# MCP server with tools that call an external REST API. + I want to write tests that verify the tools work correctly without + calling the real API. How do I structure the test project? + assertions: + - type: "output_contains" + value: "xunit" + - type: "output_matches" + pattern: "(Mock|Fake|mock|fake|HttpMessageHandler)" + - type: "output_matches" + pattern: "(CallToolAsync|McpClient)" + rubric: + - "Creates a test project with xUnit and references the server project" + - "Shows how to mock HttpClient or external dependencies" + - "Demonstrates an integration test using McpClient to call tools in-memory" + - "Separates unit tests from integration tests using traits or folders" + timeout: 120 + - name: "Test an HTTP MCP server with WebApplicationFactory" + prompt: | + I have an HTTP-based C# MCP server built with ASP.NET Core. + How do I write integration tests that test the full MCP protocol + over HTTP without spinning up a real server? + assertions: + - type: "output_contains" + value: "WebApplicationFactory" + - type: "output_matches" + pattern: "(initialize|MCP|jsonrpc)" + rubric: + - "Uses WebApplicationFactory for in-process HTTP testing" + - "Shows how to send an MCP initialize request to verify the server responds" + - "Mentions InternalsVisibleTo or making Program public for test access" + - "Tests tool invocation through the HTTP endpoint" + timeout: 120 + - name: "Create evaluations for an MCP server" + prompt: | + I have a C# MCP server that provides tools for querying a product catalog. + I want to create evaluations to measure how well an LLM uses the tools. + What format should I use and what makes a good evaluation question? + assertions: + - type: "output_matches" + pattern: "(evaluation|qa_pair|question.*answer)" + - type: "output_matches" + pattern: "(read.only|non.destructive|deterministic|verifiable)" + rubric: + - "Shows the XML evaluation format with qa_pair elements" + - "Explains that questions should require multiple tool calls and reasoning" + - "Emphasizes that questions must be read-only with deterministic answers" + - "Provides example evaluation questions appropriate for a product catalog" + timeout: 120