From f73d4caa18765057455ccebe00db404cfd1a5efa Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Tue, 10 Mar 2026 10:55:13 -0700 Subject: [PATCH 01/29] Add C# MCP server skills (create, debug, test, publish) Four new skills for the C# MCP server development lifecycle: - mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config - mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging - mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals - mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests. --- .../dotnet/skills/mcp-csharp-create/SKILL.md | 265 +++++++++++++++++ .../references/api-patterns.md | 148 ++++++++++ .../references/transport-config.md | 141 +++++++++ .../dotnet/skills/mcp-csharp-debug/SKILL.md | 234 +++++++++++++++ .../mcp-csharp-debug/references/ide-config.md | 153 ++++++++++ .../references/mcp-inspector.md | 101 +++++++ .../dotnet/skills/mcp-csharp-publish/SKILL.md | 275 ++++++++++++++++++ .../references/docker-azure.md | 187 ++++++++++++ .../references/mcp-registry.md | 140 +++++++++ .../references/nuget-packaging.md | 144 +++++++++ .../dotnet/skills/mcp-csharp-test/SKILL.md | 206 +++++++++++++ .../references/evaluation-guide.md | 90 ++++++ .../references/test-patterns.md | 175 +++++++++++ tests/dotnet/mcp-csharp-create/eval.yaml | 56 ++++ tests/dotnet/mcp-csharp-debug/eval.yaml | 49 ++++ tests/dotnet/mcp-csharp-publish/eval.yaml | 55 ++++ tests/dotnet/mcp-csharp-test/eval.yaml | 51 ++++ 17 files changed, 2470 insertions(+) create mode 100644 plugins/dotnet/skills/mcp-csharp-create/SKILL.md create mode 100644 plugins/dotnet/skills/mcp-csharp-create/references/api-patterns.md create mode 100644 plugins/dotnet/skills/mcp-csharp-create/references/transport-config.md create mode 100644 plugins/dotnet/skills/mcp-csharp-debug/SKILL.md create mode 100644 plugins/dotnet/skills/mcp-csharp-debug/references/ide-config.md create mode 100644 plugins/dotnet/skills/mcp-csharp-debug/references/mcp-inspector.md create mode 100644 plugins/dotnet/skills/mcp-csharp-publish/SKILL.md create mode 100644 plugins/dotnet/skills/mcp-csharp-publish/references/docker-azure.md create mode 100644 plugins/dotnet/skills/mcp-csharp-publish/references/mcp-registry.md create mode 100644 plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md create mode 100644 plugins/dotnet/skills/mcp-csharp-test/SKILL.md create mode 100644 plugins/dotnet/skills/mcp-csharp-test/references/evaluation-guide.md create mode 100644 plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md create mode 100644 tests/dotnet/mcp-csharp-create/eval.yaml create mode 100644 tests/dotnet/mcp-csharp-debug/eval.yaml create mode 100644 tests/dotnet/mcp-csharp-publish/eval.yaml create mode 100644 tests/dotnet/mcp-csharp-test/eval.yaml diff --git a/plugins/dotnet/skills/mcp-csharp-create/SKILL.md b/plugins/dotnet/skills/mcp-csharp-create/SKILL.md new file mode 100644 index 0000000000..295ff34ad3 --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-create/SKILL.md @@ -0,0 +1,265 @@ +--- +name: mcp-csharp-create +description: > + Create MCP servers using the C# SDK and .NET project templates. Covers scaffolding, + tool/prompt/resource implementation, and transport configuration for stdio and HTTP. + USE FOR: creating new MCP server projects, scaffolding with dotnet new mcpserver, adding + MCP tools/prompts/resources, choosing stdio vs HTTP transport, configuring MCP hosting in + Program.cs, setting up ASP.NET Core MCP endpoints with MapMcp. + DO NOT USE FOR: debugging or running existing servers (use mcp-csharp-debug), writing tests + (use mcp-csharp-test), publishing or deploying (use mcp-csharp-publish), building MCP + clients, non-.NET MCP servers. +--- + +# C# MCP Server Creation + +Create Model Context Protocol servers using the official C# SDK (`ModelContextProtocol` NuGet package) and the `dotnet new mcpserver` project template. Servers expose tools, prompts, and resources that LLMs can discover and invoke via the MCP protocol. + +## When to Use + +- Starting a new MCP server project from scratch +- Adding tools, prompts, or resources to an existing MCP server +- Choosing between stdio (`--transport local`) and HTTP (`--transport remote`) transport +- Setting up ASP.NET Core hosting for an HTTP MCP server +- Wrapping an external API or service as MCP tools + +## Stop Signals + +- **Server already exists and needs debugging?** → Use [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) +- **Need tests or evaluations?** → Use [mcp-csharp-test](../mcp-csharp-test/SKILL.md) +- **Ready to publish?** → Use [mcp-csharp-publish](../mcp-csharp-publish/SKILL.md) +- **Building an MCP client, not a server** → This skill is server-side only + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Transport type | Yes | `stdio` (local/CLI) or `http` (remote/web). Ask user if not specified — default to stdio | +| Project name | Yes | PascalCase name for the project (e.g., `WeatherMcpServer`) | +| .NET SDK version | Recommended | .NET 10.0+ required. Check with `dotnet --version` | +| Service/API to wrap | Recommended | External API or service the tools will interact with | + +## Workflow + +> **Commit strategy:** Commit after completing each step so scaffolding and implementation are separately reviewable. + +### Step 1: Verify prerequisites + +1. Confirm .NET 10+ SDK: `dotnet --version` (install from https://dotnet.microsoft.com if < 10.0) + +2. Check if the MCP server template is already installed: + ```bash + dotnet new list mcpserver + ``` + If "No templates found" → install: `dotnet new install Microsoft.McpServer.ProjectTemplates` + +### Step 2: Choose transport + +| Choose **stdio** if… | Choose **HTTP** if… | +|----------------------|---------------------| +| Local CLI tool or IDE plugin | Cloud/web service deployment | +| Single user at a time | Multiple simultaneous clients | +| Running as subprocess (VS Code, GitHub Copilot) | Cross-network access needed | +| Simpler setup, no network config | Containerized deployment (Docker/Azure) | + +**Default:** stdio — simpler, works for most local development. Users can add HTTP later. + +### Step 3: Scaffold the project + +**stdio server:** +```bash +dotnet new mcpserver -n +``` +If the template times out or is unavailable, use `dotnet new console -n ` and add `dotnet add package ModelContextProtocol`. + +**HTTP server:** +```bash +dotnet new web -n +cd +dotnet add package ModelContextProtocol.AspNetCore +``` +This is the recommended approach — faster and more reliable than the template. The template also supports HTTP via `dotnet new mcpserver -n --transport remote`, but `dotnet new web` gives you more control over the project structure. + +**Template flags reference:** `--transport local` (stdio, default), `--transport remote` (ASP.NET Core HTTP), `--aot`, `--self-contained`. + +### Step 4: Implement tools + +Tools are the primary way MCP servers expose functionality. Add a class with `[McpServerToolType]` and methods with `[McpServerTool]`: + +```csharp +using ModelContextProtocol.Server; +using System.ComponentModel; + +[McpServerToolType] +public static class MyTools +{ + [McpServerTool, Description("Brief description of what the tool does.")] + public static async Task DoSomething( + [Description("What this parameter controls")] string input, + CancellationToken cancellationToken = default) + { + // Implementation + return $"Result: {input}"; + } +} +``` + +**Critical rules:** +- Every tool method **must** have a `[Description]` attribute — LLMs use this to decide when to call the tool +- Every parameter **must** have a `[Description]` attribute +- Accept `CancellationToken` in all async tools +- Use `[McpServerTool(Name = "custom_name")]` only if the default method name is unclear + +**DI injection patterns** — the SDK supports two styles: + +1. **Method parameter injection (static class):** DI services appear as method parameters. The SDK resolves them automatically — they do not appear in the tool schema. + +2. **Constructor injection (non-static class):** Use when tools need shared state or multiple services: +```csharp +[McpServerToolType] +public class ApiTools(HttpClient httpClient, ILogger logger) +{ + [McpServerTool, Description("Fetch a resource by ID.")] + public async Task FetchResource( + [Description("Resource identifier")] string id, + CancellationToken cancellationToken = default) + { + logger.LogInformation("Fetching {Id}", id); + return await httpClient.GetStringAsync($"/api/{id}", cancellationToken); + } +} +``` +Register services in Program.cs: +```csharp +var builder = Host.CreateApplicationBuilder(args); +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); + +builder.Services.AddHttpClient(); // registers IHttpClientFactory + HttpClient +// ILogger is registered by default — no extra setup needed. + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithToolsFromAssembly(); // discovers non-static [McpServerToolType] classes + +await builder.Build().RunAsync(); +``` + +**For the full attribute reference, return types, DI injection, and builder API patterns**, see [references/api-patterns.md](references/api-patterns.md). + +### Step 5: Add prompts and resources (optional) + +**Prompts** — reusable LLM interaction templates: +```csharp +[McpServerPromptType] +public static class MyPrompts +{ + [McpServerPrompt, Description("Summarize content into one sentence.")] + public static ChatMessage Summarize( + [Description("Content to summarize")] string content) => + new(ChatRole.User, $"Summarize this into one sentence: {content}"); +} +``` + +**Resources** — data the LLM can read: +```csharp +[McpServerResourceType] +public static class MyResources +{ + [McpServerResource(UriTemplate = "config://app", Name = "App Config", + MimeType = "application/json"), Description("Application configuration")] + public static string GetConfig() => JsonSerializer.Serialize(AppConfig.Current); +} +``` + +### Step 6: Configure Program.cs + +**stdio transport:** +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using ModelContextProtocol.Server; + +var builder = Host.CreateApplicationBuilder(args); +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); // CRITICAL: stderr only + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithToolsFromAssembly(); + +await builder.Build().RunAsync(); +``` + +**HTTP transport:** +```csharp +using ModelContextProtocol.Server; + +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddMcpServer() + .WithHttpTransport() + .WithToolsFromAssembly(); + +// Register services your tools need via DI +// builder.Services.AddHttpClient(); +// builder.Services.AddSingleton(); + +var app = builder.Build(); +app.MapMcp(); // exposes MCP endpoint at /mcp (Streamable HTTP) +app.MapGet("/health", () => "ok"); // health check for container orchestrators +app.Run(); +``` + +**Key HTTP details:** `MapMcp()` defaults to `/mcp` path. For containers, set `ASPNETCORE_URLS=http://+:8080` and `EXPOSE 8080`. The MCP HTTP protocol uses Streamable HTTP — no special client config needed beyond the URL. + +**For transport configuration details** (stateless mode, auth, path prefix, `HttpContextAccessor`), see [references/transport-config.md](references/transport-config.md). + +### Step 7: Verify the server starts + +```bash +cd +dotnet build +dotnet run +``` + +For stdio: the process starts and waits for JSON-RPC input on stdin. +For HTTP: the server listens on the configured port. + +## Validation + +- [ ] Project builds with no errors (`dotnet build`) +- [ ] All tool classes have `[McpServerToolType]` attribute +- [ ] All tool methods have `[McpServerTool]` and `[Description]` attributes +- [ ] All parameters have `[Description]` attributes +- [ ] stdio: logging directed to stderr, not stdout +- [ ] HTTP: `app.MapMcp()` is called in Program.cs +- [ ] Server starts successfully with `dotnet run` + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| stdio server outputs garbage or hangs | Logging to stdout corrupts JSON-RPC protocol. Set `LogToStandardErrorThreshold = LogLevel.Trace` | +| Tool not discovered by LLM clients | Missing `[McpServerToolType]` on the class or `[McpServerTool]` on the method. Verify `.WithToolsFromAssembly()` in Program.cs | +| LLM doesn't understand when to use a tool | Add clear `[Description]` attributes on both the method and all parameters | +| `WithToolsFromAssembly()` fails in AOT | Reflection-based discovery is incompatible with Native AOT. Use `.WithTools()` instead | +| Parameters not appearing in tool schema | `CancellationToken`, `IMcpServer`, and DI services are injected automatically — they do not appear in the schema. Only parameters with `[Description]` are exposed | +| HTTP server returns 404 | `app.MapMcp()` must be called. Check the request path matches the configured route | + +## Related Skills + +- [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) — Run, debug, and test with MCP Inspector +- [mcp-csharp-test](../mcp-csharp-test/SKILL.md) — Unit tests, integration tests, evaluations +- [mcp-csharp-publish](../mcp-csharp-publish/SKILL.md) — NuGet, Docker, Azure deployment + +## Reference Files + +- [references/api-patterns.md](references/api-patterns.md) — Complete attribute reference, return types, DI injection, builder API, dynamic tools, experimental APIs. **Load when:** implementing tools, prompts, or resources beyond the basic patterns shown above. +- [references/transport-config.md](references/transport-config.md) — Detailed transport configuration: stateless HTTP mode, OAuth/auth, custom path prefix, `HttpContextAccessor`, OpenTelemetry observability. **Load when:** configuring advanced transport options or authentication. + +## More Info + +- [C# MCP SDK](https://github.com/modelcontextprotocol/csharp-sdk) — Official SDK repository +- [Build an MCP server (.NET)](https://learn.microsoft.com/dotnet/ai/quickstarts/build-mcp-server) — Microsoft quickstart +- [MCP Specification](https://modelcontextprotocol.io/specification/) — Protocol specification diff --git a/plugins/dotnet/skills/mcp-csharp-create/references/api-patterns.md b/plugins/dotnet/skills/mcp-csharp-create/references/api-patterns.md new file mode 100644 index 0000000000..f01e357c71 --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-create/references/api-patterns.md @@ -0,0 +1,148 @@ +# C# MCP SDK API Patterns + +Complete reference for MCP server implementation patterns using the C# SDK. + +## Attribute Reference + +### Tool Attributes + +| Attribute | Target | Key Properties | +|-----------|--------|----------------| +| `[McpServerToolType]` | Class | Marks class as containing tool methods | +| `[McpServerTool]` | Method | `Name`, `Title`, `Destructive`, `Idempotent`, `OpenWorld`, `ReadOnly` | +| `[Description("...")]` | Method/Parameter | From `System.ComponentModel` — provides LLM-visible descriptions | +| `[McpMeta("key", value)]` | Any | Adds `_meta` entries to the MCP protocol response | + +### Prompt Attributes + +| Attribute | Target | Key Properties | +|-----------|--------|----------------| +| `[McpServerPromptType]` | Class | Marks class as containing prompt methods | +| `[McpServerPrompt]` | Method | `Name`, `Title` | + +### Resource Attributes + +| Attribute | Target | Key Properties | +|-----------|--------|----------------| +| `[McpServerResourceType]` | Class | Marks class as containing resource methods | +| `[McpServerResource]` | Method | `UriTemplate`, `Name`, `Title`, `MimeType` | + +## Tool Return Types + +Tools can return any of these types (or their `Task`/`ValueTask` async variants): + +| Return Type | Behavior | +|-------------|----------| +| `string` | Wrapped as `TextContentBlock` | +| `TextContentBlock` | Text content with optional annotations | +| `ImageContentBlock` | Base64-encoded image data | +| `AudioContentBlock` | Base64-encoded audio data | +| `EmbeddedResourceBlock` | Resource reference | +| `CallToolResult` | Full control over content blocks and `isError` flag | +| `IEnumerable` | Multiple content blocks | + +## Injected Parameters + +These types are automatically injected by the framework and **do not appear** in the tool's JSON schema: + +| Type | Purpose | +|------|---------| +| `CancellationToken` | Cooperative cancellation | +| `IMcpServer` / `McpServer` | Access to server instance for notifications, logging | +| `RequestContext` | Full request context, progress tokens | +| `IProgress` | Report progress back to the client | +| Any DI-registered service | Constructor or method parameter injection | + +Example with DI and progress: +```csharp +[McpServerToolType] +public class MyTools(IHttpClientFactory httpFactory) +{ + [McpServerTool, Description("Fetches data from API")] + public async Task FetchData( + [Description("Resource identifier")] string resourceId, + IProgress progress, + CancellationToken cancellationToken) + { + progress.Report(new() { Progress = 0, Total = 100 }); + var client = httpFactory.CreateClient(); + var result = await client.GetStringAsync($"/api/{resourceId}", cancellationToken); + progress.Report(new() { Progress = 100, Total = 100 }); + return result; + } +} +``` + +## Builder API + +The fluent builder API configures the MCP server via dependency injection: + +```csharp +services.AddMcpServer() + // Transports (choose one) + .WithStdioServerTransport() // stdio: Generic Host + .WithHttpTransport() // HTTP: ASP.NET Core + + // Register primitives (attribute-based) + .WithTools() // Specific class + .WithToolsFromAssembly() // All [McpServerToolType] in entry assembly + .WithPrompts() + .WithPromptsFromAssembly() + .WithResources() + .WithResourcesFromAssembly() + + // Register primitives (handler-based) + .WithListToolsHandler(async (ctx, ct) => { ... }) + .WithCallToolHandler(async (ctx, ct) => { ... }) + + // Middleware + .WithRequestFilters(filters => { ... }); +``` + +> **AOT warning:** `.WithToolsFromAssembly()` uses reflection and is not compatible with Native AOT. Use `.WithTools()` for AOT scenarios. + +## Dynamic Tool Creation + +Create tools at runtime without attribute-decorated classes: + +```csharp +var tool = McpServerTool.Create( + (int count, string prefix) => Enumerable.Range(1, count).Select(i => $"{prefix}-{i}"), + new McpServerToolCreateOptions { Name = "generate_ids", Description = "Generate sequential IDs" }); +``` + +## McpServerOptions + +Configure server behavior via `McpServerOptions`: + +```csharp +services.AddMcpServer(options => +{ + options.ServerInfo = new() { Name = "MyServer", Version = "1.0.0" }; + options.ServerInstructions = "You are connected to MyService. Use tools to query data."; + options.Capabilities = new() + { + Tools = new() { ListChanged = true }, + Resources = new() { Subscribe = true, ListChanged = true } + }; +}); +``` + +Key properties: `ServerInfo`, `Capabilities`, `ServerInstructions`, `InitializationTimeout`, `ToolCollection`, `ResourceCollection`, `PromptCollection`. + +## Experimental APIs + +| Diagnostic ID | Feature | Suppression | +|---------------|---------|-------------| +| `MCPEXP001` | Tasks feature | `#pragma warning disable MCPEXP001` | +| `MCPEXP002` | Subclassing `McpServer`/`McpClient` | `#pragma warning disable MCPEXP002` | + +Suppress project-wide: `MCPEXP001;MCPEXP002` in `.csproj`. + +## NuGet Packages + +| Package | When to Use | +|---------|-------------| +| `ModelContextProtocol` | **Default.** Hosting, DI, attribute-based discovery | +| `ModelContextProtocol.AspNetCore` | HTTP servers with ASP.NET Core (`MapMcp()`) | +| `ModelContextProtocol.Core` | Minimum dependencies — low-level client/server APIs only | diff --git a/plugins/dotnet/skills/mcp-csharp-create/references/transport-config.md b/plugins/dotnet/skills/mcp-csharp-create/references/transport-config.md new file mode 100644 index 0000000000..70181831f9 --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-create/references/transport-config.md @@ -0,0 +1,141 @@ +# Transport Configuration + +Detailed configuration for stdio and HTTP transports in C# MCP servers. + +## Stdio Transport (Generic Host) + +Uses `Microsoft.Extensions.Hosting` for the application lifecycle: + +```csharp +var builder = Host.CreateApplicationBuilder(args); + +// CRITICAL: All logging must go to stderr — stdout is reserved for JSON-RPC +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithToolsFromAssembly(); + +await builder.Build().RunAsync(); +``` + +### Stdio Key Points + +- `stdout` carries JSON-RPC messages — **never** write anything else to stdout +- All logging, diagnostics, and debug output must use stderr +- The process runs as a subprocess of the MCP client +- No network configuration needed + +## HTTP Transport (ASP.NET Core) + +Uses ASP.NET Core with Streamable HTTP (default) or SSE (legacy): + +```csharp +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddMcpServer() + .WithHttpTransport() + .WithToolsFromAssembly(); + +var app = builder.Build(); +app.MapMcp(); +app.Run(); +``` + +### Custom Path Prefix + +```csharp +app.MapMcp("/custom-mcp-path"); +``` + +### Stateless Mode + +Disables session state — each request is independent: + +```csharp +builder.Services.AddMcpServer() + .WithHttpTransport(options => options.Stateless = true); +``` + +### Idle Timeout + +Configure session cleanup: + +```csharp +builder.Services.AddMcpServer() + .WithHttpTransport(options => options.IdleTimeout = TimeSpan.FromMinutes(30)); +``` + +### Port Configuration + +```csharp +app.Run("http://localhost:3001"); +// or via launchSettings.json / ASPNETCORE_URLS environment variable +``` + +## Authentication and Authorization + +### JWT Bearer Auth + +```csharp +builder.Services.AddAuthentication() + .AddJwtBearer(options => + { + options.Authority = "https://your-auth-server"; + options.Audience = "mcp-server"; + }); + +builder.Services.AddAuthorization(); + +var app = builder.Build(); +app.UseAuthentication(); +app.UseAuthorization(); +app.MapMcp().RequireAuthorization(); +``` + +### Accessing HttpContext in Tools + +Register `HttpContextAccessor` to access HTTP request details from tool methods: + +```csharp +builder.Services.AddHttpContextAccessor(); + +[McpServerToolType] +public class AuthAwareTools(IHttpContextAccessor httpContextAccessor) +{ + [McpServerTool, Description("Returns the authenticated user's ID")] + public string GetCurrentUser() + { + var user = httpContextAccessor.HttpContext?.User; + return user?.Identity?.Name ?? "anonymous"; + } +} +``` + +### OAuth 2.0 with Dynamic Client Registration + +The SDK includes a full OAuth sample in `samples/ProtectedMcpServer/` covering: +- JWT bearer token validation +- OAuth 2.0 authorization flows +- Dynamic Client Registration (RFC 7591) + +## OpenTelemetry Observability + +Built-in distributed tracing and metrics: + +| Component | Name | +|-----------|------| +| `ActivitySource` | `Experimental.ModelContextProtocol` | +| `Meter` | `Experimental.ModelContextProtocol` | + +```csharp +builder.Services.AddOpenTelemetry() + .WithTracing(tracing => tracing + .AddSource("Experimental.ModelContextProtocol") + .AddAspNetCoreInstrumentation()) + .WithMetrics(metrics => metrics + .AddMeter("Experimental.ModelContextProtocol")); +``` + +Trace context propagated via `_meta.traceparent` across client/server boundaries. +Metrics follow [MCP semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/mcp.md#metrics). diff --git a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md new file mode 100644 index 0000000000..6037d51a07 --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md @@ -0,0 +1,234 @@ +--- +name: mcp-csharp-debug +description: > + Run and debug C# MCP servers locally. Covers IDE configuration, MCP Inspector testing, + GitHub Copilot Agent Mode integration, logging setup, and troubleshooting. + USE FOR: running MCP servers locally with dotnet run, configuring VS Code or Visual Studio + for MCP debugging, testing tools with MCP Inspector, testing with GitHub Copilot Agent Mode, + diagnosing tool registration issues, setting up mcp.json configuration, debugging MCP + protocol messages, configuring logging for stdio and HTTP servers. + DO NOT USE FOR: creating new MCP servers (use mcp-csharp-create), writing automated tests + (use mcp-csharp-test), publishing or deploying to production (use mcp-csharp-publish). +--- + +# C# MCP Server Debugging + +Run, debug, and interactively test C# MCP servers. Covers local execution, IDE debugging with breakpoints, MCP Inspector for protocol-level testing, and GitHub Copilot Agent Mode integration. + +## When to Use + +- Running an MCP server locally for the first time +- Configuring VS Code or Visual Studio to debug an MCP server +- Testing tools interactively with MCP Inspector +- Verifying tools appear in GitHub Copilot Agent Mode +- Diagnosing issues: tools not discovered, protocol errors, server crashes +- Setting up `mcp.json` or `.mcp.json` configuration + +## Stop Signals + +- **No project yet?** → Use [mcp-csharp-create](../mcp-csharp-create/SKILL.md) first +- **Need automated tests?** → Use [mcp-csharp-test](../mcp-csharp-test/SKILL.md) +- **Production deployment issue?** → Use [mcp-csharp-publish](../mcp-csharp-publish/SKILL.md) + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Project path | Yes | Path to the `.csproj` file or project directory | +| Transport type | Recommended | `stdio` or `http` — detect from `.csproj` if not specified | +| IDE | Recommended | VS Code or Visual Studio — detect from environment if not specified | + +**Agent behavior:** Detect transport type by checking the `.csproj` for a `PackageReference` to `ModelContextProtocol.AspNetCore`. If present → HTTP, otherwise → stdio. + +## Workflow + +### Step 1: Run the server locally + +**stdio transport:** +```bash +cd +dotnet run +``` +The process starts and waits for JSON-RPC messages on stdin. No output on stdout means it's working correctly. + +**HTTP transport:** +```bash +cd +dotnet run +# Server listens on http://localhost:3001 (or configured port) +``` + +### Step 2: Generate MCP configuration + +Detect the IDE and transport, then create the appropriate config file. + +**For VS Code** — create `.vscode/mcp.json`: + +stdio: +```json +{ + "servers": { + "": { + "type": "stdio", + "command": "dotnet", + "args": ["run", "--project", ""] + } + } +} +``` + +HTTP: +```json +{ + "servers": { + "": { + "type": "http", + "url": "http://localhost:3001" + } + } +} +``` + +**For Visual Studio** — create `.mcp.json` at solution root (same JSON structure). + +**For detailed IDE-specific configuration** (launch.json, environment variables, secrets), see [references/ide-config.md](references/ide-config.md). + +### Step 3: Test with MCP Inspector + +The MCP Inspector provides a UI for testing tools, viewing schemas, and inspecting protocol messages. + +**stdio server:** +```bash +npx @modelcontextprotocol/inspector dotnet run --project +``` + +**HTTP server:** +1. Start your server: `dotnet run` +2. Run Inspector: `npx @modelcontextprotocol/inspector` +3. Connect to `http://localhost:3001` + +**Inspector capabilities:** +- List all registered tools, prompts, and resources +- Call tools with custom parameters and see results +- View request/response JSON-RPC messages +- Inspect tool schemas and descriptions + +**For detailed Inspector usage and troubleshooting**, see [references/mcp-inspector.md](references/mcp-inspector.md). + +### Step 4: Test with GitHub Copilot Agent Mode + +1. Open GitHub Copilot Chat → switch to **Agent** mode +2. Click **Select Tools** (wrench icon) → verify your server and tools are listed +3. Test with a prompt that should trigger your tool +4. Approve tool execution when prompted + +**If tools don't appear — troubleshoot tool discovery:** + +1. **Rebuild first** — stale builds are the #1 cause: + ```bash + dotnet build + ``` + Then restart the MCP server (click Stop → Start in VS Code, or restart `dotnet run`). + +2. **Check `[McpServerToolType]` on the class:** + ```csharp + [McpServerToolType] // ← Required on the class + public class MyTools { ... } + ``` + +3. **Check `[McpServerTool]` on each method** (must be `public static`): + ```csharp + [McpServerTool, Description("Does something")] + public static string DoSomething(string input) => input; + ``` + +4. **Verify tool registration in Program.cs** — use one of: + ```csharp + .WithTools() // register specific class + .WithToolsFromAssembly() // scan entire assembly for [McpServerToolType] + ``` + +5. **Check `mcp.json`** points to the correct project path + +6. If still not appearing, reference the tool explicitly: `Using #tool_name, do X` + +### Step 5: Set up breakpoint debugging + +1. Set breakpoints in your tool methods +2. Launch with the debugger: + - **VS Code:** F5 (requires `launch.json` — see [references/ide-config.md](references/ide-config.md)) + - **Visual Studio:** F5 or right-click project → Debug → Start +3. Trigger the tool (via Inspector, Copilot, or test client) +4. Execution pauses at breakpoints + +**Critical:** Build in Debug configuration. Breakpoints won't hit in Release builds. + +### Step 6: Configure logging + +**Critical for stdio transport:** Any output to stdout (including `Console.WriteLine`) **corrupts the MCP JSON-RPC protocol** and causes garbled responses or crashes. All logging and diagnostic output must go to stderr. + +**stdio transport** — log to stderr only: +```csharp +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); +``` + +**HTTP transport** — standard console logging: +```csharp +builder.Logging.ClearProviders(); +builder.Logging.AddConsole(); +builder.Logging.SetMinimumLevel( + builder.Environment.IsDevelopment() ? LogLevel.Debug : LogLevel.Information); +``` + +**In tool methods** — inject `ILogger` via constructor: +```csharp +[McpServerToolType] +public class MyTools(ILogger logger) +{ + [McpServerTool, Description("Processes data")] + public string ProcessData(string input) + { + logger.LogDebug("Processing: {Input}", input); + return DoProcessing(input); + } +} +``` + +## Validation + +- [ ] Server starts without errors via `dotnet run` +- [ ] MCP Inspector connects and lists all expected tools +- [ ] Tool calls via Inspector return expected results +- [ ] Breakpoints hit when debugging in IDE +- [ ] Tools appear in GitHub Copilot Agent Mode tool list +- [ ] stdio: no logging output on stdout (stderr only) + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Tools not appearing in Copilot or Inspector | **Rebuild first:** `dotnet build`, then restart the server. If still missing, verify `[McpServerToolType]` on class, `[McpServerTool]` on methods, and `WithTools()` or `WithToolsFromAssembly()` in Program.cs | +| stdio server produces garbled output | `Console.WriteLine()` or logging is writing to stdout. All output **must** go to stderr. Set `LogToStandardErrorThreshold = LogLevel.Trace` on the console logger | +| "Command not found" when starting server | .NET 10+ SDK not installed. Check with `dotnet --version` | +| HTTP server returns 404 at MCP endpoint | Missing `app.MapMcp()` in Program.cs | +| Breakpoints not hit | Building in Release mode. Rebuild in Debug: `dotnet build -c Debug`, then restart | +| Environment variables not passed to server | Add `"env"` section to `mcp.json`. For secrets in VS Code, use `"${input:var_id}"` syntax | +| MCP Inspector can't connect to HTTP server | Server not running, or wrong port. Check `dotnet run` output for the listening URL | +| Stale tools after code changes | Always `dotnet build` and restart the server after changing tool methods or attributes | + +## Related Skills + +- [mcp-csharp-create](../mcp-csharp-create/SKILL.md) — Create a new MCP server project +- [mcp-csharp-test](../mcp-csharp-test/SKILL.md) — Automated tests and evaluations +- [mcp-csharp-publish](../mcp-csharp-publish/SKILL.md) — NuGet, Docker, Azure deployment + +## Reference Files + +- [references/mcp-inspector.md](references/mcp-inspector.md) — Detailed MCP Inspector usage: installation, connecting to servers, feature walkthrough, troubleshooting. **Load when:** user needs detailed Inspector guidance or is having connection issues. +- [references/ide-config.md](references/ide-config.md) — Complete VS Code and Visual Studio configuration: mcp.json templates, launch.json, environment variables, conditional breakpoints. **Load when:** setting up IDE debugging or configuring environment-specific settings. + +## More Info + +- [MCP Inspector](https://www.npmjs.com/package/@modelcontextprotocol/inspector) — Interactive debugging tool for MCP servers +- [VS Code MCP documentation](https://code.visualstudio.com/docs/copilot/chat/mcp-servers) — Configuring MCP servers in VS Code diff --git a/plugins/dotnet/skills/mcp-csharp-debug/references/ide-config.md b/plugins/dotnet/skills/mcp-csharp-debug/references/ide-config.md new file mode 100644 index 0000000000..b807327cf2 --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-debug/references/ide-config.md @@ -0,0 +1,153 @@ +# IDE Configuration + +Complete configuration for debugging C# MCP servers in VS Code and Visual Studio. + +## VS Code Configuration + +### mcp.json (MCP Server Registration) + +Create `.vscode/mcp.json` to register your server with VS Code and GitHub Copilot: + +**stdio transport:** +```json +{ + "servers": { + "MyMcpServer": { + "type": "stdio", + "command": "dotnet", + "args": [ + "run", + "--project", + "MyMcpServer/MyMcpServer.csproj" + ], + "env": { + "API_KEY": "${input:api_key}" + } + } + }, + "inputs": [ + { + "type": "promptString", + "id": "api_key", + "description": "API key for the service", + "password": true + } + ] +} +``` + +**HTTP transport:** +```json +{ + "servers": { + "MyMcpServer": { + "type": "http", + "url": "http://localhost:3001", + "headers": {} + } + } +} +``` + +### launch.json (Debugger Configuration) + +Create `.vscode/launch.json` for F5 debugging: + +```json +{ + "version": "0.2.0", + "configurations": [ + { + "name": "Debug MCP Server", + "type": "coreclr", + "request": "launch", + "program": "${workspaceFolder}/MyMcpServer/bin/Debug/net10.0/MyMcpServer.dll", + "args": [], + "cwd": "${workspaceFolder}/MyMcpServer", + "console": "integratedTerminal", + "stopAtEntry": false, + "env": { + "DOTNET_ENVIRONMENT": "Development", + "API_KEY": "your-dev-api-key" + } + } + ] +} +``` + +### Attach to Running Process + +For debugging a server started by VS Code's MCP integration: + +```json +{ + "name": "Attach to MCP Server", + "type": "coreclr", + "request": "attach", + "processName": "MyMcpServer" +} +``` + +### Environment Variable Patterns + +| Pattern | Usage | +|---------|-------| +| `"API_KEY": "literal-value"` | Direct value (dev only) | +| `"API_KEY": "${input:api_key}"` | Prompt user for value | +| `"API_KEY": "${env:API_KEY}"` | Read from system environment | + +## Visual Studio Configuration + +### 1. Register MCP Server + +1. Open **GitHub Copilot Chat** (top right icon) +2. Click **Select Tools** (wrench icon) +3. Click **+** to add a custom MCP server +4. Configure: + - **Destination**: Solution or Global + - **Server ID**: Your server name + - **Type**: stdio or HTTP + - For stdio: **Command**: `dotnet run --project path/to/project.csproj` + - For HTTP: **URL**: `http://localhost:3001` + +This creates a `.mcp.json` file in your solution root or global config. + +### 2. Debug with F5 + +1. Right-click your MCP server project → **Set as Startup Project** +2. Open **Properties** → **Debug** → **General** → **Open debug launch profiles UI** +3. Add environment variables as needed +4. Set breakpoints in tool methods +5. Press **F5** to start debugging + +### 3. Conditional Breakpoints + +Right-click a breakpoint → **Conditions**: +- **Condition**: `query == "test"` — break only for specific input +- **Hit Count**: `>= 5` — break after N invocations +- **Filter**: `ProcessName == "MyMcpServer"` — filter by process + +## Auto-Detect and Generate mcp.json + +Script to auto-detect transport and generate config: + +```powershell +$proj = (Get-ChildItem *.csproj | Select-Object -First 1) +$name = $proj.BaseName +$isHttp = (Get-Content $proj.FullName -Raw) -match 'ModelContextProtocol\.AspNetCore' +$configDir = if (Test-Path .vscode) { ".vscode" } else { "." } +$configPath = Join-Path $configDir "mcp.json" + +if ($isHttp) { + $port = "3001" + $programCs = Get-Content "Program.cs" -Raw -ErrorAction SilentlyContinue + if ($programCs -match 'localhost:(\d+)') { $port = $matches[1] } + $config = @{ servers = @{ $name = @{ type = "http"; url = "http://localhost:$port" } } } +} else { + $config = @{ servers = @{ $name = @{ type = "stdio"; command = "dotnet"; args = @("run", "--project", $proj.Name) } } } +} + +New-Item -ItemType Directory -Path $configDir -Force | Out-Null +$config | ConvertTo-Json -Depth 5 | Out-File $configPath -Encoding utf8 +Write-Host "Created $configPath for $( if ($isHttp) {'HTTP'} else {'stdio'} ) server" +``` diff --git a/plugins/dotnet/skills/mcp-csharp-debug/references/mcp-inspector.md b/plugins/dotnet/skills/mcp-csharp-debug/references/mcp-inspector.md new file mode 100644 index 0000000000..0f3ef75918 --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-debug/references/mcp-inspector.md @@ -0,0 +1,101 @@ +# MCP Inspector + +Interactive debugging tool for testing MCP servers. Provides a web UI for listing tools, calling them with custom parameters, and inspecting protocol messages. + +## Installation + +Requires Node.js (npm/npx). No global install needed: + +```bash +npx @modelcontextprotocol/inspector +``` + +## Connecting to a Server + +### stdio Server + +Pass the server command directly: +```bash +npx @modelcontextprotocol/inspector dotnet run --project +``` + +The Inspector launches the server process and communicates via stdin/stdout. + +### HTTP Server + +1. Start the server separately: + ```bash + cd + dotnet run + ``` + +2. Launch Inspector and connect to the server URL: + ```bash + npx @modelcontextprotocol/inspector + ``` + +3. In the Inspector UI, enter the server URL (e.g., `http://localhost:3001`) + +### File-Based Server (.NET 10+ only) + +For single-file servers using `#:package` directives: +```bash +npx @modelcontextprotocol/inspector ./Program.cs +``` + +## Features + +### Tool Testing + +1. Click **Tools** tab to see all registered tools +2. View each tool's JSON schema (parameters, types, descriptions) +3. Enter parameter values and click **Call** to execute +4. View the return value and any error details + +### Prompt Testing + +1. Click **Prompts** tab to see registered prompts +2. Fill in prompt arguments +3. View the generated messages + +### Resource Browsing + +1. Click **Resources** tab to list registered resources +2. Read resource contents directly in the UI + +### Protocol Inspection + +- View raw JSON-RPC request/response messages +- Inspect headers and metadata +- See timing information for each request + +## Troubleshooting + +### Inspector won't start + +- Verify Node.js is installed: `node --version` +- Try clearing npx cache: `npx clear-npx-cache` then retry + +### Can't connect to stdio server + +- Verify the server builds: `dotnet build` (fix errors first) +- Check that the server doesn't write to stdout (logging must go to stderr for stdio transport) +- Try running the server directly first: `dotnet run` — if it hangs waiting for input, that's correct + +### Can't connect to HTTP server + +- Verify the server is running and the port is correct +- Check firewall/proxy settings +- Try `curl http://localhost:/` to verify basic connectivity + +### Tools not appearing + +- Verify `[McpServerToolType]` and `[McpServerTool]` attributes are present +- Check `.WithToolsFromAssembly()` or `.WithTools()` in Program.cs +- Rebuild the project and retry + +### Tool call returns error + +- Check the Inspector's protocol view for the full error message +- Common issues: missing required parameters, serialization errors, unhandled exceptions in tool code +- Add logging to the tool method and check stderr output (stdio) or console output (HTTP) diff --git a/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md b/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md new file mode 100644 index 0000000000..bcc73b12fe --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md @@ -0,0 +1,275 @@ +--- +name: mcp-csharp-publish +description: > + Publish and deploy C# MCP servers. Covers NuGet packaging for stdio servers, Docker + containerization for HTTP servers, Azure Container Apps and App Service deployment, + and publishing to the official MCP Registry. + USE FOR: packaging stdio MCP servers as NuGet tools, creating Dockerfiles for HTTP MCP + servers, deploying to Azure Container Apps or App Service, publishing to the MCP Registry + at registry.modelcontextprotocol.io, configuring server.json for MCP package metadata, + setting up CI/CD for MCP server publishing. + DO NOT USE FOR: publishing general NuGet libraries (not MCP-specific), general Docker + guidance unrelated to MCP, creating new servers (use mcp-csharp-create), debugging + (use mcp-csharp-debug), writing tests (use mcp-csharp-test). +--- + +# C# MCP Server Publishing + +Publish and deploy MCP servers to their target platforms. stdio servers are distributed as NuGet tool packages. HTTP servers are containerized and deployed to Azure or other container hosts. Both can optionally be listed in the official MCP Registry. + +## When to Use + +- Packaging a stdio MCP server for NuGet distribution +- Creating a Docker container for an HTTP MCP server +- Deploying to Azure Container Apps or App Service +- Publishing to the official MCP Registry for discoverability +- Setting up `server.json` metadata for the MCP Registry + +## Stop Signals + +- **Server not tested yet?** → Use [mcp-csharp-test](../mcp-csharp-test/SKILL.md) first +- **Server not working locally?** → Use [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) +- **No server project yet?** → Use [mcp-csharp-create](../mcp-csharp-create/SKILL.md) +- **Publishing a non-MCP NuGet package?** → Use [nuget-trusted-publishing](../nuget-trusted-publishing/SKILL.md) instead + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Transport type | Yes | `stdio` → NuGet path, `http` → Docker/Azure path | +| Target destination | Yes | NuGet.org, Docker registry, Azure Container Apps, Azure App Service, MCP Registry | +| Project path | Yes | Path to the `.csproj` file | +| Package ID / server name | Required for publishing | NuGet `PackageId` or MCP Registry name | + +## Workflow + +### Step 1: Choose the publishing path + +| Transport | Primary Destination | Users Run With | +|-----------|-------------------|----------------| +| **stdio** | NuGet.org | `dnx YourPackage@version` | +| **HTTP** | Docker → Azure | Container URL | + +Both paths can optionally publish to the MCP Registry for discoverability. + +### Step 2a: NuGet publishing (stdio servers) + +1. **Configure `.csproj`** with package properties: +```xml + + true + mymcpserver + YourUsername.MyMcpServer + 1.0.0 + Your Name + MCP server for interacting with MyService + MIT + mcp;modelcontextprotocol;ai;llm + README.md + + + + + +``` + +2. **Build and pack:** +```bash +dotnet build -c Release +dotnet pack -c Release +``` + +3. **Test locally before publishing:** +```bash +dotnet tool install --global --add-source bin/Release/ YourUsername.MyMcpServer +mymcpserver --help # verify it runs +dotnet tool uninstall --global YourUsername.MyMcpServer +``` + +4. **Push to NuGet.org:** +```bash +dotnet nuget push bin/Release/*.nupkg \ + --api-key YOUR_NUGET_API_KEY \ + --source https://api.nuget.org/v3/index.json +``` + +5. **Verify** — users configure in `mcp.json`: +```json +{ + "servers": { + "MyMcpServer": { + "type": "stdio", + "command": "dnx", + "args": ["YourUsername.MyMcpServer@1.0.0", "--yes"] + } + } +} +``` + +**For detailed NuGet packaging and trusted publishing setup**, see [references/nuget-packaging.md](references/nuget-packaging.md). + +### Step 2b: Docker containerization (HTTP servers) + +1. **Create Dockerfile:** +```dockerfile +FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build +WORKDIR /src +COPY *.csproj ./ +RUN dotnet restore +COPY . ./ +RUN dotnet publish -c Release -o /app + +FROM mcr.microsoft.com/dotnet/aspnet:10.0 +WORKDIR /app +COPY --from=build /app . + +# Non-root user for security +RUN adduser --disabled-password --gecos '' appuser +USER appuser + +ENV ASPNETCORE_URLS=http://+:8080 +EXPOSE 8080 +HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \ + CMD curl -f http://localhost:8080/health || exit 1 +ENTRYPOINT ["dotnet", "MyMcpServer.dll"] +``` + +2. **Build and test locally:** +```bash +docker build -t mymcpserver:latest . +docker run -d -p 3001:8080 -e API_KEY=test-key --name mymcpserver mymcpserver:latest +curl http://localhost:3001/health +``` + +3. **Push to container registry:** +```bash +# Docker Hub +docker tag mymcpserver:latest yourusername/mymcpserver:1.0.0 +docker push yourusername/mymcpserver:1.0.0 + +# Azure Container Registry +az acr login --name yourregistry +docker tag mymcpserver:latest yourregistry.azurecr.io/mymcpserver:1.0.0 +docker push yourregistry.azurecr.io/mymcpserver:1.0.0 +``` + +### Step 3: Deploy to Azure (HTTP servers) + +**Azure Container Apps** (recommended — serverless with auto-scaling): +```bash +az containerapp create \ + --name mymcpserver \ + --resource-group mygroup \ + --environment myenvironment \ + --image yourregistry.azurecr.io/mymcpserver:1.0.0 \ + --target-port 8080 \ + --ingress external \ + --min-replicas 0 \ + --max-replicas 10 \ + --secrets api-key=secretref:api-key \ + --env-vars API_KEY=secretref:api-key +``` + +**Azure App Service** (traditional web hosting): +```bash +az webapp create \ + --name mymcpserver \ + --resource-group mygroup \ + --plan myplan \ + --deployment-container-image-name yourregistry.azurecr.io/mymcpserver:1.0.0 +``` + +**For detailed Azure deployment**, see [references/docker-azure.md](references/docker-azure.md). + +### Step 4: Publish to MCP Registry (optional) + +List your server in the official MCP Registry for discoverability. + +1. **Install `mcp-publisher`:** +```bash +# macOS/Linux +brew install mcp-publisher + +# Or download from https://github.com/modelcontextprotocol/registry/releases +``` + +2. **Create `.mcp/server.json`** (or run `mcp-publisher init` to generate interactively): +```json +{ + "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", + "name": "io.github./", + "description": "Your server description", + "version": "1.0.0", + "packages": [{ + "registryType": "nuget", + "registryBaseUrl": "https://api.nuget.org", + "identifier": "YourUsername.MyMcpServer", + "version": "1.0.0", + "transport": { "type": "stdio" } + }], + "repository": { + "url": "https://github.com//", + "source": "github" + } +} +``` + +> **Version consistency (critical):** The root `version`, `packages[].version`, and `` in `.csproj` **must all match**. A mismatch causes registry validation failures or users downloading the wrong version. + +3. **Authenticate and publish:** +```bash +mcp-publisher login github # name must be io.github./... for GitHub auth +mcp-publisher publish +``` + +4. **Verify:** +```bash +curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github./" +``` + +**For Registry details** (namespace conventions, environment variables, CI/CD automation), see [references/mcp-registry.md](references/mcp-registry.md). + +### Step 5: Security checklist + +- [ ] No hardcoded secrets — use environment variables or Key Vault +- [ ] HTTPS enabled for HTTP transport in production +- [ ] Health check endpoint implemented +- [ ] Input validation on all tool parameters +- [ ] Rate limiting considered for HTTP servers + +## Validation + +- [ ] **NuGet:** Package installs and runs via `dnx PackageId@version` +- [ ] **Docker:** Container starts and health check passes +- [ ] **Azure:** Server is reachable and tools respond +- [ ] **MCP Registry:** Server appears at `registry.modelcontextprotocol.io` +- [ ] MCP client can connect and call tools on the deployed server + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| NuGet package doesn't run as a tool | Missing `true` in `.csproj` | +| Version mismatch between `.csproj` and `server.json` | Keep ``, `server.json` root `version`, and `packages[].version` in sync | +| Docker container exits immediately | Check entrypoint DLL name matches project output. Run `docker logs mymcpserver` for errors | +| Azure Container App returns 502 | Target port mismatch. Ensure `--target-port` matches `ASPNETCORE_URLS` port in the container | +| MCP Registry rejects publish | Name must follow namespace convention: `io.github./` for GitHub auth | +| API keys leaked in Docker image | Use multi-stage builds. Never `COPY` `.env` files. Pass secrets via `--env-vars` at runtime | + +## Related Skills + +- [mcp-csharp-create](../mcp-csharp-create/SKILL.md) — Create a new MCP server project +- [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) — Running and interactive debugging +- [mcp-csharp-test](../mcp-csharp-test/SKILL.md) — Automated tests and evaluations + +## Reference Files + +- [references/nuget-packaging.md](references/nuget-packaging.md) — Complete NuGet `.csproj` configuration, `server.json` for MCP, NuGet.org push, testing with `dnx`, version management. **Load when:** publishing a stdio server to NuGet. +- [references/docker-azure.md](references/docker-azure.md) — Production Dockerfile patterns, ACR setup, Azure Container Apps full configuration, App Service with Key Vault, secrets management. **Load when:** deploying an HTTP server to Docker or Azure. +- [references/mcp-registry.md](references/mcp-registry.md) — `mcp-publisher` CLI installation, `server.json` schema, namespace conventions (GitHub vs DNS auth), CI/CD automation. **Load when:** publishing to the official MCP Registry. + +## More Info + +- [NuGet publishing](https://learn.microsoft.com/nuget/nuget-org/publish-a-package) — NuGet.org publishing guide +- [Azure Container Apps](https://learn.microsoft.com/azure/container-apps/) — Serverless container hosting +- [MCP Registry](https://registry.modelcontextprotocol.io) — Official MCP server registry diff --git a/plugins/dotnet/skills/mcp-csharp-publish/references/docker-azure.md b/plugins/dotnet/skills/mcp-csharp-publish/references/docker-azure.md new file mode 100644 index 0000000000..7afa4a231f --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-publish/references/docker-azure.md @@ -0,0 +1,187 @@ +# Docker and Azure Deployment + +Production deployment patterns for HTTP MCP servers. + +## Production Dockerfile + +```dockerfile +# Build stage +FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build +WORKDIR /src + +# Restore first (layer caching) +COPY *.csproj ./ +RUN dotnet restore + +# Build and publish +COPY . ./ +RUN dotnet publish -c Release -o /app --no-restore + +# Runtime stage +FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime +WORKDIR /app +COPY --from=build /app . + +# Non-root user (security) +RUN adduser --disabled-password --gecos '' appuser +USER appuser + +# Configure +ENV ASPNETCORE_URLS=http://+:8080 +EXPOSE 8080 + +# Health check +HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \ + CMD curl -f http://localhost:8080/health || exit 1 + +ENTRYPOINT ["dotnet", "MyMcpServer.dll"] +``` + +### Key Practices + +- **Multi-stage build** — keeps the final image small (no SDK, only runtime) +- **Restore-first layer** — `COPY *.csproj` then `dotnet restore` before `COPY .` for better layer caching +- **Non-root user** — run as unprivileged user in production +- **Health check** — enables orchestrator health monitoring +- **Never copy secrets** — no `.env` files, no `appsettings.Production.json` with secrets + +## Azure Container Registry (ACR) + +```bash +# Create ACR (one-time) +az acr create --name yourregistry --resource-group mygroup --sku Basic + +# Login +az acr login --name yourregistry + +# Build and push +docker build -t yourregistry.azurecr.io/mymcpserver:1.0.0 . +docker push yourregistry.azurecr.io/mymcpserver:1.0.0 + +# Or build directly in ACR (no local Docker needed) +az acr build --registry yourregistry --image mymcpserver:1.0.0 . +``` + +## Azure Container Apps + +Serverless container hosting with auto-scaling. Best for MCP servers that need to scale to zero when idle. + +### Full Setup + +```bash +# Create environment (one-time) +az containerapp env create \ + --name myenvironment \ + --resource-group mygroup \ + --location eastus + +# Create the container app +az containerapp create \ + --name mymcpserver \ + --resource-group mygroup \ + --environment myenvironment \ + --image yourregistry.azurecr.io/mymcpserver:1.0.0 \ + --registry-server yourregistry.azurecr.io \ + --target-port 8080 \ + --ingress external \ + --min-replicas 0 \ + --max-replicas 10 \ + --cpu 0.5 \ + --memory 1.0Gi \ + --secrets api-key="your-secret-value" \ + --env-vars API_KEY=secretref:api-key + +# Get the URL +az containerapp show --name mymcpserver --resource-group mygroup \ + --query properties.configuration.ingress.fqdn -o tsv +``` + +### Update Deployment + +```bash +az containerapp update \ + --name mymcpserver \ + --resource-group mygroup \ + --image yourregistry.azurecr.io/mymcpserver:1.1.0 +``` + +### Scaling Configuration + +```bash +# Scale based on HTTP requests +az containerapp update \ + --name mymcpserver \ + --resource-group mygroup \ + --scale-rule-name http-rule \ + --scale-rule-type http \ + --scale-rule-http-concurrency 50 +``` + +## Azure App Service + +Traditional web hosting with more control over infrastructure. + +```bash +# Create App Service plan +az appservice plan create \ + --name myplan \ + --resource-group mygroup \ + --sku B1 \ + --is-linux + +# Create web app with container +az webapp create \ + --name mymcpserver \ + --resource-group mygroup \ + --plan myplan \ + --deployment-container-image-name yourregistry.azurecr.io/mymcpserver:1.0.0 + +# Configure secrets via Key Vault +az webapp config appsettings set \ + --name mymcpserver \ + --resource-group mygroup \ + --settings API_KEY=@Microsoft.KeyVault(VaultName=myvault;SecretName=api-key) +``` + +## Secrets Management + +### Azure Container Apps + +```bash +# Add a secret +az containerapp secret set --name mymcpserver --resource-group mygroup \ + --secrets api-key="value" + +# Reference in environment variables +az containerapp update --name mymcpserver --resource-group mygroup \ + --set-env-vars API_KEY=secretref:api-key +``` + +### Azure Key Vault (App Service) + +```bash +# Create Key Vault +az keyvault create --name myvault --resource-group mygroup + +# Add secret +az keyvault secret set --vault-name myvault --name api-key --value "your-secret" + +# Grant access to App Service identity +az webapp identity assign --name mymcpserver --resource-group mygroup +az keyvault set-policy --name myvault \ + --object-id \ + --secret-permissions get list +``` + +### In Application Code + +```csharp +// Read from environment (works with both approaches) +var apiKey = Environment.GetEnvironmentVariable("API_KEY") + ?? throw new InvalidOperationException("API_KEY environment variable required"); + +// Or use Azure Key Vault directly +builder.Configuration.AddAzureKeyVault( + new Uri($"https://{vaultName}.vault.azure.net/"), + new DefaultAzureCredential()); +``` diff --git a/plugins/dotnet/skills/mcp-csharp-publish/references/mcp-registry.md b/plugins/dotnet/skills/mcp-csharp-publish/references/mcp-registry.md new file mode 100644 index 0000000000..3ec98f115e --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-publish/references/mcp-registry.md @@ -0,0 +1,140 @@ +# MCP Registry + +Publish your MCP server to the official registry at [registry.modelcontextprotocol.io](https://registry.modelcontextprotocol.io) for discoverability. + +## When to Publish + +| Publish if… | Skip if… | +|-------------|----------| +| Server is for public/community use | Server is internal/private | +| You want discoverability in MCP clients | Still developing/testing | +| You want to appear in the official registry | No need for public discovery | + +## Prerequisites + +1. Package published to NuGet.org (for stdio) or container registry (for HTTP) +2. GitHub repository with the server source code +3. `mcp-publisher` CLI installed + +## Install mcp-publisher + +```bash +# macOS/Linux (Homebrew) +brew install mcp-publisher + +# Or download binary from releases +# https://github.com/modelcontextprotocol/registry/releases +``` + +## server.json Schema + +Place at `.mcp/server.json` in your repository root: + +```json +{ + "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", + "name": "io.github./", + "description": "One-line description of your server", + "version": "1.0.0", + "packages": [ + { + "registryType": "nuget", + "registryBaseUrl": "https://api.nuget.org", + "identifier": "YourUsername.MyMcpServer", + "version": "1.0.0", + "transport": { + "type": "stdio" + }, + "packageArguments": [], + "environmentVariables": [ + { + "name": "API_KEY", + "value": "{api_key}", + "variables": { + "api_key": { + "description": "API key for MyService authentication", + "isRequired": true, + "isSecret": true + } + } + } + ] + } + ], + "repository": { + "url": "https://github.com//", + "source": "github" + } +} +``` + +## Namespace Conventions + +The `name` field must follow a namespace convention based on your authentication method: + +| Auth Method | Name Format | Example | +|-------------|-------------|---------| +| GitHub | `io.github./` | `io.github.jsmith/weather-server` | +| DNS | `/` | `com.mycompany/weather-server` | + +## Publish Workflow + +```bash +# 1. Initialize server.json (interactive, if not already created) +mcp-publisher init + +# 2. Authenticate with GitHub +mcp-publisher login github + +# 3. Publish to the registry +mcp-publisher publish + +# 4. Verify publication +curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github./" +``` + +## CI/CD Automation + +### GitHub Actions + +```yaml +name: Publish to MCP Registry +on: + release: + types: [published] + +jobs: + publish: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install mcp-publisher + run: | + curl -sSL https://github.com/modelcontextprotocol/registry/releases/latest/download/mcp-publisher-linux-amd64 -o mcp-publisher + chmod +x mcp-publisher + + - name: Publish to registry + run: ./mcp-publisher publish + env: + MCP_REGISTRY_TOKEN: ${{ secrets.MCP_REGISTRY_TOKEN }} +``` + +## Version Consistency + +Always keep these three versions in sync: + +1. `` in `.csproj` +2. `version` at root of `server.json` +3. `packages[].version` in `server.json` + +A mismatch between any of these will cause registry validation to fail or users to get the wrong version. + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| "Invalid name format" | Use `io.github./` format | +| "Package not found" | Package must be published to NuGet.org first | +| "Version mismatch" | Sync `.csproj` version with both `server.json` version fields | +| "Authentication failed" | Re-run `mcp-publisher login github` | diff --git a/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md b/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md new file mode 100644 index 0000000000..ce2c79f2fe --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md @@ -0,0 +1,144 @@ +# NuGet Packaging + +Detailed guide for publishing stdio MCP servers as NuGet tool packages. + +## Complete .csproj Configuration + +```xml + + + Exe + net10.0 + enable + enable + + + true + mymcpserver + + + YourUsername.MyMcpServer + 1.0.0 + Your Name + MCP server for interacting with MyService API + + + https://github.com/yourusername/mymcpserver + https://github.com/yourusername/mymcpserver + MIT + mcp;modelcontextprotocol;ai;llm + README.md + + + win-x64;linux-x64;osx-x64;osx-arm64 + + + + + + +``` + +### Key Properties + +| Property | Required | Purpose | +|----------|----------|---------| +| `PackAsTool` | Yes | Makes the package installable as a dotnet tool | +| `ToolCommandName` | Recommended | CLI command name. Defaults to assembly name if omitted | +| `PackageId` | Yes | Unique identifier on NuGet.org | +| `Version` | Yes | SemVer version (e.g., `1.0.0`, `2.0.0-preview.1`) | +| `PackageTags` | Recommended | Include `mcp` and `modelcontextprotocol` for discoverability | + +## Build, Pack, and Push + +```bash +# Build +dotnet build -c Release + +# Create NuGet package +dotnet pack -c Release +# Output: bin/Release/YourUsername.MyMcpServer.1.0.0.nupkg + +# Test package locally +dotnet tool install --global --add-source bin/Release/ YourUsername.MyMcpServer +mymcpserver --help +dotnet tool uninstall --global YourUsername.MyMcpServer + +# Push to NuGet.org +dotnet nuget push bin/Release/*.nupkg \ + --api-key YOUR_NUGET_API_KEY \ + --source https://api.nuget.org/v3/index.json + +# Or push to NuGet test environment first +dotnet nuget push bin/Release/*.nupkg \ + --api-key YOUR_NUGET_API_KEY \ + --source https://apiint.nugettest.org/v3/index.json +``` + +## User Configuration + +After publishing, users configure their MCP client to run the tool: + +```json +{ + "servers": { + "MyMcpServer": { + "type": "stdio", + "command": "dnx", + "args": ["YourUsername.MyMcpServer@1.0.0", "--yes"], + "env": { + "API_KEY": "${input:api_key}" + } + } + } +} +``` + +The `dnx` tool runner (`dotnet tool run` shorthand) downloads and runs the tool automatically. + +## server.json for MCP Registry Integration + +If you plan to publish to the MCP Registry, include `.mcp/server.json` in your repo: + +```json +{ + "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", + "name": "io.github.yourusername/mymcpserver", + "description": "MCP server for interacting with MyService API", + "version": "1.0.0", + "packages": [ + { + "registryType": "nuget", + "registryBaseUrl": "https://api.nuget.org", + "identifier": "YourUsername.MyMcpServer", + "version": "1.0.0", + "transport": { + "type": "stdio" + }, + "environmentVariables": [ + { + "name": "API_KEY", + "value": "{api_key}", + "variables": { + "api_key": { + "description": "API key for MyService", + "isRequired": true, + "isSecret": true + } + } + } + ] + } + ], + "repository": { + "url": "https://github.com/yourusername/mymcpserver", + "source": "github" + } +} +``` + +**Version consistency:** Keep `` in `.csproj`, root `version` in `server.json`, and `packages[].version` in sync. A mismatch will cause MCP Registry validation to fail. + +## Trusted Publishing (OIDC) + +For CI/CD, use NuGet trusted publishing instead of long-lived API keys. See [nuget-trusted-publishing](../../nuget-trusted-publishing/SKILL.md) for the full setup guide. diff --git a/plugins/dotnet/skills/mcp-csharp-test/SKILL.md b/plugins/dotnet/skills/mcp-csharp-test/SKILL.md new file mode 100644 index 0000000000..9c332ae95f --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-test/SKILL.md @@ -0,0 +1,206 @@ +--- +name: mcp-csharp-test +description: > + Test C# MCP servers at multiple levels: unit tests for individual tools, integration + tests using the MCP client SDK, and LLM effectiveness evaluations. + USE FOR: unit testing MCP tool methods, integration testing with in-memory MCP + client/server, end-to-end testing via MCP protocol, creating LLM evaluation question sets, + testing HTTP MCP servers with WebApplicationFactory, mocking dependencies in tool tests. + DO NOT USE FOR: testing MCP clients (this is server testing only), load or performance + testing, testing non-.NET MCP servers, debugging server issues (use mcp-csharp-debug). +--- + +# C# MCP Server Testing + +Test MCP servers at three levels: unit tests for individual tool methods, integration tests that exercise the full MCP protocol in-memory, and evaluations that measure LLM effectiveness when using your tools. + +## When to Use + +- Adding automated tests to an MCP server +- Testing individual tool methods with mocked dependencies +- Writing integration tests that validate tool listing and invocation via MCP protocol +- Creating evaluation question sets to measure LLM effectiveness with your tools +- Setting up CI test pipelines for MCP servers + +## Stop Signals + +- **No server yet?** → Use [mcp-csharp-create](../mcp-csharp-create/SKILL.md) first +- **Server not running?** → Use [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) +- **Just need manual/interactive testing?** → Use [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) for MCP Inspector + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| MCP server project path | Yes | Path to the server `.csproj` being tested | +| Test framework | Recommended | Default: xUnit. Also supports NUnit or MSTest | +| Transport type | Recommended | Determines integration test approach (stdio vs HTTP) | + +## Workflow + +### Step 1: Create the test project + +```bash +dotnet new xunit -n .Tests +cd .Tests +dotnet add reference ..//.csproj +dotnet add package ModelContextProtocol +dotnet add package Moq +dotnet add package FluentAssertions +``` + +### Step 2: Write unit tests for tool methods + +Test tool methods directly — fastest and most isolated: + +```csharp +public class MyToolTests +{ + [Fact] + public void Echo_ReturnsFormattedMessage() + { + var result = MyTools.Echo("Hello"); + result.Should().Be("Echo: Hello"); + } + + [Theory] + [InlineData("")] + [InlineData(" ")] + public void Echo_HandlesEdgeCases(string input) + { + var result = MyTools.Echo(input); + result.Should().StartWith("Echo:"); + } +} +``` + +For tools with DI dependencies, mock the dependency: +```csharp +public class ApiToolTests +{ + [Fact] + public async Task FetchData_ReturnsApiResponse() + { + var handler = new MockHttpMessageHandler("""{"id": 1}"""); + var httpClient = new HttpClient(handler); + + var result = await ApiTools.FetchData(httpClient, "resource-1"); + result.Should().Contain("id"); + } +} +``` + +### Step 3: Write integration tests with MCP client + +Test the full MCP protocol using a client-server connection: + +```csharp +using ModelContextProtocol.Client; + +public class ServerIntegrationTests : IAsyncLifetime +{ + private McpClient _client = null!; + + public async Task InitializeAsync() + { + var transport = new StdioClientTransport(new StdioClientTransportOptions + { + Name = "TestClient", + Command = "dotnet", + Arguments = ["run", "--project", "..//.csproj"] + }); + _client = await McpClient.CreateAsync(transport); + } + + public async Task DisposeAsync() => await _client.DisposeAsync(); + + [Fact] + public async Task Server_ListsExpectedTools() + { + var tools = await _client.ListToolsAsync(); + tools.Should().Contain(t => t.Name == "echo"); + } + + [Fact] + public async Task Tool_ReturnsExpectedResult() + { + var result = await _client.CallToolAsync("echo", + new Dictionary { ["message"] = "Test" }); + var text = result.Content.OfType().First().Text; + text.Should().Contain("Test"); + } +} +``` + +**For the SDK's `ClientServerTestBase` (in-memory testing) and HTTP testing with `WebApplicationFactory`**, see [references/test-patterns.md](references/test-patterns.md). + +### Step 4: Create evaluations (optional) + +Evaluations measure how effectively LLMs use your tools to accomplish tasks. Create questions that are independent, read-only, require multiple tool calls, and have verifiable answers. + +```xml + + + + MyMcpServer + 1.0.0 + + + Using the search tool, find users in "engineering" who joined + after 2024. What domain is most common in their emails? + company.com + medium + search_users, list_teams + + +``` + +**For evaluation guidelines, good/bad question examples, and the full XML schema**, see [references/evaluation-guide.md](references/evaluation-guide.md). + +### Step 5: Run tests + +```bash +# Run all tests +dotnet test + +# Run a specific test class +dotnet test --filter "FullyQualifiedName~MyToolTests" + +# Run with coverage +dotnet test --collect:"XPlat Code Coverage" +``` + +## Validation + +- [ ] Unit tests cover all tool methods, including edge cases +- [ ] Integration tests verify tool listing via `ListToolsAsync()` +- [ ] Integration tests verify tool invocation via `CallToolAsync()` +- [ ] All tests pass: `dotnet test` +- [ ] Tests run in CI without manual setup + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Integration test hangs on `CreateAsync` | Server fails to start. Verify `dotnet build` succeeds first. For stdio, ensure no stdout logging | +| `StdioClientTransport` not finding project | Use the correct relative path to `.csproj` from the test project directory | +| Tests pass locally but fail in CI | Run `dotnet build` before test execution. Use `--no-build` only after an explicit build step | +| Mocking `HttpClient` is awkward | Mock `HttpMessageHandler`, not `HttpClient` directly. See [references/test-patterns.md](references/test-patterns.md) | +| Evaluation answers change over time | Use stable, deterministic data. Avoid questions about "latest" or time-dependent values | +| Full test suite runs are slow | Use `--filter` for development. Run the full suite only for CI verification | + +## Related Skills + +- [mcp-csharp-create](../mcp-csharp-create/SKILL.md) — Create a new MCP server project +- [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) — Running and interactive debugging +- [mcp-csharp-publish](../mcp-csharp-publish/SKILL.md) — NuGet, Docker, Azure deployment + +## Reference Files + +- [references/test-patterns.md](references/test-patterns.md) — Complete test code examples: `ClientServerTestBase` in-memory pattern, `WebApplicationFactory` for HTTP, `MockHttpMessageHandler` helper, test categorization, coverage reporting. **Load when:** writing integration tests or need detailed mock patterns. +- [references/evaluation-guide.md](references/evaluation-guide.md) — Evaluation creation guidelines, XML format reference, good/bad question examples, criteria for verifiable evaluations. **Load when:** creating LLM effectiveness evaluations. + +## More Info + +- [xUnit documentation](https://xunit.net/docs/getting-started/netcore/cmdline) — Getting started with xUnit for .NET +- [FluentAssertions](https://fluentassertions.com/) — Readable assertion library for .NET diff --git a/plugins/dotnet/skills/mcp-csharp-test/references/evaluation-guide.md b/plugins/dotnet/skills/mcp-csharp-test/references/evaluation-guide.md new file mode 100644 index 0000000000..767efb185b --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-test/references/evaluation-guide.md @@ -0,0 +1,90 @@ +# Evaluation Guide + +Create evaluations that measure how effectively LLMs use your MCP server tools. + +## Evaluation Criteria + +Every evaluation question should be: + +| Criterion | Meaning | +|-----------|---------| +| **Independent** | Not dependent on other questions or prior state | +| **Read-only** | Uses only non-destructive operations | +| **Complex** | Requires multiple tool calls and reasoning | +| **Realistic** | Based on real use cases for the server | +| **Verifiable** | Has a single, clear, deterministic answer | +| **Stable** | Answer won't change over time | + +## XML Format + +```xml + + + + MyMcpServer + 1.0.0 + 2026-03-01 + + + + + Using the user search tool, find all users in the "engineering" team + who joined after 2024. What is the email domain most commonly used + by these users? + + company.com + medium + search_users, list_teams + + + + + Find the project with the most active contributors in the last month. + What is the project's internal ID? + + proj-42 + hard + list_projects, get_project_stats + + + + +``` + +## Creating Good Questions + +### Good question patterns + +- **Multi-step retrieval**: "Find X, then use that to look up Y" — requires chaining tool calls +- **Aggregation**: "Which category has the most items matching criteria Z?" — requires gathering and comparing +- **Cross-referencing**: "Find the overlap between set A and set B" — uses multiple tools + +### Examples + +**Good:** +- "Find the user who created the most issues in 'backend' this year. What is their username?" +- "Which team has the highest average code review turnaround time? Return the team name." +- "How many open issues are labeled 'critical' and assigned to users in the 'security' team?" + +**Bad:** +- "What is 2 + 2?" — doesn't use tools +- "List all users" — too simple, single tool call, no reasoning +- "Create a new issue…" — not read-only, mutates state +- "What's the current weather?" — answer changes over time +- "Tell me about the system" — no verifiable answer + +## Evaluation Process + +1. **Explore available data** — use read-only tools to understand what data exists +2. **Draft questions** that require reasoning across multiple tool calls +3. **Solve each question manually** to confirm the answer is correct and deterministic +4. **Verify stability** — run the evaluation twice to ensure answers are consistent +5. **Aim for 10 questions** with a mix of medium and hard difficulty + +## Difficulty Guidelines + +| Difficulty | Tool Calls | Description | +|------------|-----------|-------------| +| Easy | 1 | Single tool, direct answer | +| Medium | 2-3 | Chain tools, simple reasoning | +| Hard | 4+ | Multiple tools, aggregation, cross-referencing | diff --git a/plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md b/plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md new file mode 100644 index 0000000000..f373a3c9c6 --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md @@ -0,0 +1,175 @@ +# Test Patterns + +Complete code patterns for testing C# MCP servers at every level. + +## MockHttpMessageHandler Helper + +Reusable mock for tools that use `HttpClient`: + +```csharp +public class MockHttpMessageHandler : HttpMessageHandler +{ + private readonly string _response; + private readonly HttpStatusCode _statusCode; + + public MockHttpMessageHandler( + string response = "", + HttpStatusCode statusCode = HttpStatusCode.OK) + { + _response = response; + _statusCode = statusCode; + } + + protected override Task SendAsync( + HttpRequestMessage request, + CancellationToken cancellationToken) => + Task.FromResult(new HttpResponseMessage + { + StatusCode = _statusCode, + Content = new StringContent(_response) + }); +} +``` + +## ClientServerTestBase (In-Memory Testing) + +The SDK provides `ClientServerTestBase` for zero-network integration tests using `System.IO.Pipelines`: + +```csharp +using ModelContextProtocol.Tests; // from SDK test utilities + +public class MyToolTests : ClientServerTestBase +{ + public MyToolTests(ITestOutputHelper output) : base(output) { } + + protected override void ConfigureServices( + ServiceCollection services, IMcpServerBuilder builder) + { + builder.WithTools(); + // Register any DI services your tools need + services.AddSingleton(); + } + + [Fact] + public async Task MyTool_ReturnsExpected() + { + await using var client = await CreateMcpClientForServer(); + var result = await client.CallToolAsync("my_tool", + new() { ["input"] = "test" }, + cancellationToken: TestContext.Current.CancellationToken); + Assert.NotNull(result); + } +} +``` + +**Key advantages:** +- In-memory transport — no process spawning, no network +- Full DI support — inject fakes/mocks for external dependencies +- Runs in milliseconds + +## HTTP Testing with WebApplicationFactory + +Test HTTP MCP servers using ASP.NET Core's test infrastructure: + +```csharp +using Microsoft.AspNetCore.Mvc.Testing; + +public class HttpServerTests : IClassFixture> +{ + private readonly WebApplicationFactory _factory; + + public HttpServerTests(WebApplicationFactory factory) + { + _factory = factory; + } + + [Fact] + public async Task McpEndpoint_AcceptsInitialize() + { + var client = _factory.CreateClient(); + var request = new + { + jsonrpc = "2.0", + id = 1, + method = "initialize", + @params = new + { + protocolVersion = "2024-11-05", + capabilities = new { }, + clientInfo = new { name = "test", version = "1.0" } + } + }; + + var response = await client.PostAsJsonAsync("/", request); + response.EnsureSuccessStatusCode(); + } + + [Fact] + public async Task HealthEndpoint_ReturnsOk() + { + var client = _factory.CreateClient(); + var response = await client.GetAsync("/health"); + response.EnsureSuccessStatusCode(); + } +} +``` + +**Note:** Requires `` or a public `Program` class. + +## Input Validation Tests + +```csharp +public class ValidationTests +{ + [Theory] + [InlineData(0)] + [InlineData(-1)] + [InlineData(101)] + public void Search_ClampsInvalidLimit(int invalidLimit) + { + var result = SearchTools.Search("query", limit: invalidLimit); + result.Should().NotBeNull(); + } + + [Fact] + public void Search_HandlesSpecialCharacters() + { + var result = SearchTools.Search("'; DROP TABLE users; --"); + result.Should().NotContain("DROP TABLE"); + } +} +``` + +## Test Categories + +Organize tests with traits for selective execution: + +```csharp +[Trait("Category", "Unit")] +public class UnitTests { ... } + +[Trait("Category", "Integration")] +public class IntegrationTests { ... } +``` + +Run by category: +```bash +dotnet test --filter "Category=Unit" +dotnet test --filter "Category=Integration" +``` + +## Coverage Reporting + +```bash +# Add coverage collector +dotnet add package coverlet.collector + +# Run with coverage +dotnet test --collect:"XPlat Code Coverage" + +# Generate HTML report +dotnet tool install --global dotnet-reportgenerator-globaltool +reportgenerator \ + -reports:TestResults/**/coverage.cobertura.xml \ + -targetdir:coveragereport +``` diff --git a/tests/dotnet/mcp-csharp-create/eval.yaml b/tests/dotnet/mcp-csharp-create/eval.yaml new file mode 100644 index 0000000000..214d4268da --- /dev/null +++ b/tests/dotnet/mcp-csharp-create/eval.yaml @@ -0,0 +1,56 @@ +scenarios: + - name: "Create a new stdio MCP server from scratch" + prompt: | + I want to create a new C# MCP server that provides tools for interacting + with a REST API for managing TODO items. It should use stdio transport. + Help me scaffold the project and implement a basic tool. + assertions: + - type: "output_contains" + value: "dotnet new mcpserver" + - type: "output_contains" + value: "[McpServerTool]" + - type: "output_matches" + pattern: "(WithStdioServerTransport|stdio)" + rubric: + - "Checks for or installs the mcpserver template before scaffolding" + - "Uses dotnet new mcpserver to scaffold the project" + - "Shows a tool method with [McpServerToolType] and [McpServerTool] attributes" + - "Includes [Description] attributes on the method and parameters" + - "Configures stdio transport in Program.cs" + timeout: 240 + - name: "Create an HTTP MCP server with ASP.NET Core" + prompt: | + I need to create a C# MCP server that uses HTTP transport so it can be + deployed as a web service. It should have a tool that queries a database. + How do I set this up? + assertions: + - type: "output_contains" + value: "MapMcp" + - type: "output_matches" + pattern: "(WithHttpTransport|--transport remote|--transport http)" + - type: "output_contains" + value: "ModelContextProtocol.AspNetCore" + rubric: + - "Scaffolds with dotnet new mcpserver --transport remote or configures HTTP manually" + - "Includes MapMcp() in the endpoint configuration" + - "References ModelContextProtocol.AspNetCore package" + - "Shows how to register tools with the DI container" + timeout: 300 + - name: "Add tools with dependency injection to an existing MCP server" + prompt: | + I have an existing C# MCP server project. I want to add a new tool class + that needs an HttpClient and ILogger injected. How do I set up the DI + and implement the tool? + assertions: + - type: "output_contains" + value: "[McpServerToolType]" + - type: "output_contains" + value: "[McpServerTool]" + - type: "output_matches" + pattern: "(AddHttpClient|HttpClient)" + rubric: + - "Shows a tool class with constructor injection of HttpClient and ILogger" + - "Registers HttpClient in the DI container using AddHttpClient or similar" + - "Uses [McpServerToolType] on the class and [McpServerTool] on methods" + - "Adds [Description] attributes for LLM discoverability" + timeout: 120 diff --git a/tests/dotnet/mcp-csharp-debug/eval.yaml b/tests/dotnet/mcp-csharp-debug/eval.yaml new file mode 100644 index 0000000000..e635b86e34 --- /dev/null +++ b/tests/dotnet/mcp-csharp-debug/eval.yaml @@ -0,0 +1,49 @@ +scenarios: + - name: "Debug an MCP server with MCP Inspector" + prompt: | + I have a C# MCP server that uses stdio transport. I want to test it + interactively with the MCP Inspector to see if my tools are working. + How do I connect and debug? + assertions: + - type: "output_contains" + value: "npx @modelcontextprotocol/inspector" + - type: "output_matches" + pattern: "(dotnet run|--project)" + rubric: + - "Shows how to launch MCP Inspector with npx @modelcontextprotocol/inspector" + - "Explains how to connect it to the stdio server using dotnet run" + - "Mentions that the Inspector UI shows tools, prompts, and resources" + - "Notes that server logging must go to stderr, not stdout" + timeout: 120 + - name: "Configure VS Code to use an MCP server" + prompt: | + I built a C# MCP server at src/MyMcpServer/MyMcpServer.csproj. + I want to use it with GitHub Copilot in VS Code agent mode. + How do I configure mcp.json? + assertions: + - type: "output_contains" + value: "mcp.json" + - type: "output_matches" + pattern: "(dotnet|run|--project)" + rubric: + - "Shows the mcp.json configuration with type stdio" + - "Uses dotnet run --project with the correct project path" + - "Places the config in .vscode/mcp.json for workspace or user settings" + - "Mentions testing the configuration by asking Copilot to use the tools" + timeout: 120 + - name: "Debug a failing MCP server tool" + prompt: | + My C# MCP server is returning errors when I call one of its tools from + VS Code Copilot. The tool works fine when I run it manually as a console + app. How do I debug this? + assertions: + - type: "output_matches" + pattern: "(stderr|Console\\.Error|logging)" + - type: "output_matches" + pattern: "(breakpoint|attach|debugger)" + rubric: + - "Explains that stdout is reserved for MCP protocol in stdio mode" + - "Recommends checking that all logging goes to stderr" + - "Shows how to attach a debugger to the running server process" + - "Suggests using MCP Inspector or VS Code debug config for step-through debugging" + timeout: 120 diff --git a/tests/dotnet/mcp-csharp-publish/eval.yaml b/tests/dotnet/mcp-csharp-publish/eval.yaml new file mode 100644 index 0000000000..c783fcfda5 --- /dev/null +++ b/tests/dotnet/mcp-csharp-publish/eval.yaml @@ -0,0 +1,55 @@ +scenarios: + - name: "Publish an MCP server as a NuGet tool package" + prompt: | + I have a C# MCP server using stdio transport at src/MyMcpServer/. + I want to publish it to NuGet.org so users can install it as a + dotnet tool. What do I need to configure? + assertions: + - type: "output_contains" + value: "PackAsTool" + - type: "output_contains" + value: "dotnet pack" + - type: "output_matches" + pattern: "(dotnet nuget push|nuget\\.org)" + rubric: + - "Configures PackAsTool and ToolCommandName in the csproj" + - "Shows how to build, pack, and push to NuGet.org" + - "Includes local testing with dotnet tool install --global" + - "Mentions the dnx tool runner for MCP client configuration" + timeout: 120 + - name: "Deploy an HTTP MCP server to Azure Container Apps" + prompt: | + I have an HTTP-based C# MCP server that I want to deploy to Azure. + It uses environment variables for API keys. What's the best way + to containerize and deploy it? + assertions: + - type: "output_matches" + pattern: "(Dockerfile|docker)" + - type: "output_matches" + pattern: "(Container Apps|containerapp)" + - type: "output_matches" + pattern: "(secret|Secret|KEY)" + rubric: + - "Provides a multi-stage Dockerfile with a non-root user" + - "Shows Azure Container Apps deployment commands" + - "Configures secrets using Container Apps secrets, not environment variables in plain text" + - "Includes a health check endpoint" + timeout: 120 + - name: "Publish to the MCP Registry" + prompt: | + I already published my C# MCP server to NuGet.org. Now I want to + register it in the official MCP Registry so it's discoverable. + How do I do this? + assertions: + - type: "output_contains" + value: "server.json" + - type: "output_contains" + value: "mcp-publisher" + - type: "output_matches" + pattern: "io\\.github\\." + rubric: + - "Shows the server.json schema with the correct format" + - "Uses the mcp-publisher CLI for publishing" + - "Explains the naming convention (io.github./)" + - "Emphasizes version consistency between csproj and server.json" + timeout: 120 diff --git a/tests/dotnet/mcp-csharp-test/eval.yaml b/tests/dotnet/mcp-csharp-test/eval.yaml new file mode 100644 index 0000000000..f2d04372e7 --- /dev/null +++ b/tests/dotnet/mcp-csharp-test/eval.yaml @@ -0,0 +1,51 @@ +scenarios: + - name: "Write unit and integration tests for an MCP server" + prompt: | + I have a C# MCP server with tools that call an external REST API. + I want to write tests that verify the tools work correctly without + calling the real API. How do I structure the test project? + assertions: + - type: "output_contains" + value: "xunit" + - type: "output_matches" + pattern: "(Mock|Fake|mock|fake|HttpMessageHandler)" + - type: "output_matches" + pattern: "(CallToolAsync|McpClient)" + rubric: + - "Creates a test project with xUnit and references the server project" + - "Shows how to mock HttpClient or external dependencies" + - "Demonstrates an integration test using McpClient to call tools in-memory" + - "Separates unit tests from integration tests using traits or folders" + timeout: 120 + - name: "Test an HTTP MCP server with WebApplicationFactory" + prompt: | + I have an HTTP-based C# MCP server built with ASP.NET Core. + How do I write integration tests that test the full MCP protocol + over HTTP without spinning up a real server? + assertions: + - type: "output_contains" + value: "WebApplicationFactory" + - type: "output_matches" + pattern: "(initialize|MCP|jsonrpc)" + rubric: + - "Uses WebApplicationFactory for in-process HTTP testing" + - "Shows how to send an MCP initialize request to verify the server responds" + - "Mentions InternalsVisibleTo or making Program public for test access" + - "Tests tool invocation through the HTTP endpoint" + timeout: 120 + - name: "Create evaluations for an MCP server" + prompt: | + I have a C# MCP server that provides tools for querying a product catalog. + I want to create evaluations to measure how well an LLM uses the tools. + What format should I use and what makes a good evaluation question? + assertions: + - type: "output_matches" + pattern: "(evaluation|qa_pair|question.*answer)" + - type: "output_matches" + pattern: "(read.only|non.destructive|deterministic|verifiable)" + rubric: + - "Shows the XML evaluation format with qa_pair elements" + - "Explains that questions should require multiple tool calls and reasoning" + - "Emphasizes that questions must be read-only with deterministic answers" + - "Provides example evaluation questions appropriate for a product catalog" + timeout: 120 From 67b0842e72f4385beb4e726504c8ee25b742c092 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Tue, 10 Mar 2026 11:40:28 -0700 Subject: [PATCH 02/29] Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax Replace scaffolding-heavy scenarios with implementation-focused ones that test MCP-specific features (resources, prompts, logging). Fix assertion patterns to match combined C# attribute syntax [McpServerTool, Description()] instead of requiring standalone [McpServerTool]. Increase timeouts to 180s to account for skill-reading overhead. Validator result: passed=True, improvement=44.6% (threshold=10%) --- tests/dotnet/mcp-csharp-create/eval.yaml | 81 ++++++++++++------------ 1 file changed, 41 insertions(+), 40 deletions(-) diff --git a/tests/dotnet/mcp-csharp-create/eval.yaml b/tests/dotnet/mcp-csharp-create/eval.yaml index 214d4268da..1ce9f29cac 100644 --- a/tests/dotnet/mcp-csharp-create/eval.yaml +++ b/tests/dotnet/mcp-csharp-create/eval.yaml @@ -1,56 +1,57 @@ scenarios: - - name: "Create a new stdio MCP server from scratch" + - name: "Implement MCP tools with proper attributes and DI" prompt: | - I want to create a new C# MCP server that provides tools for interacting - with a REST API for managing TODO items. It should use stdio transport. - Help me scaffold the project and implement a basic tool. + I have a new C# MCP server project. I need to implement a tool class that + wraps a REST API using HttpClient. The tools should follow MCP SDK conventions + with proper attributes for LLM discovery. Show me the tool class, Program.cs + with stdio transport, and explain how DI works for MCP tools. assertions: - - type: "output_contains" - value: "dotnet new mcpserver" - - type: "output_contains" - value: "[McpServerTool]" - type: "output_matches" - pattern: "(WithStdioServerTransport|stdio)" + pattern: "\\[McpServerToolType\\]" + - type: "output_matches" + pattern: "\\[McpServerTool[\\],\\(]" + - type: "output_matches" + pattern: "(AddHttpClient|HttpClient)" rubric: - - "Checks for or installs the mcpserver template before scaffolding" - - "Uses dotnet new mcpserver to scaffold the project" - - "Shows a tool method with [McpServerToolType] and [McpServerTool] attributes" - - "Includes [Description] attributes on the method and parameters" - - "Configures stdio transport in Program.cs" - timeout: 240 - - name: "Create an HTTP MCP server with ASP.NET Core" + - "Shows a tool class with [McpServerToolType] and [McpServerTool] attributes" + - "Injects HttpClient via DI (constructor injection or method parameter injection)" + - "Includes [Description] attributes on both the method and all parameters" + - "Configures Program.cs with AddMcpServer, WithStdioServerTransport, and logging to stderr" + timeout: 180 + - name: "Create an HTTP MCP server with tools and resources" prompt: | - I need to create a C# MCP server that uses HTTP transport so it can be - deployed as a web service. It should have a tool that queries a database. - How do I set this up? + I need to create a C# MCP server that uses HTTP transport for deployment + as a web service. It should expose both tools and resources (for example, + a resource that returns configuration data). Show me how to set up the + HTTP transport with MapMcp and implement a resource. assertions: - type: "output_contains" value: "MapMcp" - type: "output_matches" - pattern: "(WithHttpTransport|--transport remote|--transport http)" - - type: "output_contains" - value: "ModelContextProtocol.AspNetCore" + pattern: "(WithHttpTransport|ModelContextProtocol\\.AspNetCore)" + - type: "output_matches" + pattern: "(McpServerResource|McpServerResourceType)" rubric: - - "Scaffolds with dotnet new mcpserver --transport remote or configures HTTP manually" + - "Configures HTTP transport with WithHttpTransport() and references ModelContextProtocol.AspNetCore" - "Includes MapMcp() in the endpoint configuration" - - "References ModelContextProtocol.AspNetCore package" - - "Shows how to register tools with the DI container" - timeout: 300 - - name: "Add tools with dependency injection to an existing MCP server" + - "Shows a resource class with [McpServerResourceType] and [McpServerResource] attributes including UriTemplate" + - "Shows how to register tools and resources with WithToolsFromAssembly or similar DI registration" + timeout: 180 + - name: "Create an MCP server with tools, prompts, and proper logging" prompt: | - I have an existing C# MCP server project. I want to add a new tool class - that needs an HttpClient and ILogger injected. How do I set up the DI - and implement the tool? + I'm building a C# MCP server using stdio transport. I need to add a tool + that calls an external API using HttpClient (injected via DI), and a prompt + template for summarization. Make sure logging doesn't interfere with the + stdio JSON-RPC protocol. assertions: - - type: "output_contains" - value: "[McpServerToolType]" - - type: "output_contains" - value: "[McpServerTool]" - type: "output_matches" - pattern: "(AddHttpClient|HttpClient)" + pattern: "\\[McpServerTool[\\]T,\\(]" + - type: "output_matches" + pattern: "(McpServerPrompt|McpServerPromptType)" + - type: "output_matches" + pattern: "(LogToStandardErrorThreshold|stderr)" rubric: - - "Shows a tool class with constructor injection of HttpClient and ILogger" - - "Registers HttpClient in the DI container using AddHttpClient or similar" - - "Uses [McpServerToolType] on the class and [McpServerTool] on methods" - - "Adds [Description] attributes for LLM discoverability" - timeout: 120 + - "Shows a tool class with [McpServerToolType]/[McpServerTool] and HttpClient injected via DI" + - "Shows a prompt class with [McpServerPromptType] and [McpServerPrompt] returning ChatMessage" + - "Configures logging to stderr (LogToStandardErrorThreshold) to avoid corrupting stdio transport" + - "Adds [Description] attributes on tools, parameters, and prompts for LLM discoverability" From c0bd6b652b3b7729b1272caf95c00e06f9fce341 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Tue, 10 Mar 2026 12:00:08 -0700 Subject: [PATCH 03/29] Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test) --- .github/CODEOWNERS | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index b41ecd2482..49d8f8a8f9 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -50,3 +50,15 @@ /tests/dotnet/dotnet-aot-compat/ @agocke @dotnet/appmodel /plugins/dotnet/agents/optimizing-dotnet-performance.agent.md @dotnet/appmodel + +/plugins/dotnet/skills/mcp-csharp-create/ @leslierichardson95 +/tests/dotnet/mcp-csharp-create/ @leslierichardson95 + +/plugins/dotnet/skills/mcp-csharp-debug/ @leslierichardson95 +/tests/dotnet/mcp-csharp-debug/ @leslierichardson95 + +/plugins/dotnet/skills/mcp-csharp-publish/ @leslierichardson95 +/tests/dotnet/mcp-csharp-publish/ @leslierichardson95 + +/plugins/dotnet/skills/mcp-csharp-test/ @leslierichardson95 +/tests/dotnet/mcp-csharp-test/ @leslierichardson95 From e1afc1806be3f229f6feec6fa4d834beb2246fd5 Mon Sep 17 00:00:00 2001 From: leslierichardson95 Date: Tue, 10 Mar 2026 12:02:57 -0700 Subject: [PATCH 04/29] Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../dotnet/skills/mcp-csharp-test/references/test-patterns.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md b/plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md index f373a3c9c6..2f3816ccf4 100644 --- a/plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md +++ b/plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md @@ -100,7 +100,7 @@ public class HttpServerTests : IClassFixture> } }; - var response = await client.PostAsJsonAsync("/", request); + var response = await client.PostAsJsonAsync("/mcp", request); response.EnsureSuccessStatusCode(); } From 45423dc24f71130ce3bb064d6fc94e3d5be5539e Mon Sep 17 00:00:00 2001 From: leslierichardson95 Date: Tue, 10 Mar 2026 12:03:28 -0700 Subject: [PATCH 05/29] Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- plugins/dotnet/skills/mcp-csharp-debug/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md index 6037d51a07..8c8934ee2c 100644 --- a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md +++ b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md @@ -136,10 +136,10 @@ npx @modelcontextprotocol/inspector dotnet run --project input; + public string DoSomething(string input) => input; ``` 4. **Verify tool registration in Program.cs** — use one of: From a3636d2f425d2a044a295a78412c3872c2b7a738 Mon Sep 17 00:00:00 2001 From: leslierichardson95 Date: Tue, 10 Mar 2026 12:06:57 -0700 Subject: [PATCH 06/29] Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- plugins/dotnet/skills/mcp-csharp-publish/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md b/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md index bcc73b12fe..da7ac413a5 100644 --- a/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md +++ b/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md @@ -166,7 +166,7 @@ az containerapp create \ --ingress external \ --min-replicas 0 \ --max-replicas 10 \ - --secrets api-key=secretref:api-key \ + --secrets api-key=my-actual-api-key \ --env-vars API_KEY=secretref:api-key ``` From 9966e31535b3ef1a70c4e109ccf82feed0c682f8 Mon Sep 17 00:00:00 2001 From: leslierichardson95 Date: Tue, 10 Mar 2026 12:10:19 -0700 Subject: [PATCH 07/29] Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../skills/mcp-csharp-publish/references/nuget-packaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md b/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md index ce2c79f2fe..9265bbdf01 100644 --- a/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md +++ b/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md @@ -94,7 +94,7 @@ After publishing, users configure their MCP client to run the tool: } ``` -The `dnx` tool runner (`dotnet tool run` shorthand) downloads and runs the tool automatically. +The `dnx` tool runner (a `dotnet execute`-style runner for NuGet packages) downloads and runs the package automatically. For more details, see the .NET package execution docs: https://learn.microsoft.com/dotnet/core/tools/dotnet-execute ## server.json for MCP Registry Integration From afcc4567fec3941c63885b1a53db198eb8ed84b7 Mon Sep 17 00:00:00 2001 From: leslierichardson95 Date: Tue, 10 Mar 2026 12:16:03 -0700 Subject: [PATCH 08/29] Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- plugins/dotnet/skills/mcp-csharp-debug/SKILL.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md index 8c8934ee2c..29d267ce44 100644 --- a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md +++ b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md @@ -136,7 +136,9 @@ npx @modelcontextprotocol/inspector dotnet run --project ()` or `WithToolsFromAssembly()`) so DI can construct it. ```csharp [McpServerTool, Description("Does something")] public string DoSomething(string input) => input; From e96bf6cf7ec8d6dd5b9906b531f837b21f25be85 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Tue, 10 Mar 2026 12:26:49 -0700 Subject: [PATCH 09/29] Remove evaluation step and guide from mcp-csharp-test skill --- .../dotnet/skills/mcp-csharp-test/SKILL.md | 36 ++------ .../references/evaluation-guide.md | 90 ------------------- 2 files changed, 5 insertions(+), 121 deletions(-) delete mode 100644 plugins/dotnet/skills/mcp-csharp-test/references/evaluation-guide.md diff --git a/plugins/dotnet/skills/mcp-csharp-test/SKILL.md b/plugins/dotnet/skills/mcp-csharp-test/SKILL.md index 9c332ae95f..47798f141b 100644 --- a/plugins/dotnet/skills/mcp-csharp-test/SKILL.md +++ b/plugins/dotnet/skills/mcp-csharp-test/SKILL.md @@ -1,10 +1,10 @@ --- name: mcp-csharp-test description: > - Test C# MCP servers at multiple levels: unit tests for individual tools, integration - tests using the MCP client SDK, and LLM effectiveness evaluations. + Test C# MCP servers at multiple levels: unit tests for individual tools and integration + tests using the MCP client SDK. USE FOR: unit testing MCP tool methods, integration testing with in-memory MCP - client/server, end-to-end testing via MCP protocol, creating LLM evaluation question sets, + client/server, end-to-end testing via MCP protocol, testing HTTP MCP servers with WebApplicationFactory, mocking dependencies in tool tests. DO NOT USE FOR: testing MCP clients (this is server testing only), load or performance testing, testing non-.NET MCP servers, debugging server issues (use mcp-csharp-debug). @@ -12,14 +12,13 @@ description: > # C# MCP Server Testing -Test MCP servers at three levels: unit tests for individual tool methods, integration tests that exercise the full MCP protocol in-memory, and evaluations that measure LLM effectiveness when using your tools. +Test MCP servers at two levels: unit tests for individual tool methods, and integration tests that exercise the full MCP protocol in-memory. ## When to Use - Adding automated tests to an MCP server - Testing individual tool methods with mocked dependencies - Writing integration tests that validate tool listing and invocation via MCP protocol -- Creating evaluation question sets to measure LLM effectiveness with your tools - Setting up CI test pipelines for MCP servers ## Stop Signals @@ -134,30 +133,7 @@ public class ServerIntegrationTests : IAsyncLifetime **For the SDK's `ClientServerTestBase` (in-memory testing) and HTTP testing with `WebApplicationFactory`**, see [references/test-patterns.md](references/test-patterns.md). -### Step 4: Create evaluations (optional) - -Evaluations measure how effectively LLMs use your tools to accomplish tasks. Create questions that are independent, read-only, require multiple tool calls, and have verifiable answers. - -```xml - - - - MyMcpServer - 1.0.0 - - - Using the search tool, find users in "engineering" who joined - after 2024. What domain is most common in their emails? - company.com - medium - search_users, list_teams - - -``` - -**For evaluation guidelines, good/bad question examples, and the full XML schema**, see [references/evaluation-guide.md](references/evaluation-guide.md). - -### Step 5: Run tests +### Step 4: Run tests ```bash # Run all tests @@ -186,7 +162,6 @@ dotnet test --collect:"XPlat Code Coverage" | `StdioClientTransport` not finding project | Use the correct relative path to `.csproj` from the test project directory | | Tests pass locally but fail in CI | Run `dotnet build` before test execution. Use `--no-build` only after an explicit build step | | Mocking `HttpClient` is awkward | Mock `HttpMessageHandler`, not `HttpClient` directly. See [references/test-patterns.md](references/test-patterns.md) | -| Evaluation answers change over time | Use stable, deterministic data. Avoid questions about "latest" or time-dependent values | | Full test suite runs are slow | Use `--filter` for development. Run the full suite only for CI verification | ## Related Skills @@ -198,7 +173,6 @@ dotnet test --collect:"XPlat Code Coverage" ## Reference Files - [references/test-patterns.md](references/test-patterns.md) — Complete test code examples: `ClientServerTestBase` in-memory pattern, `WebApplicationFactory` for HTTP, `MockHttpMessageHandler` helper, test categorization, coverage reporting. **Load when:** writing integration tests or need detailed mock patterns. -- [references/evaluation-guide.md](references/evaluation-guide.md) — Evaluation creation guidelines, XML format reference, good/bad question examples, criteria for verifiable evaluations. **Load when:** creating LLM effectiveness evaluations. ## More Info diff --git a/plugins/dotnet/skills/mcp-csharp-test/references/evaluation-guide.md b/plugins/dotnet/skills/mcp-csharp-test/references/evaluation-guide.md deleted file mode 100644 index 767efb185b..0000000000 --- a/plugins/dotnet/skills/mcp-csharp-test/references/evaluation-guide.md +++ /dev/null @@ -1,90 +0,0 @@ -# Evaluation Guide - -Create evaluations that measure how effectively LLMs use your MCP server tools. - -## Evaluation Criteria - -Every evaluation question should be: - -| Criterion | Meaning | -|-----------|---------| -| **Independent** | Not dependent on other questions or prior state | -| **Read-only** | Uses only non-destructive operations | -| **Complex** | Requires multiple tool calls and reasoning | -| **Realistic** | Based on real use cases for the server | -| **Verifiable** | Has a single, clear, deterministic answer | -| **Stable** | Answer won't change over time | - -## XML Format - -```xml - - - - MyMcpServer - 1.0.0 - 2026-03-01 - - - - - Using the user search tool, find all users in the "engineering" team - who joined after 2024. What is the email domain most commonly used - by these users? - - company.com - medium - search_users, list_teams - - - - - Find the project with the most active contributors in the last month. - What is the project's internal ID? - - proj-42 - hard - list_projects, get_project_stats - - - - -``` - -## Creating Good Questions - -### Good question patterns - -- **Multi-step retrieval**: "Find X, then use that to look up Y" — requires chaining tool calls -- **Aggregation**: "Which category has the most items matching criteria Z?" — requires gathering and comparing -- **Cross-referencing**: "Find the overlap between set A and set B" — uses multiple tools - -### Examples - -**Good:** -- "Find the user who created the most issues in 'backend' this year. What is their username?" -- "Which team has the highest average code review turnaround time? Return the team name." -- "How many open issues are labeled 'critical' and assigned to users in the 'security' team?" - -**Bad:** -- "What is 2 + 2?" — doesn't use tools -- "List all users" — too simple, single tool call, no reasoning -- "Create a new issue…" — not read-only, mutates state -- "What's the current weather?" — answer changes over time -- "Tell me about the system" — no verifiable answer - -## Evaluation Process - -1. **Explore available data** — use read-only tools to understand what data exists -2. **Draft questions** that require reasoning across multiple tool calls -3. **Solve each question manually** to confirm the answer is correct and deterministic -4. **Verify stability** — run the evaluation twice to ensure answers are consistent -5. **Aim for 10 questions** with a mix of medium and hard difficulty - -## Difficulty Guidelines - -| Difficulty | Tool Calls | Description | -|------------|-----------|-------------| -| Easy | 1 | Single tool, direct answer | -| Medium | 2-3 | Chain tools, simple reasoning | -| Hard | 4+ | Multiple tools, aggregation, cross-referencing | From 78ccf19a8769c53b4b32d3712213e0b8b3f2d9e0 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Tue, 10 Mar 2026 12:47:31 -0700 Subject: [PATCH 10/29] Add CODEOWNERS entries for dotnet-maui skills --- .github/CODEOWNERS | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 49d8f8a8f9..0e89b47598 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -62,3 +62,7 @@ /plugins/dotnet/skills/mcp-csharp-test/ @leslierichardson95 /tests/dotnet/mcp-csharp-test/ @leslierichardson95 + +# dotnet-maui +/plugins/dotnet-maui/ @Redth @jfversluis +/tests/dotnet-maui/ @Redth @jfversluis \ No newline at end of file From 29e31d4659da4d74053e296f1a9798f1a19c12de Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Tue, 10 Mar 2026 13:55:06 -0700 Subject: [PATCH 11/29] Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills --- .github/CODEOWNERS | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index cb010b0255..7cdc19a5cd 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -57,17 +57,17 @@ /plugins/dotnet/agents/optimizing-dotnet-performance.agent.md @dotnet/appmodel -/plugins/dotnet/skills/mcp-csharp-create/ @leslierichardson95 -/tests/dotnet/mcp-csharp-create/ @leslierichardson95 +/plugins/dotnet/skills/mcp-csharp-create/ @leslierichardson95 @artl93 +/tests/dotnet/mcp-csharp-create/ @leslierichardson95 @artl93 -/plugins/dotnet/skills/mcp-csharp-debug/ @leslierichardson95 -/tests/dotnet/mcp-csharp-debug/ @leslierichardson95 +/plugins/dotnet/skills/mcp-csharp-debug/ @leslierichardson95 @artl93 +/tests/dotnet/mcp-csharp-debug/ @leslierichardson95 @artl93 -/plugins/dotnet/skills/mcp-csharp-publish/ @leslierichardson95 -/tests/dotnet/mcp-csharp-publish/ @leslierichardson95 +/plugins/dotnet/skills/mcp-csharp-publish/ @leslierichardson95 @artl93 +/tests/dotnet/mcp-csharp-publish/ @leslierichardson95 @artl93 -/plugins/dotnet/skills/mcp-csharp-test/ @leslierichardson95 -/tests/dotnet/mcp-csharp-test/ @leslierichardson95 +/plugins/dotnet/skills/mcp-csharp-test/ @leslierichardson95 @artl93 +/tests/dotnet/mcp-csharp-test/ @leslierichardson95 @artl93 # dotnet-maui /plugins/dotnet-maui/ @Redth @jfversluis From 0258ab5e995eae804fcb1777fdea1413db2630eb Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Thu, 12 Mar 2026 16:02:14 -0700 Subject: [PATCH 12/29] Replace parent-directory file references with backtick skill names Cross-skill references used ../sibling/SKILL.md paths which the validator rejects. Replace with backtick-quoted skill names to match the convention used by other skills in the repo. --- plugins/dotnet/skills/mcp-csharp-create/SKILL.md | 12 ++++++------ plugins/dotnet/skills/mcp-csharp-debug/SKILL.md | 12 ++++++------ plugins/dotnet/skills/mcp-csharp-publish/SKILL.md | 14 +++++++------- .../references/nuget-packaging.md | 2 +- plugins/dotnet/skills/mcp-csharp-test/SKILL.md | 12 ++++++------ 5 files changed, 26 insertions(+), 26 deletions(-) diff --git a/plugins/dotnet/skills/mcp-csharp-create/SKILL.md b/plugins/dotnet/skills/mcp-csharp-create/SKILL.md index 295ff34ad3..8c471c935c 100644 --- a/plugins/dotnet/skills/mcp-csharp-create/SKILL.md +++ b/plugins/dotnet/skills/mcp-csharp-create/SKILL.md @@ -25,9 +25,9 @@ Create Model Context Protocol servers using the official C# SDK (`ModelContextPr ## Stop Signals -- **Server already exists and needs debugging?** → Use [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) -- **Need tests or evaluations?** → Use [mcp-csharp-test](../mcp-csharp-test/SKILL.md) -- **Ready to publish?** → Use [mcp-csharp-publish](../mcp-csharp-publish/SKILL.md) +- **Server already exists and needs debugging?** → Use `mcp-csharp-debug` +- **Need tests or evaluations?** → Use `mcp-csharp-test` +- **Ready to publish?** → Use `mcp-csharp-publish` - **Building an MCP client, not a server** → This skill is server-side only ## Inputs @@ -249,9 +249,9 @@ For HTTP: the server listens on the configured port. ## Related Skills -- [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) — Run, debug, and test with MCP Inspector -- [mcp-csharp-test](../mcp-csharp-test/SKILL.md) — Unit tests, integration tests, evaluations -- [mcp-csharp-publish](../mcp-csharp-publish/SKILL.md) — NuGet, Docker, Azure deployment +- `mcp-csharp-debug` — Run, debug, and test with MCP Inspector +- `mcp-csharp-test` — Unit tests, integration tests, evaluations +- `mcp-csharp-publish` — NuGet, Docker, Azure deployment ## Reference Files diff --git a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md index 29d267ce44..0b6e806f44 100644 --- a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md +++ b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md @@ -26,9 +26,9 @@ Run, debug, and interactively test C# MCP servers. Covers local execution, IDE d ## Stop Signals -- **No project yet?** → Use [mcp-csharp-create](../mcp-csharp-create/SKILL.md) first -- **Need automated tests?** → Use [mcp-csharp-test](../mcp-csharp-test/SKILL.md) -- **Production deployment issue?** → Use [mcp-csharp-publish](../mcp-csharp-publish/SKILL.md) +- **No project yet?** → Use `mcp-csharp-create` first +- **Need automated tests?** → Use `mcp-csharp-test` +- **Production deployment issue?** → Use `mcp-csharp-publish` ## Inputs @@ -221,9 +221,9 @@ public class MyTools(ILogger logger) ## Related Skills -- [mcp-csharp-create](../mcp-csharp-create/SKILL.md) — Create a new MCP server project -- [mcp-csharp-test](../mcp-csharp-test/SKILL.md) — Automated tests and evaluations -- [mcp-csharp-publish](../mcp-csharp-publish/SKILL.md) — NuGet, Docker, Azure deployment +- `mcp-csharp-create` — Create a new MCP server project +- `mcp-csharp-test` — Automated tests and evaluations +- `mcp-csharp-publish` — NuGet, Docker, Azure deployment ## Reference Files diff --git a/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md b/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md index da7ac413a5..6b7aa92846 100644 --- a/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md +++ b/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md @@ -27,10 +27,10 @@ Publish and deploy MCP servers to their target platforms. stdio servers are dist ## Stop Signals -- **Server not tested yet?** → Use [mcp-csharp-test](../mcp-csharp-test/SKILL.md) first -- **Server not working locally?** → Use [mcp-csharp-debug](../mcp-csharp-debug/SKILL.md) -- **No server project yet?** → Use [mcp-csharp-create](../mcp-csharp-create/SKILL.md) -- **Publishing a non-MCP NuGet package?** → Use [nuget-trusted-publishing](../nuget-trusted-publishing/SKILL.md) instead +- **Server not tested yet?** → Use `mcp-csharp-test` first +- **Server not working locally?** → Use `mcp-csharp-debug` +- **No server project yet?** → Use `mcp-csharp-create` +- **Publishing a non-MCP NuGet package?** → Use `nuget-trusted-publishing` instead ## Inputs @@ -258,9 +258,9 @@ curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github. Date: Fri, 13 Mar 2026 13:41:20 -0600 Subject: [PATCH 13/29] Add known domains for MCP server development skills Add domains referenced by the mcp-csharp-* skills: - github.com/modelcontextprotocol/registry - github.com/open-telemetry/semantic-conventions - npmjs.com, xunit.net, fluentassertions.com, nugettest.org Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- eng/reference-scanner/known-domains.txt | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/eng/reference-scanner/known-domains.txt b/eng/reference-scanner/known-domains.txt index 4238069686..2d497d2d4b 100644 --- a/eng/reference-scanner/known-domains.txt +++ b/eng/reference-scanner/known-domains.txt @@ -21,6 +21,7 @@ download.sysinternals.com mcr.microsoft.com msdl.microsoft.com nuget.org +nugettest.org dotnetcli.blob.core.windows.net # Platforms @@ -50,10 +51,15 @@ github.com/microsoft/openjdk github.com/microsoft/perfview github.com/microsoftdocs/visualstudio-docs github.com/modelcontextprotocol/csharp-sdk +github.com/modelcontextprotocol/registry +github.com/open-telemetry/semantic-conventions # Community +fluentassertions.com +npmjs.com/package/@modelcontextprotocol ollama.com stackoverflow.com +xunit.net # UI helpers speedscope.app From e2c69205221c6709cf8fbfe325e5b49026b0b16f Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Wed, 18 Mar 2026 14:11:56 -0700 Subject: [PATCH 14/29] Update Docker commands to use placeholder for registry and image names --- plugins/dotnet/skills/mcp-csharp-publish/SKILL.md | 12 ++++++------ .../mcp-csharp-publish/references/docker-azure.md | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md b/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md index 6b7aa92846..3f53a82040 100644 --- a/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md +++ b/plugins/dotnet/skills/mcp-csharp-publish/SKILL.md @@ -144,13 +144,13 @@ curl http://localhost:3001/health 3. **Push to container registry:** ```bash # Docker Hub -docker tag mymcpserver:latest yourusername/mymcpserver:1.0.0 -docker push yourusername/mymcpserver:1.0.0 +docker tag mymcpserver:latest /mymcpserver:1.0.0 +docker push /:1.0.0 # Azure Container Registry az acr login --name yourregistry -docker tag mymcpserver:latest yourregistry.azurecr.io/mymcpserver:1.0.0 -docker push yourregistry.azurecr.io/mymcpserver:1.0.0 +docker tag mymcpserver:latest .azurecr.io/:1.0.0 +docker push .azurecr.io/:1.0.0 ``` ### Step 3: Deploy to Azure (HTTP servers) @@ -161,7 +161,7 @@ az containerapp create \ --name mymcpserver \ --resource-group mygroup \ --environment myenvironment \ - --image yourregistry.azurecr.io/mymcpserver:1.0.0 \ + --image .azurecr.io/:1.0.0 \ --target-port 8080 \ --ingress external \ --min-replicas 0 \ @@ -176,7 +176,7 @@ az webapp create \ --name mymcpserver \ --resource-group mygroup \ --plan myplan \ - --deployment-container-image-name yourregistry.azurecr.io/mymcpserver:1.0.0 + --deployment-container-image-name .azurecr.io/:1.0.0 ``` **For detailed Azure deployment**, see [references/docker-azure.md](references/docker-azure.md). diff --git a/plugins/dotnet/skills/mcp-csharp-publish/references/docker-azure.md b/plugins/dotnet/skills/mcp-csharp-publish/references/docker-azure.md index 7afa4a231f..d7bc172f12 100644 --- a/plugins/dotnet/skills/mcp-csharp-publish/references/docker-azure.md +++ b/plugins/dotnet/skills/mcp-csharp-publish/references/docker-azure.md @@ -55,8 +55,8 @@ az acr create --name yourregistry --resource-group mygroup --sku Basic az acr login --name yourregistry # Build and push -docker build -t yourregistry.azurecr.io/mymcpserver:1.0.0 . -docker push yourregistry.azurecr.io/mymcpserver:1.0.0 +docker build -t .azurecr.io/:1.0.0 . +docker push .azurecr.io/:1.0.0 # Or build directly in ACR (no local Docker needed) az acr build --registry yourregistry --image mymcpserver:1.0.0 . @@ -80,8 +80,8 @@ az containerapp create \ --name mymcpserver \ --resource-group mygroup \ --environment myenvironment \ - --image yourregistry.azurecr.io/mymcpserver:1.0.0 \ - --registry-server yourregistry.azurecr.io \ + --image .azurecr.io/:1.0.0 \ + --registry-server .azurecr.io \ --target-port 8080 \ --ingress external \ --min-replicas 0 \ @@ -102,7 +102,7 @@ az containerapp show --name mymcpserver --resource-group mygroup \ az containerapp update \ --name mymcpserver \ --resource-group mygroup \ - --image yourregistry.azurecr.io/mymcpserver:1.1.0 + --image .azurecr.io/:1.1.0 ``` ### Scaling Configuration @@ -134,7 +134,7 @@ az webapp create \ --name mymcpserver \ --resource-group mygroup \ --plan myplan \ - --deployment-container-image-name yourregistry.azurecr.io/mymcpserver:1.0.0 + --deployment-container-image-name .azurecr.io/:1.0.0 # Configure secrets via Key Vault az webapp config appsettings set \ From 61aff8735addf1810c97849f83dfb4e0130e189a Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Fri, 20 Mar 2026 12:58:11 -0700 Subject: [PATCH 15/29] moved mcp skills and their evals into dotnet-ai plugin --- .../shared/compiled/build-errors.lock.md | 2 +- .../shared/compiled/performance.lock.md | 4 ++-- .../shared/compiled/style-and-modernization.lock.md | 12 ++++-------- .../skills/mcp-csharp-create/SKILL.md | 0 .../mcp-csharp-create/references/api-patterns.md | 0 .../mcp-csharp-create/references/transport-config.md | 0 .../skills/mcp-csharp-debug/SKILL.md | 0 .../skills/mcp-csharp-debug/references/ide-config.md | 0 .../mcp-csharp-debug/references/mcp-inspector.md | 0 .../skills/mcp-csharp-publish/SKILL.md | 0 .../mcp-csharp-publish/references/docker-azure.md | 0 .../mcp-csharp-publish/references/mcp-registry.md | 0 .../mcp-csharp-publish/references/nuget-packaging.md | 0 .../skills/mcp-csharp-test/SKILL.md | 0 .../mcp-csharp-test/references/test-patterns.md | 0 .../mcp-csharp-create/eval.yaml | 0 .../{dotnet => dotnet-ai}/mcp-csharp-debug/eval.yaml | 0 .../mcp-csharp-publish/eval.yaml | 0 .../{dotnet => dotnet-ai}/mcp-csharp-test/eval.yaml | 0 19 files changed, 7 insertions(+), 11 deletions(-) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-create/SKILL.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-create/references/api-patterns.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-create/references/transport-config.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-debug/SKILL.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-debug/references/ide-config.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-debug/references/mcp-inspector.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-publish/SKILL.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-publish/references/docker-azure.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-publish/references/mcp-registry.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-publish/references/nuget-packaging.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-test/SKILL.md (100%) rename plugins/{dotnet => dotnet-ai}/skills/mcp-csharp-test/references/test-patterns.md (100%) rename tests/{dotnet => dotnet-ai}/mcp-csharp-create/eval.yaml (100%) rename tests/{dotnet => dotnet-ai}/mcp-csharp-debug/eval.yaml (100%) rename tests/{dotnet => dotnet-ai}/mcp-csharp-publish/eval.yaml (100%) rename tests/{dotnet => dotnet-ai}/mcp-csharp-test/eval.yaml (100%) diff --git a/agentic-workflows/dotnet-msbuild/shared/compiled/build-errors.lock.md b/agentic-workflows/dotnet-msbuild/shared/compiled/build-errors.lock.md index 44764e5f5b..18ade40c65 100644 --- a/agentic-workflows/dotnet-msbuild/shared/compiled/build-errors.lock.md +++ b/agentic-workflows/dotnet-msbuild/shared/compiled/build-errors.lock.md @@ -1,4 +1,4 @@ - + # Analyzing MSBuild Failures with Binary Logs diff --git a/agentic-workflows/dotnet-msbuild/shared/compiled/performance.lock.md b/agentic-workflows/dotnet-msbuild/shared/compiled/performance.lock.md index 0fba30cecf..10eaa47783 100644 --- a/agentic-workflows/dotnet-msbuild/shared/compiled/performance.lock.md +++ b/agentic-workflows/dotnet-msbuild/shared/compiled/performance.lock.md @@ -1,4 +1,4 @@ - + # Build Performance Baseline & Optimization @@ -401,7 +401,7 @@ Is your no-op build slow (> 10s per project)? │ Is compilation slow? │ ├── YES │ │ Are analyzers/generators slow? - │ │ ├── YES → See `analyzer-performance` skill + │ │ ├── YES → See `build-perf-diagnostics` skill │ │ └── NO → Check parallelism, graph build, critical path (this skill + `build-parallelism`) │ └── NO → Check custom targets (binlog analysis via `build-perf-diagnostics`) └── NO diff --git a/agentic-workflows/dotnet-msbuild/shared/compiled/style-and-modernization.lock.md b/agentic-workflows/dotnet-msbuild/shared/compiled/style-and-modernization.lock.md index 6253533276..de1d425429 100644 --- a/agentic-workflows/dotnet-msbuild/shared/compiled/style-and-modernization.lock.md +++ b/agentic-workflows/dotnet-msbuild/shared/compiled/style-and-modernization.lock.md @@ -1,4 +1,4 @@ - + # MSBuild Anti-Pattern Catalog @@ -431,7 +431,7 @@ See `incremental-build` skill for deep guidance on Inputs/Outputs, FileWrites, a --- -## AP-16: Using `` for String/Path Operations +For additional anti-patterns (AP-16 through AP-21) and a quick-reference checklist, see ## AP-16: Using `` for String/Path Operations **Smell**: `` or `` for simple string manipulation. @@ -630,7 +630,7 @@ When reviewing an MSBuild file, scan for these in order: | AP-16 | `` for string operations | 🔵 Preference | | AP-17 | Mixed Include/Update in one ItemGroup | 🔵 Subtle bugs | | AP-18 | Redundant transitive ProjectReferences | 🔵 Graph noise | -| AP-20 | Platform-specific Exec without guard | 🔵 Cross-platform | +| AP-20 | Platform-specific Exec without guard | 🔵 Cross-platform |. --- @@ -1091,10 +1091,6 @@ Identify properties repeated across multiple `.csproj` files and move them to sh ``` -**`Directory.Build.targets`** (for targets/tasks — placed at repo or src root): - -```xml - - Date: Fri, 20 Mar 2026 13:02:19 -0700 Subject: [PATCH 16/29] cleaned up moved files to dotnet-ai --- .../dotnet/skills/mcp-csharp-create/SKILL.md | 265 ++++++++++++++++++ .../dotnet/skills/mcp-csharp-debug/SKILL.md | 236 ++++++++++++++++ .../references/nuget-packaging.md | 144 ++++++++++ .../dotnet/skills/mcp-csharp-test/SKILL.md | 180 ++++++++++++ tests/dotnet/mcp-csharp-create/eval.yaml | 57 ++++ 5 files changed, 882 insertions(+) create mode 100644 plugins/dotnet/skills/mcp-csharp-create/SKILL.md create mode 100644 plugins/dotnet/skills/mcp-csharp-debug/SKILL.md create mode 100644 plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md create mode 100644 plugins/dotnet/skills/mcp-csharp-test/SKILL.md create mode 100644 tests/dotnet/mcp-csharp-create/eval.yaml diff --git a/plugins/dotnet/skills/mcp-csharp-create/SKILL.md b/plugins/dotnet/skills/mcp-csharp-create/SKILL.md new file mode 100644 index 0000000000..8c471c935c --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-create/SKILL.md @@ -0,0 +1,265 @@ +--- +name: mcp-csharp-create +description: > + Create MCP servers using the C# SDK and .NET project templates. Covers scaffolding, + tool/prompt/resource implementation, and transport configuration for stdio and HTTP. + USE FOR: creating new MCP server projects, scaffolding with dotnet new mcpserver, adding + MCP tools/prompts/resources, choosing stdio vs HTTP transport, configuring MCP hosting in + Program.cs, setting up ASP.NET Core MCP endpoints with MapMcp. + DO NOT USE FOR: debugging or running existing servers (use mcp-csharp-debug), writing tests + (use mcp-csharp-test), publishing or deploying (use mcp-csharp-publish), building MCP + clients, non-.NET MCP servers. +--- + +# C# MCP Server Creation + +Create Model Context Protocol servers using the official C# SDK (`ModelContextProtocol` NuGet package) and the `dotnet new mcpserver` project template. Servers expose tools, prompts, and resources that LLMs can discover and invoke via the MCP protocol. + +## When to Use + +- Starting a new MCP server project from scratch +- Adding tools, prompts, or resources to an existing MCP server +- Choosing between stdio (`--transport local`) and HTTP (`--transport remote`) transport +- Setting up ASP.NET Core hosting for an HTTP MCP server +- Wrapping an external API or service as MCP tools + +## Stop Signals + +- **Server already exists and needs debugging?** → Use `mcp-csharp-debug` +- **Need tests or evaluations?** → Use `mcp-csharp-test` +- **Ready to publish?** → Use `mcp-csharp-publish` +- **Building an MCP client, not a server** → This skill is server-side only + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Transport type | Yes | `stdio` (local/CLI) or `http` (remote/web). Ask user if not specified — default to stdio | +| Project name | Yes | PascalCase name for the project (e.g., `WeatherMcpServer`) | +| .NET SDK version | Recommended | .NET 10.0+ required. Check with `dotnet --version` | +| Service/API to wrap | Recommended | External API or service the tools will interact with | + +## Workflow + +> **Commit strategy:** Commit after completing each step so scaffolding and implementation are separately reviewable. + +### Step 1: Verify prerequisites + +1. Confirm .NET 10+ SDK: `dotnet --version` (install from https://dotnet.microsoft.com if < 10.0) + +2. Check if the MCP server template is already installed: + ```bash + dotnet new list mcpserver + ``` + If "No templates found" → install: `dotnet new install Microsoft.McpServer.ProjectTemplates` + +### Step 2: Choose transport + +| Choose **stdio** if… | Choose **HTTP** if… | +|----------------------|---------------------| +| Local CLI tool or IDE plugin | Cloud/web service deployment | +| Single user at a time | Multiple simultaneous clients | +| Running as subprocess (VS Code, GitHub Copilot) | Cross-network access needed | +| Simpler setup, no network config | Containerized deployment (Docker/Azure) | + +**Default:** stdio — simpler, works for most local development. Users can add HTTP later. + +### Step 3: Scaffold the project + +**stdio server:** +```bash +dotnet new mcpserver -n +``` +If the template times out or is unavailable, use `dotnet new console -n ` and add `dotnet add package ModelContextProtocol`. + +**HTTP server:** +```bash +dotnet new web -n +cd +dotnet add package ModelContextProtocol.AspNetCore +``` +This is the recommended approach — faster and more reliable than the template. The template also supports HTTP via `dotnet new mcpserver -n --transport remote`, but `dotnet new web` gives you more control over the project structure. + +**Template flags reference:** `--transport local` (stdio, default), `--transport remote` (ASP.NET Core HTTP), `--aot`, `--self-contained`. + +### Step 4: Implement tools + +Tools are the primary way MCP servers expose functionality. Add a class with `[McpServerToolType]` and methods with `[McpServerTool]`: + +```csharp +using ModelContextProtocol.Server; +using System.ComponentModel; + +[McpServerToolType] +public static class MyTools +{ + [McpServerTool, Description("Brief description of what the tool does.")] + public static async Task DoSomething( + [Description("What this parameter controls")] string input, + CancellationToken cancellationToken = default) + { + // Implementation + return $"Result: {input}"; + } +} +``` + +**Critical rules:** +- Every tool method **must** have a `[Description]` attribute — LLMs use this to decide when to call the tool +- Every parameter **must** have a `[Description]` attribute +- Accept `CancellationToken` in all async tools +- Use `[McpServerTool(Name = "custom_name")]` only if the default method name is unclear + +**DI injection patterns** — the SDK supports two styles: + +1. **Method parameter injection (static class):** DI services appear as method parameters. The SDK resolves them automatically — they do not appear in the tool schema. + +2. **Constructor injection (non-static class):** Use when tools need shared state or multiple services: +```csharp +[McpServerToolType] +public class ApiTools(HttpClient httpClient, ILogger logger) +{ + [McpServerTool, Description("Fetch a resource by ID.")] + public async Task FetchResource( + [Description("Resource identifier")] string id, + CancellationToken cancellationToken = default) + { + logger.LogInformation("Fetching {Id}", id); + return await httpClient.GetStringAsync($"/api/{id}", cancellationToken); + } +} +``` +Register services in Program.cs: +```csharp +var builder = Host.CreateApplicationBuilder(args); +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); + +builder.Services.AddHttpClient(); // registers IHttpClientFactory + HttpClient +// ILogger is registered by default — no extra setup needed. + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithToolsFromAssembly(); // discovers non-static [McpServerToolType] classes + +await builder.Build().RunAsync(); +``` + +**For the full attribute reference, return types, DI injection, and builder API patterns**, see [references/api-patterns.md](references/api-patterns.md). + +### Step 5: Add prompts and resources (optional) + +**Prompts** — reusable LLM interaction templates: +```csharp +[McpServerPromptType] +public static class MyPrompts +{ + [McpServerPrompt, Description("Summarize content into one sentence.")] + public static ChatMessage Summarize( + [Description("Content to summarize")] string content) => + new(ChatRole.User, $"Summarize this into one sentence: {content}"); +} +``` + +**Resources** — data the LLM can read: +```csharp +[McpServerResourceType] +public static class MyResources +{ + [McpServerResource(UriTemplate = "config://app", Name = "App Config", + MimeType = "application/json"), Description("Application configuration")] + public static string GetConfig() => JsonSerializer.Serialize(AppConfig.Current); +} +``` + +### Step 6: Configure Program.cs + +**stdio transport:** +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using ModelContextProtocol.Server; + +var builder = Host.CreateApplicationBuilder(args); +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); // CRITICAL: stderr only + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithToolsFromAssembly(); + +await builder.Build().RunAsync(); +``` + +**HTTP transport:** +```csharp +using ModelContextProtocol.Server; + +var builder = WebApplication.CreateBuilder(args); +builder.Services.AddMcpServer() + .WithHttpTransport() + .WithToolsFromAssembly(); + +// Register services your tools need via DI +// builder.Services.AddHttpClient(); +// builder.Services.AddSingleton(); + +var app = builder.Build(); +app.MapMcp(); // exposes MCP endpoint at /mcp (Streamable HTTP) +app.MapGet("/health", () => "ok"); // health check for container orchestrators +app.Run(); +``` + +**Key HTTP details:** `MapMcp()` defaults to `/mcp` path. For containers, set `ASPNETCORE_URLS=http://+:8080` and `EXPOSE 8080`. The MCP HTTP protocol uses Streamable HTTP — no special client config needed beyond the URL. + +**For transport configuration details** (stateless mode, auth, path prefix, `HttpContextAccessor`), see [references/transport-config.md](references/transport-config.md). + +### Step 7: Verify the server starts + +```bash +cd +dotnet build +dotnet run +``` + +For stdio: the process starts and waits for JSON-RPC input on stdin. +For HTTP: the server listens on the configured port. + +## Validation + +- [ ] Project builds with no errors (`dotnet build`) +- [ ] All tool classes have `[McpServerToolType]` attribute +- [ ] All tool methods have `[McpServerTool]` and `[Description]` attributes +- [ ] All parameters have `[Description]` attributes +- [ ] stdio: logging directed to stderr, not stdout +- [ ] HTTP: `app.MapMcp()` is called in Program.cs +- [ ] Server starts successfully with `dotnet run` + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| stdio server outputs garbage or hangs | Logging to stdout corrupts JSON-RPC protocol. Set `LogToStandardErrorThreshold = LogLevel.Trace` | +| Tool not discovered by LLM clients | Missing `[McpServerToolType]` on the class or `[McpServerTool]` on the method. Verify `.WithToolsFromAssembly()` in Program.cs | +| LLM doesn't understand when to use a tool | Add clear `[Description]` attributes on both the method and all parameters | +| `WithToolsFromAssembly()` fails in AOT | Reflection-based discovery is incompatible with Native AOT. Use `.WithTools()` instead | +| Parameters not appearing in tool schema | `CancellationToken`, `IMcpServer`, and DI services are injected automatically — they do not appear in the schema. Only parameters with `[Description]` are exposed | +| HTTP server returns 404 | `app.MapMcp()` must be called. Check the request path matches the configured route | + +## Related Skills + +- `mcp-csharp-debug` — Run, debug, and test with MCP Inspector +- `mcp-csharp-test` — Unit tests, integration tests, evaluations +- `mcp-csharp-publish` — NuGet, Docker, Azure deployment + +## Reference Files + +- [references/api-patterns.md](references/api-patterns.md) — Complete attribute reference, return types, DI injection, builder API, dynamic tools, experimental APIs. **Load when:** implementing tools, prompts, or resources beyond the basic patterns shown above. +- [references/transport-config.md](references/transport-config.md) — Detailed transport configuration: stateless HTTP mode, OAuth/auth, custom path prefix, `HttpContextAccessor`, OpenTelemetry observability. **Load when:** configuring advanced transport options or authentication. + +## More Info + +- [C# MCP SDK](https://github.com/modelcontextprotocol/csharp-sdk) — Official SDK repository +- [Build an MCP server (.NET)](https://learn.microsoft.com/dotnet/ai/quickstarts/build-mcp-server) — Microsoft quickstart +- [MCP Specification](https://modelcontextprotocol.io/specification/) — Protocol specification diff --git a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md new file mode 100644 index 0000000000..0b6e806f44 --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md @@ -0,0 +1,236 @@ +--- +name: mcp-csharp-debug +description: > + Run and debug C# MCP servers locally. Covers IDE configuration, MCP Inspector testing, + GitHub Copilot Agent Mode integration, logging setup, and troubleshooting. + USE FOR: running MCP servers locally with dotnet run, configuring VS Code or Visual Studio + for MCP debugging, testing tools with MCP Inspector, testing with GitHub Copilot Agent Mode, + diagnosing tool registration issues, setting up mcp.json configuration, debugging MCP + protocol messages, configuring logging for stdio and HTTP servers. + DO NOT USE FOR: creating new MCP servers (use mcp-csharp-create), writing automated tests + (use mcp-csharp-test), publishing or deploying to production (use mcp-csharp-publish). +--- + +# C# MCP Server Debugging + +Run, debug, and interactively test C# MCP servers. Covers local execution, IDE debugging with breakpoints, MCP Inspector for protocol-level testing, and GitHub Copilot Agent Mode integration. + +## When to Use + +- Running an MCP server locally for the first time +- Configuring VS Code or Visual Studio to debug an MCP server +- Testing tools interactively with MCP Inspector +- Verifying tools appear in GitHub Copilot Agent Mode +- Diagnosing issues: tools not discovered, protocol errors, server crashes +- Setting up `mcp.json` or `.mcp.json` configuration + +## Stop Signals + +- **No project yet?** → Use `mcp-csharp-create` first +- **Need automated tests?** → Use `mcp-csharp-test` +- **Production deployment issue?** → Use `mcp-csharp-publish` + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Project path | Yes | Path to the `.csproj` file or project directory | +| Transport type | Recommended | `stdio` or `http` — detect from `.csproj` if not specified | +| IDE | Recommended | VS Code or Visual Studio — detect from environment if not specified | + +**Agent behavior:** Detect transport type by checking the `.csproj` for a `PackageReference` to `ModelContextProtocol.AspNetCore`. If present → HTTP, otherwise → stdio. + +## Workflow + +### Step 1: Run the server locally + +**stdio transport:** +```bash +cd +dotnet run +``` +The process starts and waits for JSON-RPC messages on stdin. No output on stdout means it's working correctly. + +**HTTP transport:** +```bash +cd +dotnet run +# Server listens on http://localhost:3001 (or configured port) +``` + +### Step 2: Generate MCP configuration + +Detect the IDE and transport, then create the appropriate config file. + +**For VS Code** — create `.vscode/mcp.json`: + +stdio: +```json +{ + "servers": { + "": { + "type": "stdio", + "command": "dotnet", + "args": ["run", "--project", ""] + } + } +} +``` + +HTTP: +```json +{ + "servers": { + "": { + "type": "http", + "url": "http://localhost:3001" + } + } +} +``` + +**For Visual Studio** — create `.mcp.json` at solution root (same JSON structure). + +**For detailed IDE-specific configuration** (launch.json, environment variables, secrets), see [references/ide-config.md](references/ide-config.md). + +### Step 3: Test with MCP Inspector + +The MCP Inspector provides a UI for testing tools, viewing schemas, and inspecting protocol messages. + +**stdio server:** +```bash +npx @modelcontextprotocol/inspector dotnet run --project +``` + +**HTTP server:** +1. Start your server: `dotnet run` +2. Run Inspector: `npx @modelcontextprotocol/inspector` +3. Connect to `http://localhost:3001` + +**Inspector capabilities:** +- List all registered tools, prompts, and resources +- Call tools with custom parameters and see results +- View request/response JSON-RPC messages +- Inspect tool schemas and descriptions + +**For detailed Inspector usage and troubleshooting**, see [references/mcp-inspector.md](references/mcp-inspector.md). + +### Step 4: Test with GitHub Copilot Agent Mode + +1. Open GitHub Copilot Chat → switch to **Agent** mode +2. Click **Select Tools** (wrench icon) → verify your server and tools are listed +3. Test with a prompt that should trigger your tool +4. Approve tool execution when prompted + +**If tools don't appear — troubleshoot tool discovery:** + +1. **Rebuild first** — stale builds are the #1 cause: + ```bash + dotnet build + ``` + Then restart the MCP server (click Stop → Start in VS Code, or restart `dotnet run`). + +2. **Check `[McpServerToolType]` on the class:** + ```csharp + [McpServerToolType] // ← Required on the class + public class MyTools { ... } + ``` + +3. **Check `[McpServerTool]` on each tool method:** + - The method must be `public`. + - It can be `static` or instance. For instance methods, ensure the containing type is discoverable/registered (for example via `WithTools()` or `WithToolsFromAssembly()`) so DI can construct it. + ```csharp + [McpServerTool, Description("Does something")] + public string DoSomething(string input) => input; + ``` + +4. **Verify tool registration in Program.cs** — use one of: + ```csharp + .WithTools() // register specific class + .WithToolsFromAssembly() // scan entire assembly for [McpServerToolType] + ``` + +5. **Check `mcp.json`** points to the correct project path + +6. If still not appearing, reference the tool explicitly: `Using #tool_name, do X` + +### Step 5: Set up breakpoint debugging + +1. Set breakpoints in your tool methods +2. Launch with the debugger: + - **VS Code:** F5 (requires `launch.json` — see [references/ide-config.md](references/ide-config.md)) + - **Visual Studio:** F5 or right-click project → Debug → Start +3. Trigger the tool (via Inspector, Copilot, or test client) +4. Execution pauses at breakpoints + +**Critical:** Build in Debug configuration. Breakpoints won't hit in Release builds. + +### Step 6: Configure logging + +**Critical for stdio transport:** Any output to stdout (including `Console.WriteLine`) **corrupts the MCP JSON-RPC protocol** and causes garbled responses or crashes. All logging and diagnostic output must go to stderr. + +**stdio transport** — log to stderr only: +```csharp +builder.Logging.AddConsole(options => + options.LogToStandardErrorThreshold = LogLevel.Trace); +``` + +**HTTP transport** — standard console logging: +```csharp +builder.Logging.ClearProviders(); +builder.Logging.AddConsole(); +builder.Logging.SetMinimumLevel( + builder.Environment.IsDevelopment() ? LogLevel.Debug : LogLevel.Information); +``` + +**In tool methods** — inject `ILogger` via constructor: +```csharp +[McpServerToolType] +public class MyTools(ILogger logger) +{ + [McpServerTool, Description("Processes data")] + public string ProcessData(string input) + { + logger.LogDebug("Processing: {Input}", input); + return DoProcessing(input); + } +} +``` + +## Validation + +- [ ] Server starts without errors via `dotnet run` +- [ ] MCP Inspector connects and lists all expected tools +- [ ] Tool calls via Inspector return expected results +- [ ] Breakpoints hit when debugging in IDE +- [ ] Tools appear in GitHub Copilot Agent Mode tool list +- [ ] stdio: no logging output on stdout (stderr only) + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Tools not appearing in Copilot or Inspector | **Rebuild first:** `dotnet build`, then restart the server. If still missing, verify `[McpServerToolType]` on class, `[McpServerTool]` on methods, and `WithTools()` or `WithToolsFromAssembly()` in Program.cs | +| stdio server produces garbled output | `Console.WriteLine()` or logging is writing to stdout. All output **must** go to stderr. Set `LogToStandardErrorThreshold = LogLevel.Trace` on the console logger | +| "Command not found" when starting server | .NET 10+ SDK not installed. Check with `dotnet --version` | +| HTTP server returns 404 at MCP endpoint | Missing `app.MapMcp()` in Program.cs | +| Breakpoints not hit | Building in Release mode. Rebuild in Debug: `dotnet build -c Debug`, then restart | +| Environment variables not passed to server | Add `"env"` section to `mcp.json`. For secrets in VS Code, use `"${input:var_id}"` syntax | +| MCP Inspector can't connect to HTTP server | Server not running, or wrong port. Check `dotnet run` output for the listening URL | +| Stale tools after code changes | Always `dotnet build` and restart the server after changing tool methods or attributes | + +## Related Skills + +- `mcp-csharp-create` — Create a new MCP server project +- `mcp-csharp-test` — Automated tests and evaluations +- `mcp-csharp-publish` — NuGet, Docker, Azure deployment + +## Reference Files + +- [references/mcp-inspector.md](references/mcp-inspector.md) — Detailed MCP Inspector usage: installation, connecting to servers, feature walkthrough, troubleshooting. **Load when:** user needs detailed Inspector guidance or is having connection issues. +- [references/ide-config.md](references/ide-config.md) — Complete VS Code and Visual Studio configuration: mcp.json templates, launch.json, environment variables, conditional breakpoints. **Load when:** setting up IDE debugging or configuring environment-specific settings. + +## More Info + +- [MCP Inspector](https://www.npmjs.com/package/@modelcontextprotocol/inspector) — Interactive debugging tool for MCP servers +- [VS Code MCP documentation](https://code.visualstudio.com/docs/copilot/chat/mcp-servers) — Configuring MCP servers in VS Code diff --git a/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md b/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md new file mode 100644 index 0000000000..739f485320 --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md @@ -0,0 +1,144 @@ +# NuGet Packaging + +Detailed guide for publishing stdio MCP servers as NuGet tool packages. + +## Complete .csproj Configuration + +```xml + + + Exe + net10.0 + enable + enable + + + true + mymcpserver + + + YourUsername.MyMcpServer + 1.0.0 + Your Name + MCP server for interacting with MyService API + + + https://github.com/yourusername/mymcpserver + https://github.com/yourusername/mymcpserver + MIT + mcp;modelcontextprotocol;ai;llm + README.md + + + win-x64;linux-x64;osx-x64;osx-arm64 + + + + + + +``` + +### Key Properties + +| Property | Required | Purpose | +|----------|----------|---------| +| `PackAsTool` | Yes | Makes the package installable as a dotnet tool | +| `ToolCommandName` | Recommended | CLI command name. Defaults to assembly name if omitted | +| `PackageId` | Yes | Unique identifier on NuGet.org | +| `Version` | Yes | SemVer version (e.g., `1.0.0`, `2.0.0-preview.1`) | +| `PackageTags` | Recommended | Include `mcp` and `modelcontextprotocol` for discoverability | + +## Build, Pack, and Push + +```bash +# Build +dotnet build -c Release + +# Create NuGet package +dotnet pack -c Release +# Output: bin/Release/YourUsername.MyMcpServer.1.0.0.nupkg + +# Test package locally +dotnet tool install --global --add-source bin/Release/ YourUsername.MyMcpServer +mymcpserver --help +dotnet tool uninstall --global YourUsername.MyMcpServer + +# Push to NuGet.org +dotnet nuget push bin/Release/*.nupkg \ + --api-key YOUR_NUGET_API_KEY \ + --source https://api.nuget.org/v3/index.json + +# Or push to NuGet test environment first +dotnet nuget push bin/Release/*.nupkg \ + --api-key YOUR_NUGET_API_KEY \ + --source https://apiint.nugettest.org/v3/index.json +``` + +## User Configuration + +After publishing, users configure their MCP client to run the tool: + +```json +{ + "servers": { + "MyMcpServer": { + "type": "stdio", + "command": "dnx", + "args": ["YourUsername.MyMcpServer@1.0.0", "--yes"], + "env": { + "API_KEY": "${input:api_key}" + } + } + } +} +``` + +The `dnx` tool runner (a `dotnet execute`-style runner for NuGet packages) downloads and runs the package automatically. For more details, see the .NET package execution docs: https://learn.microsoft.com/dotnet/core/tools/dotnet-execute + +## server.json for MCP Registry Integration + +If you plan to publish to the MCP Registry, include `.mcp/server.json` in your repo: + +```json +{ + "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", + "name": "io.github.yourusername/mymcpserver", + "description": "MCP server for interacting with MyService API", + "version": "1.0.0", + "packages": [ + { + "registryType": "nuget", + "registryBaseUrl": "https://api.nuget.org", + "identifier": "YourUsername.MyMcpServer", + "version": "1.0.0", + "transport": { + "type": "stdio" + }, + "environmentVariables": [ + { + "name": "API_KEY", + "value": "{api_key}", + "variables": { + "api_key": { + "description": "API key for MyService", + "isRequired": true, + "isSecret": true + } + } + } + ] + } + ], + "repository": { + "url": "https://github.com/yourusername/mymcpserver", + "source": "github" + } +} +``` + +**Version consistency:** Keep `` in `.csproj`, root `version` in `server.json`, and `packages[].version` in sync. A mismatch will cause MCP Registry validation to fail. + +## Trusted Publishing (OIDC) + +For CI/CD, use NuGet trusted publishing instead of long-lived API keys. See `nuget-trusted-publishing` skill for the full setup guide. diff --git a/plugins/dotnet/skills/mcp-csharp-test/SKILL.md b/plugins/dotnet/skills/mcp-csharp-test/SKILL.md new file mode 100644 index 0000000000..a373cd109d --- /dev/null +++ b/plugins/dotnet/skills/mcp-csharp-test/SKILL.md @@ -0,0 +1,180 @@ +--- +name: mcp-csharp-test +description: > + Test C# MCP servers at multiple levels: unit tests for individual tools and integration + tests using the MCP client SDK. + USE FOR: unit testing MCP tool methods, integration testing with in-memory MCP + client/server, end-to-end testing via MCP protocol, + testing HTTP MCP servers with WebApplicationFactory, mocking dependencies in tool tests. + DO NOT USE FOR: testing MCP clients (this is server testing only), load or performance + testing, testing non-.NET MCP servers, debugging server issues (use mcp-csharp-debug). +--- + +# C# MCP Server Testing + +Test MCP servers at two levels: unit tests for individual tool methods, and integration tests that exercise the full MCP protocol in-memory. + +## When to Use + +- Adding automated tests to an MCP server +- Testing individual tool methods with mocked dependencies +- Writing integration tests that validate tool listing and invocation via MCP protocol +- Setting up CI test pipelines for MCP servers + +## Stop Signals + +- **No server yet?** → Use `mcp-csharp-create` first +- **Server not running?** → Use `mcp-csharp-debug` +- **Just need manual/interactive testing?** → Use `mcp-csharp-debug` for MCP Inspector + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| MCP server project path | Yes | Path to the server `.csproj` being tested | +| Test framework | Recommended | Default: xUnit. Also supports NUnit or MSTest | +| Transport type | Recommended | Determines integration test approach (stdio vs HTTP) | + +## Workflow + +### Step 1: Create the test project + +```bash +dotnet new xunit -n .Tests +cd .Tests +dotnet add reference ..//.csproj +dotnet add package ModelContextProtocol +dotnet add package Moq +dotnet add package FluentAssertions +``` + +### Step 2: Write unit tests for tool methods + +Test tool methods directly — fastest and most isolated: + +```csharp +public class MyToolTests +{ + [Fact] + public void Echo_ReturnsFormattedMessage() + { + var result = MyTools.Echo("Hello"); + result.Should().Be("Echo: Hello"); + } + + [Theory] + [InlineData("")] + [InlineData(" ")] + public void Echo_HandlesEdgeCases(string input) + { + var result = MyTools.Echo(input); + result.Should().StartWith("Echo:"); + } +} +``` + +For tools with DI dependencies, mock the dependency: +```csharp +public class ApiToolTests +{ + [Fact] + public async Task FetchData_ReturnsApiResponse() + { + var handler = new MockHttpMessageHandler("""{"id": 1}"""); + var httpClient = new HttpClient(handler); + + var result = await ApiTools.FetchData(httpClient, "resource-1"); + result.Should().Contain("id"); + } +} +``` + +### Step 3: Write integration tests with MCP client + +Test the full MCP protocol using a client-server connection: + +```csharp +using ModelContextProtocol.Client; + +public class ServerIntegrationTests : IAsyncLifetime +{ + private McpClient _client = null!; + + public async Task InitializeAsync() + { + var transport = new StdioClientTransport(new StdioClientTransportOptions + { + Name = "TestClient", + Command = "dotnet", + Arguments = ["run", "--project", "..//.csproj"] + }); + _client = await McpClient.CreateAsync(transport); + } + + public async Task DisposeAsync() => await _client.DisposeAsync(); + + [Fact] + public async Task Server_ListsExpectedTools() + { + var tools = await _client.ListToolsAsync(); + tools.Should().Contain(t => t.Name == "echo"); + } + + [Fact] + public async Task Tool_ReturnsExpectedResult() + { + var result = await _client.CallToolAsync("echo", + new Dictionary { ["message"] = "Test" }); + var text = result.Content.OfType().First().Text; + text.Should().Contain("Test"); + } +} +``` + +**For the SDK's `ClientServerTestBase` (in-memory testing) and HTTP testing with `WebApplicationFactory`**, see [references/test-patterns.md](references/test-patterns.md). + +### Step 4: Run tests + +```bash +# Run all tests +dotnet test + +# Run a specific test class +dotnet test --filter "FullyQualifiedName~MyToolTests" + +# Run with coverage +dotnet test --collect:"XPlat Code Coverage" +``` + +## Validation + +- [ ] Unit tests cover all tool methods, including edge cases +- [ ] Integration tests verify tool listing via `ListToolsAsync()` +- [ ] Integration tests verify tool invocation via `CallToolAsync()` +- [ ] All tests pass: `dotnet test` +- [ ] Tests run in CI without manual setup + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Integration test hangs on `CreateAsync` | Server fails to start. Verify `dotnet build` succeeds first. For stdio, ensure no stdout logging | +| `StdioClientTransport` not finding project | Use the correct relative path to `.csproj` from the test project directory | +| Tests pass locally but fail in CI | Run `dotnet build` before test execution. Use `--no-build` only after an explicit build step | +| Mocking `HttpClient` is awkward | Mock `HttpMessageHandler`, not `HttpClient` directly. See [references/test-patterns.md](references/test-patterns.md) | +| Full test suite runs are slow | Use `--filter` for development. Run the full suite only for CI verification | + +## Related Skills + +- `mcp-csharp-create` — Create a new MCP server project +- `mcp-csharp-debug` — Running and interactive debugging +- `mcp-csharp-publish` — NuGet, Docker, Azure deployment + +## Reference Files + +- [references/test-patterns.md](references/test-patterns.md) — Complete test code examples: `ClientServerTestBase` in-memory pattern, `WebApplicationFactory` for HTTP, `MockHttpMessageHandler` helper, test categorization, coverage reporting. **Load when:** writing integration tests or need detailed mock patterns. + +## More Info + +- [xUnit documentation](https://xunit.net/docs/getting-started/netcore/cmdline) — Getting started with xUnit for .NET +- [FluentAssertions](https://fluentassertions.com/) — Readable assertion library for .NET diff --git a/tests/dotnet/mcp-csharp-create/eval.yaml b/tests/dotnet/mcp-csharp-create/eval.yaml new file mode 100644 index 0000000000..1ce9f29cac --- /dev/null +++ b/tests/dotnet/mcp-csharp-create/eval.yaml @@ -0,0 +1,57 @@ +scenarios: + - name: "Implement MCP tools with proper attributes and DI" + prompt: | + I have a new C# MCP server project. I need to implement a tool class that + wraps a REST API using HttpClient. The tools should follow MCP SDK conventions + with proper attributes for LLM discovery. Show me the tool class, Program.cs + with stdio transport, and explain how DI works for MCP tools. + assertions: + - type: "output_matches" + pattern: "\\[McpServerToolType\\]" + - type: "output_matches" + pattern: "\\[McpServerTool[\\],\\(]" + - type: "output_matches" + pattern: "(AddHttpClient|HttpClient)" + rubric: + - "Shows a tool class with [McpServerToolType] and [McpServerTool] attributes" + - "Injects HttpClient via DI (constructor injection or method parameter injection)" + - "Includes [Description] attributes on both the method and all parameters" + - "Configures Program.cs with AddMcpServer, WithStdioServerTransport, and logging to stderr" + timeout: 180 + - name: "Create an HTTP MCP server with tools and resources" + prompt: | + I need to create a C# MCP server that uses HTTP transport for deployment + as a web service. It should expose both tools and resources (for example, + a resource that returns configuration data). Show me how to set up the + HTTP transport with MapMcp and implement a resource. + assertions: + - type: "output_contains" + value: "MapMcp" + - type: "output_matches" + pattern: "(WithHttpTransport|ModelContextProtocol\\.AspNetCore)" + - type: "output_matches" + pattern: "(McpServerResource|McpServerResourceType)" + rubric: + - "Configures HTTP transport with WithHttpTransport() and references ModelContextProtocol.AspNetCore" + - "Includes MapMcp() in the endpoint configuration" + - "Shows a resource class with [McpServerResourceType] and [McpServerResource] attributes including UriTemplate" + - "Shows how to register tools and resources with WithToolsFromAssembly or similar DI registration" + timeout: 180 + - name: "Create an MCP server with tools, prompts, and proper logging" + prompt: | + I'm building a C# MCP server using stdio transport. I need to add a tool + that calls an external API using HttpClient (injected via DI), and a prompt + template for summarization. Make sure logging doesn't interfere with the + stdio JSON-RPC protocol. + assertions: + - type: "output_matches" + pattern: "\\[McpServerTool[\\]T,\\(]" + - type: "output_matches" + pattern: "(McpServerPrompt|McpServerPromptType)" + - type: "output_matches" + pattern: "(LogToStandardErrorThreshold|stderr)" + rubric: + - "Shows a tool class with [McpServerToolType]/[McpServerTool] and HttpClient injected via DI" + - "Shows a prompt class with [McpServerPromptType] and [McpServerPrompt] returning ChatMessage" + - "Configures logging to stderr (LogToStandardErrorThreshold) to avoid corrupting stdio transport" + - "Adds [Description] attributes on tools, parameters, and prompts for LLM discoverability" From ada9dd3d7bb4b0b8de07938dbdfb851ef4478cca Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Fri, 20 Mar 2026 13:02:32 -0700 Subject: [PATCH 17/29] cleaned up files --- .../dotnet/skills/mcp-csharp-create/SKILL.md | 265 ------------------ .../dotnet/skills/mcp-csharp-debug/SKILL.md | 236 ---------------- .../references/nuget-packaging.md | 144 ---------- .../dotnet/skills/mcp-csharp-test/SKILL.md | 180 ------------ tests/dotnet/mcp-csharp-create/eval.yaml | 57 ---- 5 files changed, 882 deletions(-) delete mode 100644 plugins/dotnet/skills/mcp-csharp-create/SKILL.md delete mode 100644 plugins/dotnet/skills/mcp-csharp-debug/SKILL.md delete mode 100644 plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md delete mode 100644 plugins/dotnet/skills/mcp-csharp-test/SKILL.md delete mode 100644 tests/dotnet/mcp-csharp-create/eval.yaml diff --git a/plugins/dotnet/skills/mcp-csharp-create/SKILL.md b/plugins/dotnet/skills/mcp-csharp-create/SKILL.md deleted file mode 100644 index 8c471c935c..0000000000 --- a/plugins/dotnet/skills/mcp-csharp-create/SKILL.md +++ /dev/null @@ -1,265 +0,0 @@ ---- -name: mcp-csharp-create -description: > - Create MCP servers using the C# SDK and .NET project templates. Covers scaffolding, - tool/prompt/resource implementation, and transport configuration for stdio and HTTP. - USE FOR: creating new MCP server projects, scaffolding with dotnet new mcpserver, adding - MCP tools/prompts/resources, choosing stdio vs HTTP transport, configuring MCP hosting in - Program.cs, setting up ASP.NET Core MCP endpoints with MapMcp. - DO NOT USE FOR: debugging or running existing servers (use mcp-csharp-debug), writing tests - (use mcp-csharp-test), publishing or deploying (use mcp-csharp-publish), building MCP - clients, non-.NET MCP servers. ---- - -# C# MCP Server Creation - -Create Model Context Protocol servers using the official C# SDK (`ModelContextProtocol` NuGet package) and the `dotnet new mcpserver` project template. Servers expose tools, prompts, and resources that LLMs can discover and invoke via the MCP protocol. - -## When to Use - -- Starting a new MCP server project from scratch -- Adding tools, prompts, or resources to an existing MCP server -- Choosing between stdio (`--transport local`) and HTTP (`--transport remote`) transport -- Setting up ASP.NET Core hosting for an HTTP MCP server -- Wrapping an external API or service as MCP tools - -## Stop Signals - -- **Server already exists and needs debugging?** → Use `mcp-csharp-debug` -- **Need tests or evaluations?** → Use `mcp-csharp-test` -- **Ready to publish?** → Use `mcp-csharp-publish` -- **Building an MCP client, not a server** → This skill is server-side only - -## Inputs - -| Input | Required | Description | -|-------|----------|-------------| -| Transport type | Yes | `stdio` (local/CLI) or `http` (remote/web). Ask user if not specified — default to stdio | -| Project name | Yes | PascalCase name for the project (e.g., `WeatherMcpServer`) | -| .NET SDK version | Recommended | .NET 10.0+ required. Check with `dotnet --version` | -| Service/API to wrap | Recommended | External API or service the tools will interact with | - -## Workflow - -> **Commit strategy:** Commit after completing each step so scaffolding and implementation are separately reviewable. - -### Step 1: Verify prerequisites - -1. Confirm .NET 10+ SDK: `dotnet --version` (install from https://dotnet.microsoft.com if < 10.0) - -2. Check if the MCP server template is already installed: - ```bash - dotnet new list mcpserver - ``` - If "No templates found" → install: `dotnet new install Microsoft.McpServer.ProjectTemplates` - -### Step 2: Choose transport - -| Choose **stdio** if… | Choose **HTTP** if… | -|----------------------|---------------------| -| Local CLI tool or IDE plugin | Cloud/web service deployment | -| Single user at a time | Multiple simultaneous clients | -| Running as subprocess (VS Code, GitHub Copilot) | Cross-network access needed | -| Simpler setup, no network config | Containerized deployment (Docker/Azure) | - -**Default:** stdio — simpler, works for most local development. Users can add HTTP later. - -### Step 3: Scaffold the project - -**stdio server:** -```bash -dotnet new mcpserver -n -``` -If the template times out or is unavailable, use `dotnet new console -n ` and add `dotnet add package ModelContextProtocol`. - -**HTTP server:** -```bash -dotnet new web -n -cd -dotnet add package ModelContextProtocol.AspNetCore -``` -This is the recommended approach — faster and more reliable than the template. The template also supports HTTP via `dotnet new mcpserver -n --transport remote`, but `dotnet new web` gives you more control over the project structure. - -**Template flags reference:** `--transport local` (stdio, default), `--transport remote` (ASP.NET Core HTTP), `--aot`, `--self-contained`. - -### Step 4: Implement tools - -Tools are the primary way MCP servers expose functionality. Add a class with `[McpServerToolType]` and methods with `[McpServerTool]`: - -```csharp -using ModelContextProtocol.Server; -using System.ComponentModel; - -[McpServerToolType] -public static class MyTools -{ - [McpServerTool, Description("Brief description of what the tool does.")] - public static async Task DoSomething( - [Description("What this parameter controls")] string input, - CancellationToken cancellationToken = default) - { - // Implementation - return $"Result: {input}"; - } -} -``` - -**Critical rules:** -- Every tool method **must** have a `[Description]` attribute — LLMs use this to decide when to call the tool -- Every parameter **must** have a `[Description]` attribute -- Accept `CancellationToken` in all async tools -- Use `[McpServerTool(Name = "custom_name")]` only if the default method name is unclear - -**DI injection patterns** — the SDK supports two styles: - -1. **Method parameter injection (static class):** DI services appear as method parameters. The SDK resolves them automatically — they do not appear in the tool schema. - -2. **Constructor injection (non-static class):** Use when tools need shared state or multiple services: -```csharp -[McpServerToolType] -public class ApiTools(HttpClient httpClient, ILogger logger) -{ - [McpServerTool, Description("Fetch a resource by ID.")] - public async Task FetchResource( - [Description("Resource identifier")] string id, - CancellationToken cancellationToken = default) - { - logger.LogInformation("Fetching {Id}", id); - return await httpClient.GetStringAsync($"/api/{id}", cancellationToken); - } -} -``` -Register services in Program.cs: -```csharp -var builder = Host.CreateApplicationBuilder(args); -builder.Logging.AddConsole(options => - options.LogToStandardErrorThreshold = LogLevel.Trace); - -builder.Services.AddHttpClient(); // registers IHttpClientFactory + HttpClient -// ILogger is registered by default — no extra setup needed. - -builder.Services.AddMcpServer() - .WithStdioServerTransport() - .WithToolsFromAssembly(); // discovers non-static [McpServerToolType] classes - -await builder.Build().RunAsync(); -``` - -**For the full attribute reference, return types, DI injection, and builder API patterns**, see [references/api-patterns.md](references/api-patterns.md). - -### Step 5: Add prompts and resources (optional) - -**Prompts** — reusable LLM interaction templates: -```csharp -[McpServerPromptType] -public static class MyPrompts -{ - [McpServerPrompt, Description("Summarize content into one sentence.")] - public static ChatMessage Summarize( - [Description("Content to summarize")] string content) => - new(ChatRole.User, $"Summarize this into one sentence: {content}"); -} -``` - -**Resources** — data the LLM can read: -```csharp -[McpServerResourceType] -public static class MyResources -{ - [McpServerResource(UriTemplate = "config://app", Name = "App Config", - MimeType = "application/json"), Description("Application configuration")] - public static string GetConfig() => JsonSerializer.Serialize(AppConfig.Current); -} -``` - -### Step 6: Configure Program.cs - -**stdio transport:** -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.Logging; -using ModelContextProtocol.Server; - -var builder = Host.CreateApplicationBuilder(args); -builder.Logging.AddConsole(options => - options.LogToStandardErrorThreshold = LogLevel.Trace); // CRITICAL: stderr only - -builder.Services.AddMcpServer() - .WithStdioServerTransport() - .WithToolsFromAssembly(); - -await builder.Build().RunAsync(); -``` - -**HTTP transport:** -```csharp -using ModelContextProtocol.Server; - -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddMcpServer() - .WithHttpTransport() - .WithToolsFromAssembly(); - -// Register services your tools need via DI -// builder.Services.AddHttpClient(); -// builder.Services.AddSingleton(); - -var app = builder.Build(); -app.MapMcp(); // exposes MCP endpoint at /mcp (Streamable HTTP) -app.MapGet("/health", () => "ok"); // health check for container orchestrators -app.Run(); -``` - -**Key HTTP details:** `MapMcp()` defaults to `/mcp` path. For containers, set `ASPNETCORE_URLS=http://+:8080` and `EXPOSE 8080`. The MCP HTTP protocol uses Streamable HTTP — no special client config needed beyond the URL. - -**For transport configuration details** (stateless mode, auth, path prefix, `HttpContextAccessor`), see [references/transport-config.md](references/transport-config.md). - -### Step 7: Verify the server starts - -```bash -cd -dotnet build -dotnet run -``` - -For stdio: the process starts and waits for JSON-RPC input on stdin. -For HTTP: the server listens on the configured port. - -## Validation - -- [ ] Project builds with no errors (`dotnet build`) -- [ ] All tool classes have `[McpServerToolType]` attribute -- [ ] All tool methods have `[McpServerTool]` and `[Description]` attributes -- [ ] All parameters have `[Description]` attributes -- [ ] stdio: logging directed to stderr, not stdout -- [ ] HTTP: `app.MapMcp()` is called in Program.cs -- [ ] Server starts successfully with `dotnet run` - -## Common Pitfalls - -| Pitfall | Solution | -|---------|----------| -| stdio server outputs garbage or hangs | Logging to stdout corrupts JSON-RPC protocol. Set `LogToStandardErrorThreshold = LogLevel.Trace` | -| Tool not discovered by LLM clients | Missing `[McpServerToolType]` on the class or `[McpServerTool]` on the method. Verify `.WithToolsFromAssembly()` in Program.cs | -| LLM doesn't understand when to use a tool | Add clear `[Description]` attributes on both the method and all parameters | -| `WithToolsFromAssembly()` fails in AOT | Reflection-based discovery is incompatible with Native AOT. Use `.WithTools()` instead | -| Parameters not appearing in tool schema | `CancellationToken`, `IMcpServer`, and DI services are injected automatically — they do not appear in the schema. Only parameters with `[Description]` are exposed | -| HTTP server returns 404 | `app.MapMcp()` must be called. Check the request path matches the configured route | - -## Related Skills - -- `mcp-csharp-debug` — Run, debug, and test with MCP Inspector -- `mcp-csharp-test` — Unit tests, integration tests, evaluations -- `mcp-csharp-publish` — NuGet, Docker, Azure deployment - -## Reference Files - -- [references/api-patterns.md](references/api-patterns.md) — Complete attribute reference, return types, DI injection, builder API, dynamic tools, experimental APIs. **Load when:** implementing tools, prompts, or resources beyond the basic patterns shown above. -- [references/transport-config.md](references/transport-config.md) — Detailed transport configuration: stateless HTTP mode, OAuth/auth, custom path prefix, `HttpContextAccessor`, OpenTelemetry observability. **Load when:** configuring advanced transport options or authentication. - -## More Info - -- [C# MCP SDK](https://github.com/modelcontextprotocol/csharp-sdk) — Official SDK repository -- [Build an MCP server (.NET)](https://learn.microsoft.com/dotnet/ai/quickstarts/build-mcp-server) — Microsoft quickstart -- [MCP Specification](https://modelcontextprotocol.io/specification/) — Protocol specification diff --git a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md b/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md deleted file mode 100644 index 0b6e806f44..0000000000 --- a/plugins/dotnet/skills/mcp-csharp-debug/SKILL.md +++ /dev/null @@ -1,236 +0,0 @@ ---- -name: mcp-csharp-debug -description: > - Run and debug C# MCP servers locally. Covers IDE configuration, MCP Inspector testing, - GitHub Copilot Agent Mode integration, logging setup, and troubleshooting. - USE FOR: running MCP servers locally with dotnet run, configuring VS Code or Visual Studio - for MCP debugging, testing tools with MCP Inspector, testing with GitHub Copilot Agent Mode, - diagnosing tool registration issues, setting up mcp.json configuration, debugging MCP - protocol messages, configuring logging for stdio and HTTP servers. - DO NOT USE FOR: creating new MCP servers (use mcp-csharp-create), writing automated tests - (use mcp-csharp-test), publishing or deploying to production (use mcp-csharp-publish). ---- - -# C# MCP Server Debugging - -Run, debug, and interactively test C# MCP servers. Covers local execution, IDE debugging with breakpoints, MCP Inspector for protocol-level testing, and GitHub Copilot Agent Mode integration. - -## When to Use - -- Running an MCP server locally for the first time -- Configuring VS Code or Visual Studio to debug an MCP server -- Testing tools interactively with MCP Inspector -- Verifying tools appear in GitHub Copilot Agent Mode -- Diagnosing issues: tools not discovered, protocol errors, server crashes -- Setting up `mcp.json` or `.mcp.json` configuration - -## Stop Signals - -- **No project yet?** → Use `mcp-csharp-create` first -- **Need automated tests?** → Use `mcp-csharp-test` -- **Production deployment issue?** → Use `mcp-csharp-publish` - -## Inputs - -| Input | Required | Description | -|-------|----------|-------------| -| Project path | Yes | Path to the `.csproj` file or project directory | -| Transport type | Recommended | `stdio` or `http` — detect from `.csproj` if not specified | -| IDE | Recommended | VS Code or Visual Studio — detect from environment if not specified | - -**Agent behavior:** Detect transport type by checking the `.csproj` for a `PackageReference` to `ModelContextProtocol.AspNetCore`. If present → HTTP, otherwise → stdio. - -## Workflow - -### Step 1: Run the server locally - -**stdio transport:** -```bash -cd -dotnet run -``` -The process starts and waits for JSON-RPC messages on stdin. No output on stdout means it's working correctly. - -**HTTP transport:** -```bash -cd -dotnet run -# Server listens on http://localhost:3001 (or configured port) -``` - -### Step 2: Generate MCP configuration - -Detect the IDE and transport, then create the appropriate config file. - -**For VS Code** — create `.vscode/mcp.json`: - -stdio: -```json -{ - "servers": { - "": { - "type": "stdio", - "command": "dotnet", - "args": ["run", "--project", ""] - } - } -} -``` - -HTTP: -```json -{ - "servers": { - "": { - "type": "http", - "url": "http://localhost:3001" - } - } -} -``` - -**For Visual Studio** — create `.mcp.json` at solution root (same JSON structure). - -**For detailed IDE-specific configuration** (launch.json, environment variables, secrets), see [references/ide-config.md](references/ide-config.md). - -### Step 3: Test with MCP Inspector - -The MCP Inspector provides a UI for testing tools, viewing schemas, and inspecting protocol messages. - -**stdio server:** -```bash -npx @modelcontextprotocol/inspector dotnet run --project -``` - -**HTTP server:** -1. Start your server: `dotnet run` -2. Run Inspector: `npx @modelcontextprotocol/inspector` -3. Connect to `http://localhost:3001` - -**Inspector capabilities:** -- List all registered tools, prompts, and resources -- Call tools with custom parameters and see results -- View request/response JSON-RPC messages -- Inspect tool schemas and descriptions - -**For detailed Inspector usage and troubleshooting**, see [references/mcp-inspector.md](references/mcp-inspector.md). - -### Step 4: Test with GitHub Copilot Agent Mode - -1. Open GitHub Copilot Chat → switch to **Agent** mode -2. Click **Select Tools** (wrench icon) → verify your server and tools are listed -3. Test with a prompt that should trigger your tool -4. Approve tool execution when prompted - -**If tools don't appear — troubleshoot tool discovery:** - -1. **Rebuild first** — stale builds are the #1 cause: - ```bash - dotnet build - ``` - Then restart the MCP server (click Stop → Start in VS Code, or restart `dotnet run`). - -2. **Check `[McpServerToolType]` on the class:** - ```csharp - [McpServerToolType] // ← Required on the class - public class MyTools { ... } - ``` - -3. **Check `[McpServerTool]` on each tool method:** - - The method must be `public`. - - It can be `static` or instance. For instance methods, ensure the containing type is discoverable/registered (for example via `WithTools()` or `WithToolsFromAssembly()`) so DI can construct it. - ```csharp - [McpServerTool, Description("Does something")] - public string DoSomething(string input) => input; - ``` - -4. **Verify tool registration in Program.cs** — use one of: - ```csharp - .WithTools() // register specific class - .WithToolsFromAssembly() // scan entire assembly for [McpServerToolType] - ``` - -5. **Check `mcp.json`** points to the correct project path - -6. If still not appearing, reference the tool explicitly: `Using #tool_name, do X` - -### Step 5: Set up breakpoint debugging - -1. Set breakpoints in your tool methods -2. Launch with the debugger: - - **VS Code:** F5 (requires `launch.json` — see [references/ide-config.md](references/ide-config.md)) - - **Visual Studio:** F5 or right-click project → Debug → Start -3. Trigger the tool (via Inspector, Copilot, or test client) -4. Execution pauses at breakpoints - -**Critical:** Build in Debug configuration. Breakpoints won't hit in Release builds. - -### Step 6: Configure logging - -**Critical for stdio transport:** Any output to stdout (including `Console.WriteLine`) **corrupts the MCP JSON-RPC protocol** and causes garbled responses or crashes. All logging and diagnostic output must go to stderr. - -**stdio transport** — log to stderr only: -```csharp -builder.Logging.AddConsole(options => - options.LogToStandardErrorThreshold = LogLevel.Trace); -``` - -**HTTP transport** — standard console logging: -```csharp -builder.Logging.ClearProviders(); -builder.Logging.AddConsole(); -builder.Logging.SetMinimumLevel( - builder.Environment.IsDevelopment() ? LogLevel.Debug : LogLevel.Information); -``` - -**In tool methods** — inject `ILogger` via constructor: -```csharp -[McpServerToolType] -public class MyTools(ILogger logger) -{ - [McpServerTool, Description("Processes data")] - public string ProcessData(string input) - { - logger.LogDebug("Processing: {Input}", input); - return DoProcessing(input); - } -} -``` - -## Validation - -- [ ] Server starts without errors via `dotnet run` -- [ ] MCP Inspector connects and lists all expected tools -- [ ] Tool calls via Inspector return expected results -- [ ] Breakpoints hit when debugging in IDE -- [ ] Tools appear in GitHub Copilot Agent Mode tool list -- [ ] stdio: no logging output on stdout (stderr only) - -## Common Pitfalls - -| Pitfall | Solution | -|---------|----------| -| Tools not appearing in Copilot or Inspector | **Rebuild first:** `dotnet build`, then restart the server. If still missing, verify `[McpServerToolType]` on class, `[McpServerTool]` on methods, and `WithTools()` or `WithToolsFromAssembly()` in Program.cs | -| stdio server produces garbled output | `Console.WriteLine()` or logging is writing to stdout. All output **must** go to stderr. Set `LogToStandardErrorThreshold = LogLevel.Trace` on the console logger | -| "Command not found" when starting server | .NET 10+ SDK not installed. Check with `dotnet --version` | -| HTTP server returns 404 at MCP endpoint | Missing `app.MapMcp()` in Program.cs | -| Breakpoints not hit | Building in Release mode. Rebuild in Debug: `dotnet build -c Debug`, then restart | -| Environment variables not passed to server | Add `"env"` section to `mcp.json`. For secrets in VS Code, use `"${input:var_id}"` syntax | -| MCP Inspector can't connect to HTTP server | Server not running, or wrong port. Check `dotnet run` output for the listening URL | -| Stale tools after code changes | Always `dotnet build` and restart the server after changing tool methods or attributes | - -## Related Skills - -- `mcp-csharp-create` — Create a new MCP server project -- `mcp-csharp-test` — Automated tests and evaluations -- `mcp-csharp-publish` — NuGet, Docker, Azure deployment - -## Reference Files - -- [references/mcp-inspector.md](references/mcp-inspector.md) — Detailed MCP Inspector usage: installation, connecting to servers, feature walkthrough, troubleshooting. **Load when:** user needs detailed Inspector guidance or is having connection issues. -- [references/ide-config.md](references/ide-config.md) — Complete VS Code and Visual Studio configuration: mcp.json templates, launch.json, environment variables, conditional breakpoints. **Load when:** setting up IDE debugging or configuring environment-specific settings. - -## More Info - -- [MCP Inspector](https://www.npmjs.com/package/@modelcontextprotocol/inspector) — Interactive debugging tool for MCP servers -- [VS Code MCP documentation](https://code.visualstudio.com/docs/copilot/chat/mcp-servers) — Configuring MCP servers in VS Code diff --git a/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md b/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md deleted file mode 100644 index 739f485320..0000000000 --- a/plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md +++ /dev/null @@ -1,144 +0,0 @@ -# NuGet Packaging - -Detailed guide for publishing stdio MCP servers as NuGet tool packages. - -## Complete .csproj Configuration - -```xml - - - Exe - net10.0 - enable - enable - - - true - mymcpserver - - - YourUsername.MyMcpServer - 1.0.0 - Your Name - MCP server for interacting with MyService API - - - https://github.com/yourusername/mymcpserver - https://github.com/yourusername/mymcpserver - MIT - mcp;modelcontextprotocol;ai;llm - README.md - - - win-x64;linux-x64;osx-x64;osx-arm64 - - - - - - -``` - -### Key Properties - -| Property | Required | Purpose | -|----------|----------|---------| -| `PackAsTool` | Yes | Makes the package installable as a dotnet tool | -| `ToolCommandName` | Recommended | CLI command name. Defaults to assembly name if omitted | -| `PackageId` | Yes | Unique identifier on NuGet.org | -| `Version` | Yes | SemVer version (e.g., `1.0.0`, `2.0.0-preview.1`) | -| `PackageTags` | Recommended | Include `mcp` and `modelcontextprotocol` for discoverability | - -## Build, Pack, and Push - -```bash -# Build -dotnet build -c Release - -# Create NuGet package -dotnet pack -c Release -# Output: bin/Release/YourUsername.MyMcpServer.1.0.0.nupkg - -# Test package locally -dotnet tool install --global --add-source bin/Release/ YourUsername.MyMcpServer -mymcpserver --help -dotnet tool uninstall --global YourUsername.MyMcpServer - -# Push to NuGet.org -dotnet nuget push bin/Release/*.nupkg \ - --api-key YOUR_NUGET_API_KEY \ - --source https://api.nuget.org/v3/index.json - -# Or push to NuGet test environment first -dotnet nuget push bin/Release/*.nupkg \ - --api-key YOUR_NUGET_API_KEY \ - --source https://apiint.nugettest.org/v3/index.json -``` - -## User Configuration - -After publishing, users configure their MCP client to run the tool: - -```json -{ - "servers": { - "MyMcpServer": { - "type": "stdio", - "command": "dnx", - "args": ["YourUsername.MyMcpServer@1.0.0", "--yes"], - "env": { - "API_KEY": "${input:api_key}" - } - } - } -} -``` - -The `dnx` tool runner (a `dotnet execute`-style runner for NuGet packages) downloads and runs the package automatically. For more details, see the .NET package execution docs: https://learn.microsoft.com/dotnet/core/tools/dotnet-execute - -## server.json for MCP Registry Integration - -If you plan to publish to the MCP Registry, include `.mcp/server.json` in your repo: - -```json -{ - "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", - "name": "io.github.yourusername/mymcpserver", - "description": "MCP server for interacting with MyService API", - "version": "1.0.0", - "packages": [ - { - "registryType": "nuget", - "registryBaseUrl": "https://api.nuget.org", - "identifier": "YourUsername.MyMcpServer", - "version": "1.0.0", - "transport": { - "type": "stdio" - }, - "environmentVariables": [ - { - "name": "API_KEY", - "value": "{api_key}", - "variables": { - "api_key": { - "description": "API key for MyService", - "isRequired": true, - "isSecret": true - } - } - } - ] - } - ], - "repository": { - "url": "https://github.com/yourusername/mymcpserver", - "source": "github" - } -} -``` - -**Version consistency:** Keep `` in `.csproj`, root `version` in `server.json`, and `packages[].version` in sync. A mismatch will cause MCP Registry validation to fail. - -## Trusted Publishing (OIDC) - -For CI/CD, use NuGet trusted publishing instead of long-lived API keys. See `nuget-trusted-publishing` skill for the full setup guide. diff --git a/plugins/dotnet/skills/mcp-csharp-test/SKILL.md b/plugins/dotnet/skills/mcp-csharp-test/SKILL.md deleted file mode 100644 index a373cd109d..0000000000 --- a/plugins/dotnet/skills/mcp-csharp-test/SKILL.md +++ /dev/null @@ -1,180 +0,0 @@ ---- -name: mcp-csharp-test -description: > - Test C# MCP servers at multiple levels: unit tests for individual tools and integration - tests using the MCP client SDK. - USE FOR: unit testing MCP tool methods, integration testing with in-memory MCP - client/server, end-to-end testing via MCP protocol, - testing HTTP MCP servers with WebApplicationFactory, mocking dependencies in tool tests. - DO NOT USE FOR: testing MCP clients (this is server testing only), load or performance - testing, testing non-.NET MCP servers, debugging server issues (use mcp-csharp-debug). ---- - -# C# MCP Server Testing - -Test MCP servers at two levels: unit tests for individual tool methods, and integration tests that exercise the full MCP protocol in-memory. - -## When to Use - -- Adding automated tests to an MCP server -- Testing individual tool methods with mocked dependencies -- Writing integration tests that validate tool listing and invocation via MCP protocol -- Setting up CI test pipelines for MCP servers - -## Stop Signals - -- **No server yet?** → Use `mcp-csharp-create` first -- **Server not running?** → Use `mcp-csharp-debug` -- **Just need manual/interactive testing?** → Use `mcp-csharp-debug` for MCP Inspector - -## Inputs - -| Input | Required | Description | -|-------|----------|-------------| -| MCP server project path | Yes | Path to the server `.csproj` being tested | -| Test framework | Recommended | Default: xUnit. Also supports NUnit or MSTest | -| Transport type | Recommended | Determines integration test approach (stdio vs HTTP) | - -## Workflow - -### Step 1: Create the test project - -```bash -dotnet new xunit -n .Tests -cd .Tests -dotnet add reference ..//.csproj -dotnet add package ModelContextProtocol -dotnet add package Moq -dotnet add package FluentAssertions -``` - -### Step 2: Write unit tests for tool methods - -Test tool methods directly — fastest and most isolated: - -```csharp -public class MyToolTests -{ - [Fact] - public void Echo_ReturnsFormattedMessage() - { - var result = MyTools.Echo("Hello"); - result.Should().Be("Echo: Hello"); - } - - [Theory] - [InlineData("")] - [InlineData(" ")] - public void Echo_HandlesEdgeCases(string input) - { - var result = MyTools.Echo(input); - result.Should().StartWith("Echo:"); - } -} -``` - -For tools with DI dependencies, mock the dependency: -```csharp -public class ApiToolTests -{ - [Fact] - public async Task FetchData_ReturnsApiResponse() - { - var handler = new MockHttpMessageHandler("""{"id": 1}"""); - var httpClient = new HttpClient(handler); - - var result = await ApiTools.FetchData(httpClient, "resource-1"); - result.Should().Contain("id"); - } -} -``` - -### Step 3: Write integration tests with MCP client - -Test the full MCP protocol using a client-server connection: - -```csharp -using ModelContextProtocol.Client; - -public class ServerIntegrationTests : IAsyncLifetime -{ - private McpClient _client = null!; - - public async Task InitializeAsync() - { - var transport = new StdioClientTransport(new StdioClientTransportOptions - { - Name = "TestClient", - Command = "dotnet", - Arguments = ["run", "--project", "..//.csproj"] - }); - _client = await McpClient.CreateAsync(transport); - } - - public async Task DisposeAsync() => await _client.DisposeAsync(); - - [Fact] - public async Task Server_ListsExpectedTools() - { - var tools = await _client.ListToolsAsync(); - tools.Should().Contain(t => t.Name == "echo"); - } - - [Fact] - public async Task Tool_ReturnsExpectedResult() - { - var result = await _client.CallToolAsync("echo", - new Dictionary { ["message"] = "Test" }); - var text = result.Content.OfType().First().Text; - text.Should().Contain("Test"); - } -} -``` - -**For the SDK's `ClientServerTestBase` (in-memory testing) and HTTP testing with `WebApplicationFactory`**, see [references/test-patterns.md](references/test-patterns.md). - -### Step 4: Run tests - -```bash -# Run all tests -dotnet test - -# Run a specific test class -dotnet test --filter "FullyQualifiedName~MyToolTests" - -# Run with coverage -dotnet test --collect:"XPlat Code Coverage" -``` - -## Validation - -- [ ] Unit tests cover all tool methods, including edge cases -- [ ] Integration tests verify tool listing via `ListToolsAsync()` -- [ ] Integration tests verify tool invocation via `CallToolAsync()` -- [ ] All tests pass: `dotnet test` -- [ ] Tests run in CI without manual setup - -## Common Pitfalls - -| Pitfall | Solution | -|---------|----------| -| Integration test hangs on `CreateAsync` | Server fails to start. Verify `dotnet build` succeeds first. For stdio, ensure no stdout logging | -| `StdioClientTransport` not finding project | Use the correct relative path to `.csproj` from the test project directory | -| Tests pass locally but fail in CI | Run `dotnet build` before test execution. Use `--no-build` only after an explicit build step | -| Mocking `HttpClient` is awkward | Mock `HttpMessageHandler`, not `HttpClient` directly. See [references/test-patterns.md](references/test-patterns.md) | -| Full test suite runs are slow | Use `--filter` for development. Run the full suite only for CI verification | - -## Related Skills - -- `mcp-csharp-create` — Create a new MCP server project -- `mcp-csharp-debug` — Running and interactive debugging -- `mcp-csharp-publish` — NuGet, Docker, Azure deployment - -## Reference Files - -- [references/test-patterns.md](references/test-patterns.md) — Complete test code examples: `ClientServerTestBase` in-memory pattern, `WebApplicationFactory` for HTTP, `MockHttpMessageHandler` helper, test categorization, coverage reporting. **Load when:** writing integration tests or need detailed mock patterns. - -## More Info - -- [xUnit documentation](https://xunit.net/docs/getting-started/netcore/cmdline) — Getting started with xUnit for .NET -- [FluentAssertions](https://fluentassertions.com/) — Readable assertion library for .NET diff --git a/tests/dotnet/mcp-csharp-create/eval.yaml b/tests/dotnet/mcp-csharp-create/eval.yaml deleted file mode 100644 index 1ce9f29cac..0000000000 --- a/tests/dotnet/mcp-csharp-create/eval.yaml +++ /dev/null @@ -1,57 +0,0 @@ -scenarios: - - name: "Implement MCP tools with proper attributes and DI" - prompt: | - I have a new C# MCP server project. I need to implement a tool class that - wraps a REST API using HttpClient. The tools should follow MCP SDK conventions - with proper attributes for LLM discovery. Show me the tool class, Program.cs - with stdio transport, and explain how DI works for MCP tools. - assertions: - - type: "output_matches" - pattern: "\\[McpServerToolType\\]" - - type: "output_matches" - pattern: "\\[McpServerTool[\\],\\(]" - - type: "output_matches" - pattern: "(AddHttpClient|HttpClient)" - rubric: - - "Shows a tool class with [McpServerToolType] and [McpServerTool] attributes" - - "Injects HttpClient via DI (constructor injection or method parameter injection)" - - "Includes [Description] attributes on both the method and all parameters" - - "Configures Program.cs with AddMcpServer, WithStdioServerTransport, and logging to stderr" - timeout: 180 - - name: "Create an HTTP MCP server with tools and resources" - prompt: | - I need to create a C# MCP server that uses HTTP transport for deployment - as a web service. It should expose both tools and resources (for example, - a resource that returns configuration data). Show me how to set up the - HTTP transport with MapMcp and implement a resource. - assertions: - - type: "output_contains" - value: "MapMcp" - - type: "output_matches" - pattern: "(WithHttpTransport|ModelContextProtocol\\.AspNetCore)" - - type: "output_matches" - pattern: "(McpServerResource|McpServerResourceType)" - rubric: - - "Configures HTTP transport with WithHttpTransport() and references ModelContextProtocol.AspNetCore" - - "Includes MapMcp() in the endpoint configuration" - - "Shows a resource class with [McpServerResourceType] and [McpServerResource] attributes including UriTemplate" - - "Shows how to register tools and resources with WithToolsFromAssembly or similar DI registration" - timeout: 180 - - name: "Create an MCP server with tools, prompts, and proper logging" - prompt: | - I'm building a C# MCP server using stdio transport. I need to add a tool - that calls an external API using HttpClient (injected via DI), and a prompt - template for summarization. Make sure logging doesn't interfere with the - stdio JSON-RPC protocol. - assertions: - - type: "output_matches" - pattern: "\\[McpServerTool[\\]T,\\(]" - - type: "output_matches" - pattern: "(McpServerPrompt|McpServerPromptType)" - - type: "output_matches" - pattern: "(LogToStandardErrorThreshold|stderr)" - rubric: - - "Shows a tool class with [McpServerToolType]/[McpServerTool] and HttpClient injected via DI" - - "Shows a prompt class with [McpServerPromptType] and [McpServerPrompt] returning ChatMessage" - - "Configures logging to stderr (LogToStandardErrorThreshold) to avoid corrupting stdio transport" - - "Adds [Description] attributes on tools, parameters, and prompts for LLM discoverability" From a027525b0392e88f4a7efe00d07b7aaebb566396 Mon Sep 17 00:00:00 2001 From: leslierichardson95 Date: Fri, 20 Mar 2026 13:17:49 -0700 Subject: [PATCH 18/29] Update .github/CODEOWNERS Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/CODEOWNERS | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index c9be2d39b6..d4b966ecaa 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -24,17 +24,17 @@ /plugins/dotnet/agents/optimizing-dotnet-performance.agent.md @dotnet/appmodel -/plugins/dotnet/skills/mcp-csharp-create/ @leslierichardson95 @artl93 -/tests/dotnet/mcp-csharp-create/ @leslierichardson95 @artl93 +/plugins/dotnet-ai/skills/mcp-csharp-create/ @leslierichardson95 @artl93 +/tests/dotnet-ai/mcp-csharp-create/ @leslierichardson95 @artl93 -/plugins/dotnet/skills/mcp-csharp-debug/ @leslierichardson95 @artl93 -/tests/dotnet/mcp-csharp-debug/ @leslierichardson95 @artl93 +/plugins/dotnet-ai/skills/mcp-csharp-debug/ @leslierichardson95 @artl93 +/tests/dotnet-ai/mcp-csharp-debug/ @leslierichardson95 @artl93 -/plugins/dotnet/skills/mcp-csharp-publish/ @leslierichardson95 @artl93 -/tests/dotnet/mcp-csharp-publish/ @leslierichardson95 @artl93 +/plugins/dotnet-ai/skills/mcp-csharp-publish/ @leslierichardson95 @artl93 +/tests/dotnet-ai/mcp-csharp-publish/ @leslierichardson95 @artl93 -/plugins/dotnet/skills/mcp-csharp-test/ @leslierichardson95 @artl93 -/tests/dotnet/mcp-csharp-test/ @leslierichardson95 @artl93 +/plugins/dotnet-ai/skills/mcp-csharp-test/ @leslierichardson95 @artl93 +/tests/dotnet-ai/mcp-csharp-test/ @leslierichardson95 @artl93 # dotnet-upgrade (migrating and upgrading .NET projects) /plugins/dotnet-upgrade/skills/thread-abort-migration/ @dotnet/appmodel From e2501963d9962ba550f2cd3772308183459cea5d Mon Sep 17 00:00:00 2001 From: Dan Moseley Date: Fri, 20 Mar 2026 14:34:32 -0600 Subject: [PATCH 19/29] Update nuget-packaging.md --- .../skills/mcp-csharp-publish/references/nuget-packaging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md index 739f485320..4a0af183da 100644 --- a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md @@ -23,8 +23,8 @@ Detailed guide for publishing stdio MCP servers as NuGet tool packages. MCP server for interacting with MyService API - https://github.com/yourusername/mymcpserver - https://github.com/yourusername/mymcpserver + https://github.com// + https://github.com// MIT mcp;modelcontextprotocol;ai;llm README.md From edae0d0d0b91cce00046f0d015e6c9de895af414 Mon Sep 17 00:00:00 2001 From: Dan Moseley Date: Fri, 20 Mar 2026 14:37:49 -0600 Subject: [PATCH 20/29] Update repository URL format in nuget-packaging.md --- .../skills/mcp-csharp-publish/references/nuget-packaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md index 4a0af183da..eb2791943d 100644 --- a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md @@ -131,7 +131,7 @@ If you plan to publish to the MCP Registry, include `.mcp/server.json` in your r } ], "repository": { - "url": "https://github.com/yourusername/mymcpserver", + "url": "https://github.com//", "source": "github" } } From 21792d1cc0506ecb1420b853568d9de431614431 Mon Sep 17 00:00:00 2001 From: Dan Moseley Date: Fri, 20 Mar 2026 14:49:09 -0600 Subject: [PATCH 21/29] Add github.com/ to known-domains to allow truncated placeholder URLs The URL regex truncates at angle brackets (invalid in URIs), so placeholder URLs like https://github.com// become https://github.com/ which didn't match any path-scoped entry. Adding github.com/ (with trailing slash) covers only this case without broadly allowing all github.com URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- eng/reference-scanner/known-domains.txt | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/eng/reference-scanner/known-domains.txt b/eng/reference-scanner/known-domains.txt index 28f42d44be..c1f0316119 100644 --- a/eng/reference-scanner/known-domains.txt +++ b/eng/reference-scanner/known-domains.txt @@ -6,7 +6,8 @@ # alongside the skill content. # # Format: one domain per line. Lines starting with # are comments. -# For GitHub, use github.com// to scope to a specific repo. +# Entries containing a slash match URLs with that exact prefix followed by +# '/', '?', '#', or end-of-URL. Entries without a slash match the entire domain. # Standards agentskills.io @@ -29,6 +30,10 @@ developer.android.com developer.apple.com # Repos +# github.com/ (trailing slash required — without it, ALL github.com URLs would be +# allowed) covers placeholder URLs like https://github.com// +# which truncate to https://github.com/ since angle brackets are invalid in URIs. +github.com/ dotnet.github.io github.com/dotnet/csharplang github.com/dotnet/diagnostics From 3c69ef95925bf865610df51ab5ee2747ccb4d96e Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Fri, 20 Mar 2026 15:36:47 -0700 Subject: [PATCH 22/29] Update Docker tag command to use placeholder for server name --- plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md index 3f53a82040..ef2131f5dd 100644 --- a/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md @@ -144,7 +144,7 @@ curl http://localhost:3001/health 3. **Push to container registry:** ```bash # Docker Hub -docker tag mymcpserver:latest /mymcpserver:1.0.0 +docker tag mymcpserver:latest /:1.0.0 docker push /:1.0.0 # Azure Container Registry From c25e3dd10623c5cc2c5af522b2bd37b72359ee5f Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Mon, 23 Mar 2026 13:17:09 -0700 Subject: [PATCH 23/29] Update known domains and fix GitHub username placeholders in documentation --- eng/known-domains.txt | 9 +++++++++ plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md | 4 ++-- .../skills/mcp-csharp-publish/references/mcp-registry.md | 6 +++--- .../mcp-csharp-publish/references/nuget-packaging.md | 6 +++--- 4 files changed, 17 insertions(+), 8 deletions(-) diff --git a/eng/known-domains.txt b/eng/known-domains.txt index 29b3b98b9b..34a9e621d6 100644 --- a/eng/known-domains.txt +++ b/eng/known-domains.txt @@ -47,6 +47,15 @@ github.com/microsoft/perfview github.com/microsoftdocs/visualstudio-docs github.com/NuGet/docs.microsoft.com-nuget +# MCP ecosystem +code.visualstudio.com +github.com/modelcontextprotocol +github.com/open-telemetry/semantic-conventions + +# Example paths used in skill documentation +github.com/yourusername +github.com/username + # Community fluentassertions.com npmjs.com/package/@modelcontextprotocol diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md index ef2131f5dd..76abca1ebd 100644 --- a/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/SKILL.md @@ -197,7 +197,7 @@ brew install mcp-publisher ```json { "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", - "name": "io.github./", + "name": "io.github.username/servername", "description": "Your server description", "version": "1.0.0", "packages": [{ @@ -208,7 +208,7 @@ brew install mcp-publisher "transport": { "type": "stdio" } }], "repository": { - "url": "https://github.com//", + "url": "https://github.com/username/repo", "source": "github" } } diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/mcp-registry.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/mcp-registry.md index 3ec98f115e..ce50cb2284 100644 --- a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/mcp-registry.md +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/mcp-registry.md @@ -62,7 +62,7 @@ Place at `.mcp/server.json` in your repository root: } ], "repository": { - "url": "https://github.com//", + "url": "https://github.com/username/repo", "source": "github" } } @@ -74,8 +74,8 @@ The `name` field must follow a namespace convention based on your authentication | Auth Method | Name Format | Example | |-------------|-------------|---------| -| GitHub | `io.github./` | `io.github.jsmith/weather-server` | -| DNS | `/` | `com.mycompany/weather-server` | +| GitHub | `io.github.{github-username}/{server-name}` | `io.github.jsmith/weather-server` | +| DNS | `{reverse-domain}/{server-name}` | `com.mycompany/weather-server` | ## Publish Workflow diff --git a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md index eb2791943d..739f485320 100644 --- a/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md +++ b/plugins/dotnet-ai/skills/mcp-csharp-publish/references/nuget-packaging.md @@ -23,8 +23,8 @@ Detailed guide for publishing stdio MCP servers as NuGet tool packages. MCP server for interacting with MyService API - https://github.com// - https://github.com// + https://github.com/yourusername/mymcpserver + https://github.com/yourusername/mymcpserver MIT mcp;modelcontextprotocol;ai;llm README.md @@ -131,7 +131,7 @@ If you plan to publish to the MCP Registry, include `.mcp/server.json` in your r } ], "repository": { - "url": "https://github.com//", + "url": "https://github.com/yourusername/mymcpserver", "source": "github" } } From b182f1e56079d31dd50b71e85ddb86986f486e35 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Tue, 24 Mar 2026 16:34:01 -0700 Subject: [PATCH 24/29] Remove McpServerToolType from mcp-csharp-create eval McpServerToolType is only needed with WithToolsFromAssembly(), which is discouraged because it is not Native AOT compatible. Updated eval assertions and rubrics to prefer WithTools() instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/dotnet-ai/mcp-csharp-create/eval.yaml | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/tests/dotnet-ai/mcp-csharp-create/eval.yaml b/tests/dotnet-ai/mcp-csharp-create/eval.yaml index 1ce9f29cac..9af3c58ad9 100644 --- a/tests/dotnet-ai/mcp-csharp-create/eval.yaml +++ b/tests/dotnet-ai/mcp-csharp-create/eval.yaml @@ -6,14 +6,12 @@ scenarios: with proper attributes for LLM discovery. Show me the tool class, Program.cs with stdio transport, and explain how DI works for MCP tools. assertions: - - type: "output_matches" - pattern: "\\[McpServerToolType\\]" - type: "output_matches" pattern: "\\[McpServerTool[\\],\\(]" - type: "output_matches" pattern: "(AddHttpClient|HttpClient)" rubric: - - "Shows a tool class with [McpServerToolType] and [McpServerTool] attributes" + - "Shows a tool class with [McpServerTool] attributes" - "Injects HttpClient via DI (constructor injection or method parameter injection)" - "Includes [Description] attributes on both the method and all parameters" - "Configures Program.cs with AddMcpServer, WithStdioServerTransport, and logging to stderr" @@ -35,7 +33,7 @@ scenarios: - "Configures HTTP transport with WithHttpTransport() and references ModelContextProtocol.AspNetCore" - "Includes MapMcp() in the endpoint configuration" - "Shows a resource class with [McpServerResourceType] and [McpServerResource] attributes including UriTemplate" - - "Shows how to register tools and resources with WithToolsFromAssembly or similar DI registration" + - "Shows how to register tools and resources with WithTools or similar DI registration" timeout: 180 - name: "Create an MCP server with tools, prompts, and proper logging" prompt: | @@ -51,7 +49,7 @@ scenarios: - type: "output_matches" pattern: "(LogToStandardErrorThreshold|stderr)" rubric: - - "Shows a tool class with [McpServerToolType]/[McpServerTool] and HttpClient injected via DI" + - "Shows a tool class with [McpServerTool] and HttpClient injected via DI" - "Shows a prompt class with [McpServerPromptType] and [McpServerPrompt] returning ChatMessage" - "Configures logging to stderr (LogToStandardErrorThreshold) to avoid corrupting stdio transport" - "Adds [Description] attributes on tools, parameters, and prompts for LLM discoverability" From 12891265bc2ac9a2d7e8d548ee1afb6d83061e48 Mon Sep 17 00:00:00 2001 From: Dan Moseley Date: Tue, 24 Mar 2026 20:29:24 -0600 Subject: [PATCH 25/29] Improve eval resilience for mcp-csharp-create and mcp-csharp-debug - Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s for all 3 scenarios. These scenarios consistently time out because the model spends time on bash exploration/scaffolding before writing code. - Add explicit timeout to scenario 3 which was relying on the 120s default. - Broaden mcp-csharp-debug Inspector rubric to accept both the concise single CLI command style and the step-by-step UI configuration walkthrough, since both are correct approaches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/dotnet-ai/mcp-csharp-create/eval.yaml | 5 +++-- tests/dotnet-ai/mcp-csharp-debug/eval.yaml | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/tests/dotnet-ai/mcp-csharp-create/eval.yaml b/tests/dotnet-ai/mcp-csharp-create/eval.yaml index 9af3c58ad9..1a563f413b 100644 --- a/tests/dotnet-ai/mcp-csharp-create/eval.yaml +++ b/tests/dotnet-ai/mcp-csharp-create/eval.yaml @@ -15,7 +15,7 @@ scenarios: - "Injects HttpClient via DI (constructor injection or method parameter injection)" - "Includes [Description] attributes on both the method and all parameters" - "Configures Program.cs with AddMcpServer, WithStdioServerTransport, and logging to stderr" - timeout: 180 + timeout: 360 - name: "Create an HTTP MCP server with tools and resources" prompt: | I need to create a C# MCP server that uses HTTP transport for deployment @@ -34,7 +34,7 @@ scenarios: - "Includes MapMcp() in the endpoint configuration" - "Shows a resource class with [McpServerResourceType] and [McpServerResource] attributes including UriTemplate" - "Shows how to register tools and resources with WithTools or similar DI registration" - timeout: 180 + timeout: 360 - name: "Create an MCP server with tools, prompts, and proper logging" prompt: | I'm building a C# MCP server using stdio transport. I need to add a tool @@ -53,3 +53,4 @@ scenarios: - "Shows a prompt class with [McpServerPromptType] and [McpServerPrompt] returning ChatMessage" - "Configures logging to stderr (LogToStandardErrorThreshold) to avoid corrupting stdio transport" - "Adds [Description] attributes on tools, parameters, and prompts for LLM discoverability" + timeout: 360 diff --git a/tests/dotnet-ai/mcp-csharp-debug/eval.yaml b/tests/dotnet-ai/mcp-csharp-debug/eval.yaml index e635b86e34..0b58def8c5 100644 --- a/tests/dotnet-ai/mcp-csharp-debug/eval.yaml +++ b/tests/dotnet-ai/mcp-csharp-debug/eval.yaml @@ -11,7 +11,7 @@ scenarios: pattern: "(dotnet run|--project)" rubric: - "Shows how to launch MCP Inspector with npx @modelcontextprotocol/inspector" - - "Explains how to connect it to the stdio server using dotnet run" + - "Explains how to connect it to the stdio server using dotnet run — either as a single CLI command (npx @modelcontextprotocol/inspector dotnet run ...) or via the Inspector UI configuration (Transport Type, Command, Arguments)" - "Mentions that the Inspector UI shows tools, prompts, and resources" - "Notes that server logging must go to stderr, not stdout" timeout: 120 From 94c271045801c0bdc348e91aa640a52fb279c308 Mon Sep 17 00:00:00 2001 From: Dan Moseley Date: Tue, 24 Mar 2026 20:52:08 -0600 Subject: [PATCH 26/29] Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- eng/known-domains.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/eng/known-domains.txt b/eng/known-domains.txt index 34a9e621d6..486544091c 100644 --- a/eng/known-domains.txt +++ b/eng/known-domains.txt @@ -7,7 +7,7 @@ # # Format: one domain per line. Lines starting with # are comments. # Entries containing a slash match URLs with that exact prefix followed by -# '/', '?', '#', or end-of-URL. Entries without a slash match the entire domain. +# '/', '?', '#', or end-of-URL. Entries without a slash match the domain and all its subdomains. # Standards dot.net From 869808b608414a573fed9d8806d23f7b513dd0fc Mon Sep 17 00:00:00 2001 From: Dan Moseley Date: Tue, 24 Mar 2026 20:36:08 -0600 Subject: [PATCH 27/29] Add evaluation troubleshooting guide and link from PR comments Add InvestigatingResults.md with: - How to find and download evaluation artifacts from workflow runs - Full results.json schema documentation - Common failure patterns (timeouts, empty output, variance, rubric issues, skill not activated) with diagnosis steps and fixes - Example analysis script for AI agents to parse results Link the doc from: - Reporter.cs: show a link in the PR comment when any skill fails - skill-validator README.md: Output section - CONTRIBUTING.md: CI evaluation section Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/evaluation.yml | 8 + CONTRIBUTING.md | 2 + eng/skill-validator/InvestigatingResults.md | 264 +++++++++++++++++++ eng/skill-validator/README.md | 2 + eng/skill-validator/src/Evaluate/Reporter.cs | 4 + 5 files changed, 280 insertions(+) create mode 100644 eng/skill-validator/InvestigatingResults.md diff --git a/.github/workflows/evaluation.yml b/.github/workflows/evaluation.yml index 304df67b2c..c7e9da03d7 100644 --- a/.github/workflows/evaluation.yml +++ b/.github/workflows/evaluation.yml @@ -642,6 +642,14 @@ jobs: cat summary-body.md echo "" echo "[Full results]($RUN_URL)" + # If any skill failed, add a copy-paste prompt for AI-assisted investigation + if grep -q '❌' summary-body.md; then + RUN_ID="${{ github.run_id }}" + echo "" + echo "> **To investigate failures**, paste this to your AI coding agent:" + echo ">" + echo "> _Download eval artifacts with \`gh run download ${RUN_ID} --repo ${{ github.repository }} --dir /tmp/eval-results\`, then fetch https://raw.githubusercontent.com/${{ github.repository }}/main/eng/skill-validator/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first._" + fi } > consolidated-comment.md cat consolidated-comment.md >> $GITHUB_STEP_SUMMARY diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bfb2b839ab..6ff7fda26d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -288,6 +288,8 @@ dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- evaluate - Tests run automatically on pull requests that modify files under `plugins/`. The evaluation workflow discovers changed plugins and runs the skill-validator for each one. Results are posted as a PR comment and uploaded as build artifacts. +If a scenario fails or regresses, see [Investigating Results](eng/skill-validator/InvestigatingResults.md) for how to download artifacts, interpret `results.json`, and diagnose common failure patterns. + ## Writing style - Be concise and specific. diff --git a/eng/skill-validator/InvestigatingResults.md b/eng/skill-validator/InvestigatingResults.md new file mode 100644 index 0000000000..1ef832e6c3 --- /dev/null +++ b/eng/skill-validator/InvestigatingResults.md @@ -0,0 +1,264 @@ +# Investigating Evaluation Results + +This guide is intended primarily for AI agents investigating skill evaluation failures, though humans will find it useful too. It documents the `results.json` schema, common failure patterns, and recommended fixes. + +## Using this guide with an AI agent + +This document is designed to be read by AI coding agents. When a skill evaluation has failures, the PR comment includes a ready-to-use prompt — just copy and paste it to your AI agent. The agent will download the artifacts, read this guide, analyze the results, and suggest fixes. + +If you need to run the investigation manually, follow the [Quick start](#quick-start) below. + +## Quick start + +1. **Download the results artifact:** `gh run download --repo dotnet/skills --dir ` +2. **Read `summary.md` first** for a quick overview of which scenarios passed/failed +3. **Read `results.json`** for the full metrics, agent output, assertions, and judge reasoning +4. **Identify the failure pattern** using the categories below — most failures match multiple patterns; fix them in priority order (timeouts first, then activation, then quality/rubric issues) +5. **Apply the fix** and re-run with `/evaluate` + +## Finding the artifacts + +### Via CLI (recommended for AI agents) + +Extract the workflow run ID from the **Full results** link in the PR eval comment (e.g., `https://github.com/dotnet/skills/actions/runs/23520818616` → `23520818616`), then: + +```bash +gh run download --repo dotnet/skills --dir /tmp/eval-results +``` + +This downloads all artifacts into subdirectories, each containing `results.json` and `summary.md`. + +### Via browser + +From the PR comment, click the **Full results** link to open the GitHub Actions workflow run. Then: + +1. Click on any job (e.g., `evaluate (mcp-csharp-debug)`) +2. Expand the **Upload results** step +3. Find the `Artifact download URL` in the log output +4. Download and extract + +Alternatively, scroll to the bottom of the workflow run summary page and download from the **Artifacts** section. + +## Understanding `results.json` + +Each file contains a top-level object with: + +| Field | Description | +|-------|-------------| +| `model` | Model used for agent runs | +| `judgeModel` | Model used for judging | +| `timestamp` | When the run started | +| `verdicts[]` | Array of per-skill results | + +### Verdict structure + +Each verdict contains: + +| Field | Description | +|-------|-------------| +| `skillName` | Name of the skill being evaluated | +| `passed` | Overall pass/fail | +| `scenarios[]` | Array of per-scenario comparisons | +| `overfittingResult` | Overfitting analysis (if enabled) | + +### Scenario structure + +Each scenario contains three runs and their comparison: + +| Field | Description | +|-------|-------------| +| `scenarioName` | Human-readable scenario name | +| `baseline` | Run without the skill | +| `skilledIsolated` | Run with only this skill loaded | +| `skilledPlugin` | Run with the full plugin loaded | +| `timedOut` | Whether any run hit the timeout | +| `isolatedImprovementScore` | Weighted improvement (isolated vs baseline) | +| `pluginImprovementScore` | Weighted improvement (plugin vs baseline) | +| `isolatedBreakdown` | Per-metric contribution to the score (see below) | +| `pluginBreakdown` | Per-metric contribution to the score (see below) | +| `pairwiseResult` | Judge's rubric-by-rubric comparison | +| `perRunScores` | Individual run scores (shows variance) | + +### Breakdown fields + +The `isolatedBreakdown` and `pluginBreakdown` objects show how each metric contributed to the improvement score. Each field is a raw delta (not yet weighted). The final score is computed as a weighted sum: + +| Field | Weight | Range | Meaning | +|-------|--------|-------|---------| +| `qualityImprovement` | 0.40 | [-1, 1] | Rubric-based quality delta | +| `overallJudgmentImprovement` | 0.30 | [-1, 1] | Holistic judge assessment delta | +| `taskCompletionImprovement` | 0.15 | {-1, 0, 1} | Did assertions pass? | +| `tokenReduction` | 0.05 | [-1, 1] | Positive = fewer tokens (more efficient) | +| `errorReduction` | 0.05 | [-1, 1] | Positive = fewer errors | +| `toolCallReduction` | 0.025 | [-1, 1] | Positive = fewer tool calls | +| `timeReduction` | 0.025 | [-1, 1] | Positive = faster | + +A `tokenReduction` of -1.0 means the skilled run used ≥2× the baseline's tokens. This is common when a skill is loaded (the skill content itself consumes tokens) but is only -0.05 in the final score, so it rarely causes failure on its own. + +### Run metrics + +Each of `baseline`, `skilledIsolated`, and `skilledPlugin` contains a `metrics` object: + +| Field | Description | +|-------|-------------| +| `timedOut` | Whether this run hit the timeout | +| `wallTimeMs` | Total wall-clock time | +| `taskCompleted` | Whether assertions passed | +| `tokenEstimate` | Total tokens used | +| `turnCount` | Number of agent turns | +| `toolCallCount` | Number of tool calls | +| `toolCallBreakdown` | Tool call counts by tool name | +| `errorCount` | Number of errors during the run | +| `assertionResults[]` | Per-assertion pass/fail with messages | +| `agentOutput` | The agent's final text output | + +## Common failure patterns + +### 1. Timeout with empty output + +**Symptoms:** +- `timedOut: true` +- `agentOutput` is empty or just `\n\n` +- All assertions fail +- `toolCallBreakdown` shows `bash` usage + +**Cause:** The model spent its entire time budget running shell commands (e.g., `dotnet new`, `dotnet add package`, exploring NuGet contents) and never produced user-facing text. + +**Fixes:** +- **Increase `timeout`** in `eval.yaml` — 180s is often not enough for scenarios that involve code generation. Try 360s. +- **Restructure the prompt** to discourage bash exploration (e.g., "Show me the code" rather than "Create a project") +- **Add `reject_tools: ["bash"]`** if the scenario should be answerable without shell commands + +### 2. Baseline already bad + +**Symptoms:** +- Baseline scores are very low (1.0–2.0/5) +- Skilled scores are also low +- Quality improvement shows 0 or negative + +**Cause:** The question is too hard for the model even without the skill. The skill can't fix what the model can't do. + +**Fixes:** +- Simplify the scenario prompt +- Verify the baseline is working by examining `baseline.metrics.agentOutput` +- Consider whether the scenario is testing the right thing + +### 3. High variance across runs + +**Symptoms:** +- `perRunScores` contains both positive and negative values (e.g., `[0.07, -0.85, 0.04]`) +- A spread greater than ~0.3 between min and max scores suggests problematic variance +- Results flip between passing and failing across eval runs +- Isolated and plugin scores disagree + +**Cause:** LLM non-determinism. The model takes different strategies on different runs. + +**Fixes:** +- **Increase `--runs`** for more statistical stability (5 is the default; consider 7–10 for noisy scenarios) +- **Tighten the prompt** to reduce the space of valid strategies +- **Add `setup.files`** to give the model concrete files to work with rather than letting it scaffold from scratch + +### 4. Quality unchanged but weighted score negative + +**Symptoms:** +- Footnote says "Quality unchanged but weighted score is -X% due to: judgment, tokens, tool calls" +- The skilled output is roughly as good as baseline + +**Cause:** The skill adds token overhead (the skill content itself uses tokens) but doesn't improve quality enough to offset it. + +**Fixes:** +- **Improve the skill content** to produce clearly better output for this scenario +- **Reduce skill size** — shorter skills have less token overhead +- **Check if the rubric matches** what the skill actually teaches + +### 5. Skill not activated + +**Symptoms:** +- Skills Loaded column shows `⚠️ NOT ACTIVATED` +- Skilled run has near-zero tokens (e.g., <100), 0 turns, 0 tools +- The `turnCount` being 0 is the clearest signal — a small token count with 0 turns indicates the skill was loaded but the agent never ran + +**Cause:** The agent runtime didn't select the skill for this prompt. The skill's frontmatter `description` didn't match. + +**Fixes:** +- Update the skill's `description` in SKILL.md frontmatter to better match the scenario prompt +- Make sure the description includes keywords from the scenario + +### 6. Rubric penalizes valid alternatives + +**Symptoms:** +- Pairwise judge picks baseline over skill +- Both outputs are correct but use different approaches +- `pairwiseResult.rubricResults` shows the rubric criterion is too narrow + +**Cause:** The rubric item favors one specific approach (e.g., step-by-step UI walkthrough) over an equally valid alternative (e.g., single CLI command). + +**Fixes:** +- **Broaden the rubric** to explicitly accept multiple valid approaches +- Example: Instead of `"Shows step-by-step UI configuration"`, use `"Explains how to connect — either as a single CLI command or via the UI configuration"` + +### 7. Judge regressions on close calls + +**Symptoms:** +- `overallJudgmentImprovement` is -0.4 even though quality scores are similar +- Pairwise judge is inconsistent between position-swapped runs + +**Cause:** When outputs are nearly equal, the judge's position bias can dominate. The position-swap mitigation defaults to "tie" on inconsistency, but the weighted scoring still penalizes. + +**Fixes:** +- This is usually noise — re-run the eval to see if it persists +- If it consistently happens, improve the skill to produce clearly differentiated output + +## When multiple patterns apply + +Most failing scenarios match 2–3 patterns simultaneously (e.g., timeout + token overhead + high variance). Fix them in this priority order: + +1. **Timeouts (#1)** — if the model can't finish, nothing else matters. Increase timeout first. +2. **Skill not activated (#5)** — if the skill never loaded, fix the description before tuning anything else. +3. **Baseline already bad (#2)** — if the baseline scores ≤2.0/5, the scenario may need simplification regardless of the skill. +4. **High variance (#3)** — if `perRunScores` are unstable, a single eval run is unreliable. Re-run before concluding the skill is broken. +5. **Rubric/judgment issues (#6, #7)** — once the runs are stable, tune the rubric. +6. **Token overhead (#4)** — only optimize if quality is already good but the weighted score is marginally negative. + +## Analyzing results with an AI agent + +The `results.json` file is designed to be machine-readable. An AI agent can: + +1. **Parse the JSON** and extract metrics for each scenario +2. **Compare baseline vs skilled** metrics to identify regressions +3. **Read `agentOutput`** to see what the model actually produced +4. **Check `assertionResults`** to see which assertions failed +5. **Read `pairwiseResult.rubricResults`** for the judge's per-criterion reasoning +6. **Examine `perRunScores`** to assess variance +7. **Look at `toolCallBreakdown`** to understand what the model spent time on +8. **Cross-reference `isolatedBreakdown`** to see which metrics drove the score + +### Example analysis script + +```python +import json + +def analyze(path): + with open(path) as f: + data = json.load(f) + for verdict in data['verdicts']: + for scenario in verdict['scenarios']: + name = scenario['scenarioName'] + bl = scenario['baseline']['metrics'] + sk = scenario['skilledIsolated']['metrics'] + print(f"--- {name} ---") + print(f" Baseline: timedOut={bl['timedOut']}, output={len(bl.get('agentOutput',''))} chars") + print(f" Skilled: timedOut={sk['timedOut']}, output={len(sk.get('agentOutput',''))} chars") + print(f" Improvement: {scenario.get('isolatedImprovementScore', 0):.1%}") + for a in bl.get('assertionResults', []): + status = 'PASS' if a['passed'] else 'FAIL' + print(f" Baseline assertion [{status}]: {a['message']}") + +analyze('results.json') +``` + +## See also + +- [skill-validator README](README.md) — CLI usage, eval file format, scoring weights +- [Overfitting detection](OverfittingDetection.md) — how overfitting scores are computed +- [CONTRIBUTING.md](../../CONTRIBUTING.md) — writing eval files and running tests locally diff --git a/eng/skill-validator/README.md b/eng/skill-validator/README.md index 2b96ff5ba3..b1e30527ca 100644 --- a/eng/skill-validator/README.md +++ b/eng/skill-validator/README.md @@ -155,6 +155,8 @@ Results are displayed in the console with color-coded scores and metric deltas. - `junit` — `results.xml` with JUnit XML test results - `markdown` — `summary.md` with a results table, plus per-skill directories with per-scenario judge reports +See [Investigating Results](InvestigatingResults.md) for how to diagnose poor scores, download artifacts, and interpret `results.json`. + ### Consolidating results across matrix jobs When evaluating multiple plugins in parallel CI matrix jobs, use the `consolidate` subcommand to merge individual `results.json` files into a single markdown summary: diff --git a/eng/skill-validator/src/Evaluate/Reporter.cs b/eng/skill-validator/src/Evaluate/Reporter.cs index 59f06679c0..7e08a37b60 100644 --- a/eng/skill-validator/src/Evaluate/Reporter.cs +++ b/eng/skill-validator/src/Evaluate/Reporter.cs @@ -609,6 +609,10 @@ public static string GenerateMarkdownSummary( sb.AppendLine($"\nModel: {model ?? "unknown"} | Judge: {judgeModel ?? "unknown"}"); + bool anyFailure = verdicts.Any(v => !v.Passed); + if (anyFailure) + sb.AppendLine("\n> 📖 See [InvestigatingResults.md](https://github.com/dotnet/skills/blob/main/eng/skill-validator/InvestigatingResults.md) for how to diagnose failures — or use the copy-paste prompt below."); + return sb.ToString(); } From 4a2d6ce6b838e742a9260ac0c9a3b3f192414339 Mon Sep 17 00:00:00 2001 From: Dan Moseley Date: Tue, 24 Mar 2026 21:04:41 -0600 Subject: [PATCH 28/29] Update InvestigatingResults.md --- eng/skill-validator/InvestigatingResults.md | 1 + 1 file changed, 1 insertion(+) diff --git a/eng/skill-validator/InvestigatingResults.md b/eng/skill-validator/InvestigatingResults.md index 1ef832e6c3..3303743284 100644 --- a/eng/skill-validator/InvestigatingResults.md +++ b/eng/skill-validator/InvestigatingResults.md @@ -183,6 +183,7 @@ Each of `baseline`, `skilledIsolated`, and `skilledPlugin` contains a `metrics` **Fixes:** - Update the skill's `description` in SKILL.md frontmatter to better match the scenario prompt - Make sure the description includes keywords from the scenario +- Check the scenario itself has sufficient information that the agent can reason that it needs the skill. (It should not cheat and suggest the skill.) ### 6. Rubric penalizes valid alternatives From 511cb8d37e0c746d6e98dcd77df899782604d12c Mon Sep 17 00:00:00 2001 From: Dan Moseley Date: Tue, 24 Mar 2026 21:06:59 -0600 Subject: [PATCH 29/29] Broaden debug 'failing tool' rubric to accept multiple debugging approaches The rubric criterion 'Shows how to attach a debugger' was too narrow. The skilled answer correctly focused on the #1 cause (stdout pollution) but scored low because it didn't show a specific 'attach to process' flow. Broadened to accept any valid debugging approach: attaching, Debugger.Launch(), or launch.json configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/dotnet-ai/mcp-csharp-debug/eval.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/dotnet-ai/mcp-csharp-debug/eval.yaml b/tests/dotnet-ai/mcp-csharp-debug/eval.yaml index 0b58def8c5..cf4b1fdaa6 100644 --- a/tests/dotnet-ai/mcp-csharp-debug/eval.yaml +++ b/tests/dotnet-ai/mcp-csharp-debug/eval.yaml @@ -44,6 +44,6 @@ scenarios: rubric: - "Explains that stdout is reserved for MCP protocol in stdio mode" - "Recommends checking that all logging goes to stderr" - - "Shows how to attach a debugger to the running server process" - - "Suggests using MCP Inspector or VS Code debug config for step-through debugging" + - "Shows how to further diagnose the issue — such as attaching a debugger to the running process, using Debugger.Launch(), or configuring launch.json to debug the server" + - "Suggests using MCP Inspector or VS Code debug config to test or step through the server" timeout: 120