From 157db6c918e97c47061198239c771993e5693957 Mon Sep 17 00:00:00 2001 From: Jeff Schwartz Date: Fri, 27 Feb 2026 16:57:52 -0800 Subject: [PATCH 1/3] Add dotnet-ai-ml skill Add SKILL.md with guidance for .NET AI/ML library selection and eval.yaml with validation scenarios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- plugins/dotnet/skills/dotnet-ai-ml/SKILL.md | 357 ++++++++++++++++++++ tests/dotnet/dotnet-ai-ml/eval.yaml | 240 +++++++++++++ 2 files changed, 597 insertions(+) create mode 100644 plugins/dotnet/skills/dotnet-ai-ml/SKILL.md create mode 100644 tests/dotnet/dotnet-ai-ml/eval.yaml diff --git a/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md b/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md new file mode 100644 index 0000000000..d6c95a694e --- /dev/null +++ b/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md @@ -0,0 +1,357 @@ +--- +name: dotnet-ai-ml +description: Guides technology selection and implementation of AI and ML features in .NET 8+ applications using ML.NET, Microsoft.Extensions.AI, Microsoft Agent Framework, GitHub Copilot SDK, ONNX Runtime, and LLamaSharp. Use when adding classification, regression, LLM integration, RAG, agentic workflows, Copilot extensions, or custom model inference to a .NET project. +--- + +# .NET AI and Machine Learning + +Help developers select the right AI/ML technology for their task and produce production-ready .NET code that follows efficiency, determinism, and cost-control guardrails. The skill covers the full spectrum from classic ML (ML.NET) through modern LLM orchestration (Microsoft Agent Framework) to local inference (ONNX Runtime, LLamaSharp). + +## When to Use + +- Adding classification, regression, clustering, anomaly detection, or recommendation to a .NET application +- Integrating LLM capabilities (text generation, summarization, reasoning) into a .NET service +- Building RAG (retrieval-augmented generation) pipelines with vector search +- Deploying pre-trained or fine-tuned models via ONNX Runtime +- Implementing agentic workflows with tool calling and orchestration +- Choosing between ML.NET, LLM APIs, ONNX Runtime, or LLamaSharp for a given task + +## When Not to Use + +- The project targets .NET Framework (not .NET 8+) +- The task is pure data engineering or ETL with no ML/AI component +- The project needs a custom deep learning training loop (use Python with PyTorch/TensorFlow, then export to ONNX for .NET inference) + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Task description | Yes | What the AI/ML feature should accomplish (e.g., "classify support tickets", "summarize documents") | +| Data description | Yes | Type and shape of input data (structured/tabular, unstructured text, images, mixed) | +| Deployment constraints | No | Cloud vs. local, latency SLO, cost budget, offline requirements | +| Existing project context | No | Current .csproj, existing packages, target framework | + +## Workflow + +### Step 1: Classify the task using the decision tree + +Evaluate the developer's task against this decision tree and select the appropriate technology. State which branch applies and why. + +| Task type | Technology | Rationale | +|-----------|-----------|-----------| +| Structured/tabular data: classification, regression, clustering, anomaly detection, recommendation | **ML.NET** (`Microsoft.ML`) | Deterministic, reproducible, no cloud dependency, purpose-built for these tasks | +| Natural language understanding, generation, summarization, reasoning, workflow over unstructured text | **LLM via Microsoft.Extensions.AI** with **Microsoft Agent Framework** for orchestration | Requires language model capabilities beyond pattern matching | +| Building GitHub Copilot extensions, custom agents, or developer workflow tools | **GitHub Copilot SDK** (`GitHub.Copilot.SDK`) | Integrates with the Copilot agent runtime for IDE and CLI extensibility | +| Running a pre-trained or fine-tuned custom model in production | **ONNX Runtime** (`Microsoft.ML.OnnxRuntime`) | Hardware-accelerated inference, model-format agnostic | +| Local/offline LLM inference with no cloud dependency | **LLamaSharp** with quantized GGUF models | Privacy-sensitive, air-gapped, or cost-constrained scenarios | +| Semantic search, RAG, or embedding storage | **Microsoft.Extensions.VectorData.Abstractions** + a vector database provider (e.g., Azure AI Search, Milvus, MongoDB, pgvector, Pinecone, Qdrant, Redis, SQL) | Provider-agnostic abstractions for vector similarity search; pair with a database-specific connector package (many are moving to community toolkits) | +| Ingesting, chunking, and loading documents into a vector store | **Microsoft.Extensions.AI.DataIngestion** (preview) + MEVD | Handles document parsing, text chunking, embedding generation, and upserting into a vector database; pairs with Microsoft.Extensions.VectorData.Abstractions | +| Both structured ML predictions AND natural language reasoning | **Hybrid**: ML.NET for predictions + LLM for reasoning layer | Keep loosely coupled; ML.NET handles deterministic scoring, LLM adds explanation | + +**Critical rule:** Do NOT use an LLM for tasks that ML.NET handles well (classification on tabular data, regression, clustering). LLMs are slower, more expensive, and non-deterministic for these tasks. + +### Step 1b: Select the correct library layer + +After identifying the task type, select the right library layer. These libraries form a stack — each builds on the one below it. Using the wrong layer is a major source of non-deterministic agent behavior. + +| Layer | Library | NuGet package | Use when | +|-------|---------|---------------|----------| +| **Abstraction** | Microsoft.Extensions.AI (MEAI) | `Microsoft.Extensions.AI` | You need a provider-agnostic interface for chat, embeddings, or tool calling. This is the foundation — always include it. Use it directly for simple prompt-in/response-out scenarios with no orchestration. | +| **Provider SDK** | OpenAI, Azure.AI.OpenAI, Azure.AI.Inference, OllamaSharp | `OpenAI`, `Azure.AI.OpenAI`, `Azure.AI.Inference`, `OllamaSharp` | You need a concrete LLM provider implementation. These wire into MEAI via `AddChatClient`. Use `OpenAI` for direct OpenAI access, `Azure.AI.OpenAI` for Azure OpenAI, `Azure.AI.Inference` for Azure AI Foundry / GitHub Models, or `OllamaSharp` for local Ollama. Use directly only if you need provider-specific features not exposed through MEAI. | +| **Orchestration** | Microsoft Agent Framework | `Microsoft.Agents.AI` | You need multi-step agent workflows, tool execution loops, multi-agent coordination, durable context, or graph-based workflows. Builds on top of MEAI. | +| **Copilot integration** | GitHub Copilot SDK | `GitHub.Copilot.SDK` | You are building extensions or tools that integrate with the GitHub Copilot runtime — custom agents, IDE extensions, or developer workflow automation that leverages the Copilot agent platform. | + +#### Decision rules for library selection + +1. **Start with MEAI.** Every AI integration begins with `Microsoft.Extensions.AI` for the `IChatClient` / `IEmbeddingGenerator` abstractions. This ensures provider-swappability and testability. + +2. **Add a provider SDK** (`OpenAI`, `Azure.AI.OpenAI`) as the concrete implementation behind MEAI. Do not call the provider SDK directly in business logic — always go through the MEAI abstraction. + +3. **Add Agent Framework only when you need orchestration.** If the task is a single prompt → response, MEAI is sufficient. Add `Microsoft.Agents.AI` when you need: + - Tool/function calling loops (agent decides which tools to invoke) + - Multi-step reasoning with state carried across turns + - Multi-agent collaboration with handoff protocols + - Graph-based or durable workflows + +4. **Add Copilot SDK only when building Copilot extensions.** Use `GitHub.Copilot.SDK` when the goal is to build a custom agent or tool that runs inside the GitHub Copilot platform (CLI, IDE, or Copilot Chat). This is not a general-purpose LLM orchestration library — it is specifically for Copilot extensibility. + +5. **Never skip layers.** Do not use Agent Framework without MEAI underneath. Do not call `HttpClient` to OpenAI alongside MEAI in the same workflow. Each layer depends on the one below it. + +> **Why this matters for determinism:** Agents frequently produce non-deterministic code when the library boundaries are unclear. An agent may mix raw `OpenAI` SDK calls with MEAI `IChatClient`, or add Agent Framework for a task that only needs a single MEAI call. This skill enforces clear layering rules so the agent selects the minimal correct layer every time. + +### Step 2: Select packages and set up the project + +Install only the packages needed for the selected technology branch. Do not mix competing abstractions. + +#### Classic ML packages + +```xml + + + + + + +``` + +> **Do NOT use** Accord.NET — it is archived and unmaintained. + +#### Modern AI packages + +```xml + + + + + + + + + + + + + + + + + + + + + + +``` + +> **Stack coherence rule:** Never mix raw SDK calls (`HttpClient` to OpenAI) with `Microsoft.Extensions.AI`, Microsoft Agent Framework, or Copilot SDK in the same workflow. Pick one abstraction layer per workflow boundary and commit to it. See Step 1b for the layering rules. + +#### Register services with dependency injection + +All AI/ML services must be registered via DI. Never instantiate clients directly in business logic. + +```csharp +// Configuration via IOptions +services.Configure(configuration.GetSection("AI")); + +// Register the AI client through the abstraction +services.AddChatClient(builder => builder + .UseOpenAIChatClient("gpt-4o-mini-2024-07-18")); +``` + +### Step 3: Implement with guardrails + +Apply the guardrails for the selected technology branch. Every generated implementation must follow these rules. + +#### Classic ML guardrails + +1. **Reproducibility**: Always set a random seed in the ML context: + ```csharp + var mlContext = new MLContext(seed: 42); + ``` + +2. **Data splitting**: Always split into train/test (and optionally validation). Never evaluate on training data: + ```csharp + var split = mlContext.Data.TrainTestSplit(data, testFraction: 0.2); + ``` + +3. **Metrics logging**: Always compute and log evaluation metrics appropriate to the task: + ```csharp + var metrics = mlContext.BinaryClassification.Evaluate(predictions); + logger.LogInformation("AUC: {Auc:F4}, F1: {F1:F4}", metrics.AreaUnderRocCurve, metrics.F1Score); + ``` + +4. **AutoML first**: Prefer `mlContext.Auto()` for initial model selection, then refine manually. + +5. **PredictionEngine pooling**: In ASP.NET Core, always use the pooled prediction engine — never a singleton: + ```csharp + services.AddPredictionEnginePool() + .FromFile(modelPath); + ``` + +#### LLM integration guardrails + +1. **Temperature**: Always set explicitly. Use `0` for factual/deterministic tasks: + ```csharp + var options = new ChatOptions + { + Temperature = 0f, + MaxOutputTokens = 1024, + }; + ``` + +2. **Structured output**: Always parse LLM output into strongly-typed objects with fallback handling: + ```csharp + var result = await chatClient.GetResponseAsync(prompt, options, cancellationToken); + ``` + +3. **Retry logic**: Always implement retry with exponential backoff: + ```csharp + services.AddChatClient(builder => builder + .UseOpenAIChatClient(modelId) + .Use(new RetryingChatClient(maxRetries: 3))); + ``` + +4. **Cost control**: Always estimate and log token usage. Choose the smallest model tier that meets quality requirements (e.g., `gpt-4o-mini` before `gpt-4o`). + +5. **Secret management**: Never hardcode API keys. Use Azure Key Vault, user-secrets, or environment variables: + ```csharp + var apiKey = configuration["AI:ApiKey"] + ?? throw new InvalidOperationException("AI:ApiKey not configured"); + ``` + +6. **Model version pinning**: Specify exact model versions to reduce behavioral drift: + ```csharp + // Pin to a specific dated version, not just "gpt-4o" + var modelId = "gpt-4o-2024-08-06"; + ``` + +#### Agentic workflow guardrails + +1. **Iteration limits**: Always cap agentic loops to prevent runaway execution: + ```csharp + var settings = new AgentInvokeOptions + { + MaximumIterations = 10, + }; + ``` + +2. **Cost ceiling**: Implement a token budget per execution and terminate when reached. + +3. **Observability**: Log every agent step — tool selected, input, output, and reasoning: + ```csharp + await foreach (var message in agent.InvokeStreamingAsync(history, settings)) + { + logger.LogDebug("Agent step: {Role} {Content}", message.Role, message.Content); + } + ``` + +4. **Tool schemas**: Define explicit tool/function schemas with descriptions. Never rely on implicit tool discovery. + +5. **Simplicity preference**: Prefer single-agent with tools over multi-agent unless the task genuinely requires agent collaboration. + +#### RAG guardrails + +1. **Embedding caching**: Never re-embed the same content on every query. Cache embeddings in the vector store. + +2. **Chunking strategy**: Use semantic chunking (split on paragraph/section boundaries) over fixed-size chunking. Ensure chunks have enough context to be useful on their own. + +3. **Relevance thresholds**: Do not inject low-relevance chunks into context. Set a minimum similarity score: + ```csharp + var results = await vectorStore.SearchAsync(query, new VectorSearchOptions + { + Top = 5, + MinimumScore = 0.75f, + }); + ``` + +4. **Source attribution**: Track which chunks contributed to the final response. Include source references in the output. + +5. **Batch embeddings**: Batch embedding API calls where possible to reduce latency and cost. + +### Step 4: Handle non-determinism + +When the solution involves LLM calls or agentic workflows, explicitly address non-determinism: + +1. **Acknowledge it**: Inform the developer that LLM outputs are non-deterministic even at temperature 0 (due to batching, quantization, and model updates). + +2. **Validate outputs**: Implement schema validation and content assertion checks on every LLM response. + +3. **Graceful degradation**: Design a fallback path for when the LLM returns unexpected, malformed, or empty output: + ```csharp + var response = await chatClient.GetResponseAsync(prompt, options); + if (response is null || !response.IsValid()) + { + logger.LogWarning("LLM returned invalid response, falling back to rule-based classifier"); + return ruleBasedClassifier.Classify(input); + } + ``` + +4. **Evaluation harness**: For any prompt that will be iterated on, recommend creating a golden dataset and evaluation scaffold to measure prompt quality over time. + +5. **Model version pinning**: Pin to specific dated model versions (e.g., `gpt-4o-2024-08-06`) to reduce drift between deployments. + +### Step 5: Apply performance and cost controls + +1. **Connection pooling**: Use `IHttpClientFactory` and DI-managed clients for all external services. + +2. **Response caching**: Cache repeated or similar queries. Consider semantic caching for LLM responses where appropriate. + +3. **Streaming**: Use `IAsyncEnumerable` for LLM responses in user-facing scenarios to reduce time-to-first-token: + ```csharp + await foreach (var update in chatClient.GetStreamingResponseAsync(prompt, options)) + { + yield return update.Text; + } + ``` + +4. **Health checks**: Implement health checks for external AI service dependencies: + ```csharp + services.AddHealthChecks() + .AddCheck("openai"); + ``` + +5. **ML.NET prediction pooling**: In web applications, always use `PredictionEnginePool`, never a single `PredictionEngine` instance (it is not thread-safe). + +### Step 6: Validate the implementation + +1. Build the project and verify no warnings: + ```bash + dotnet build -c Release -warnaserror + ``` + +2. Run tests, including integration tests that validate AI/ML behavior: + ```bash + dotnet test -c Release + ``` + +3. For ML.NET pipelines, verify that evaluation metrics meet the project's quality bar and that the model can be serialized and loaded correctly. + +4. For LLM integrations, verify that structured output parsing handles both valid and malformed responses. + +5. For RAG pipelines, verify that retrieval returns relevant results and that irrelevant chunks are filtered out. + +## Validation + +- [ ] Technology selection follows the decision tree — LLMs are not used for tasks ML.NET handles +- [ ] All AI/ML services are registered via dependency injection +- [ ] Configuration uses `IOptions` pattern — no hardcoded values +- [ ] API keys are loaded from secure sources — not in source code or committed config files +- [ ] ML.NET pipelines set a random seed and split data for evaluation +- [ ] LLM calls set temperature, max tokens, and retry logic explicitly +- [ ] Agentic workflows have iteration limits and cost ceilings +- [ ] RAG pipelines implement chunking, relevance thresholds, and source attribution +- [ ] Non-deterministic outputs have validation and fallback paths +- [ ] `dotnet build -c Release -warnaserror` completes cleanly + +## Anti-Patterns to Reject + +When reviewing or generating code, flag and redirect the developer if any of these patterns are detected: + +| Anti-pattern | Redirect | +|-------------|----------| +| Using an LLM for classification on structured/tabular data | Use ML.NET instead — it is faster, cheaper, and deterministic | +| Calling LLM APIs without retry or timeout logic | Add `RetryingChatClient` or Polly-based retry with exponential backoff | +| Storing API keys in `appsettings.json` committed to source control | Use user-secrets (dev), environment variables, or Azure Key Vault (prod) | +| Using Accord.NET for new projects | Migrate to ML.NET — Accord.NET is archived and unmaintained | +| Building custom neural networks in .NET from scratch | Use a pre-trained model via ONNX Runtime or call an LLM API | +| RAG without chunking strategy or relevance filtering | Implement semantic chunking and set a minimum similarity score threshold | +| Agentic loops without iteration limits or cost ceilings | Add `MaximumIterations` and a token budget ceiling | +| Mixing `Microsoft.Extensions.AI` with raw `HttpClient` calls to the same provider | Pick one abstraction layer and commit to it | +| Using Agent Framework for a single prompt→response call | Use MEAI `IChatClient` directly — Agent Framework is for multi-step orchestration | +| Using Copilot SDK for general-purpose LLM apps | Copilot SDK is for Copilot platform extensions only — use MEAI + Agent Framework for standalone apps | +| Calling OpenAI SDK directly in business logic instead of through MEAI | Register the provider via `AddChatClient` and depend on `IChatClient` in business code | +| Using `PredictionEngine` as a singleton in ASP.NET Core | Use `PredictionEnginePool` — `PredictionEngine` is not thread-safe | +| Using `Func>` for delegates with ref struct parameters | Define a custom delegate type — ref structs cannot be generic type arguments | + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Over-engineering with LLMs | Start with the simplest approach (rules, ML.NET) and add LLM capability only when simpler methods fall short | +| Evaluating ML models on training data | Always use `TrainTestSplit` and report metrics on the held-out test set | +| LLM output drift between deployments | Pin to specific dated model versions (e.g., `gpt-4o-2024-08-06`) | +| Token cost surprises | Set `MaxOutputTokens`, log token counts per request, and alert on budget thresholds | +| Non-reproducible ML training | Set `MLContext(seed: N)` and version your training data alongside the code | +| RAG returning irrelevant context | Set a minimum similarity score and limit the number of injected chunks | +| Cold start latency on ML.NET models | Pre-warm the `PredictionEnginePool` during application startup | +| Microsoft Agent Framework + raw OpenAI SDK in same class | Choose one orchestration layer per workflow boundary | diff --git a/tests/dotnet/dotnet-ai-ml/eval.yaml b/tests/dotnet/dotnet-ai-ml/eval.yaml new file mode 100644 index 0000000000..1a711bdbc1 --- /dev/null +++ b/tests/dotnet/dotnet-ai-ml/eval.yaml @@ -0,0 +1,240 @@ +scenarios: + - name: "ML.NET classification on tabular data" + prompt: "Add a support ticket classifier to this .NET 8 Web API. Tickets have a Subject (string), Description (string), Priority (int 1-3), and Category should be predicted as one of: Billing, Technical, General. Use the provided training data CSV. Show me the full implementation including model training, evaluation metrics, and a prediction endpoint." + setup: + files: + - path: "TicketClassifier/TicketClassifier.csproj" + content: | + + + net8.0 + enable + enable + + + - path: "TicketClassifier/Program.cs" + content: | + var builder = WebApplication.CreateBuilder(args); + builder.Services.AddControllers(); + var app = builder.Build(); + app.MapControllers(); + app.Run(); + - path: "TicketClassifier/Data/tickets.csv" + content: | + Subject,Description,Priority,Category + "Cannot login","I am unable to login to my account",1,Technical + "Billing error","I was charged twice for my subscription",2,Billing + "How to reset password","I need help resetting my password",3,General + "Server down","The API server is returning 500 errors",1,Technical + "Refund request","I would like a refund for last month",2,Billing + "Feature question","Does the product support SSO?",3,General + assertions: + - type: "output_contains" + value: "MLContext" + - type: "output_contains" + value: "seed" + - type: "output_contains" + value: "TrainTestSplit" + - type: "exit_success" + rubric: + - "Uses ML.NET (Microsoft.ML) for classification — does NOT use an LLM for this structured data task" + - "Sets a random seed on MLContext for reproducibility" + - "Splits data into train/test sets and evaluates on held-out test data" + - "Logs or reports evaluation metrics appropriate for multiclass classification (e.g., MicroAccuracy, MacroAccuracy, LogLoss)" + - "Uses PredictionEnginePool in the ASP.NET Core endpoint — not a singleton PredictionEngine" + timeout: 180 + + - name: "LLM integration with MEAI abstraction" + prompt: "Add a document summarization endpoint to this .NET 8 Web API. The endpoint should accept a document text and return a concise summary. Use Azure OpenAI as the LLM provider. Make sure to follow best practices for production use." + setup: + files: + - path: "DocSummary/DocSummary.csproj" + content: | + + + net8.0 + enable + enable + + + - path: "DocSummary/Program.cs" + content: | + var builder = WebApplication.CreateBuilder(args); + builder.Services.AddControllers(); + var app = builder.Build(); + app.MapControllers(); + app.Run(); + - path: "DocSummary/appsettings.json" + content: | + { + "Logging": { + "LogLevel": { + "Default": "Information" + } + } + } + assertions: + - type: "output_contains" + value: "IChatClient" + - type: "output_contains" + value: "Temperature" + - type: "output_not_contains" + value: "hardcoded" + - type: "exit_success" + rubric: + - "Uses Microsoft.Extensions.AI (IChatClient) as the abstraction layer — does not call Azure.AI.OpenAI or OpenAI SDK directly in business logic" + - "Registers the AI client via dependency injection using AddChatClient" + - "Sets Temperature and MaxOutputTokens explicitly in ChatOptions" + - "Implements retry logic (e.g., RetryingChatClient or Polly-based)" + - "Loads API keys from configuration or environment — not hardcoded in source" + - "Pins to a specific dated model version (e.g., gpt-4o-2024-08-06) rather than an unversioned alias" + timeout: 180 + + - name: "Reject LLM for tabular classification" + prompt: "I have a .NET 8 project with a CSV of customer data (Age, Income, Region, PurchaseHistory columns). I want to predict whether a customer will churn (yes/no). Please add churn prediction using GPT-4 to classify each row." + setup: + files: + - path: "ChurnPredictor/ChurnPredictor.csproj" + content: | + + + net8.0 + enable + enable + + + - path: "ChurnPredictor/Program.cs" + content: | + var builder = WebApplication.CreateBuilder(args); + var app = builder.Build(); + app.MapGet("/", () => "Hello"); + app.Run(); + - path: "ChurnPredictor/Data/customers.csv" + content: | + Age,Income,Region,PurchaseHistory,Churned + 25,45000,East,12,No + 45,80000,West,3,Yes + 35,60000,North,8,No + 55,90000,South,1,Yes + 30,50000,East,15,No + assertions: + - type: "output_contains" + value: "ML.NET" + - type: "exit_success" + rubric: + - "Redirects the user away from using GPT-4/LLM for this tabular classification task" + - "Recommends ML.NET as the appropriate technology with a clear rationale (faster, cheaper, deterministic)" + - "Provides an ML.NET-based binary classification implementation instead" + - "Follows the decision tree from the skill: structured tabular data → ML.NET" + timeout: 180 + + - name: "Agentic workflow with guardrails" + prompt: "Build a .NET 8 console app that uses an AI agent to research a topic by searching the web, then summarize findings. The agent should have access to a web search tool and a note-taking tool. Use the Microsoft Agent Framework." + setup: + files: + - path: "ResearchAgent/ResearchAgent.csproj" + content: | + + + net8.0 + Exe + enable + enable + + + - path: "ResearchAgent/Program.cs" + content: | + Console.WriteLine("Research Agent Starting..."); + assertions: + - type: "output_contains" + value: "MaximumIterations" + - type: "output_contains" + value: "Microsoft.Extensions.AI" + - type: "exit_success" + rubric: + - "Uses Microsoft Agent Framework (Microsoft.Agents.AI) for orchestration — not raw LLM calls in a loop" + - "Builds on top of Microsoft.Extensions.AI (IChatClient) as the foundation layer" + - "Sets MaximumIterations to cap the agentic loop and prevent runaway execution" + - "Defines explicit tool/function schemas with descriptions for each tool" + - "Implements observability by logging agent steps (tool selected, input, output)" + - "Mentions or implements a cost ceiling or token budget" + timeout: 180 + + - name: "Natural-language scenario decomposition — RAG chatbot" + prompt: "I want to build a web-based LLM driven chatbot using C# that allows customers to get help with our internal product PDF documentation using Google Gemini and Blazor. I want to use Postgres for data storage along with just a file folder for our documentation. Show me your plan before continuing with the implementation." + setup: + files: + - path: "HelpChatbot/HelpChatbot.csproj" + content: | + + + net8.0 + enable + enable + + + - path: "HelpChatbot/Program.cs" + content: | + var builder = WebApplication.CreateBuilder(args); + var app = builder.Build(); + app.Run(); + - path: "HelpChatbot/Docs/product-guide.pdf" + content: "(placeholder PDF content)" + assertions: + - type: "output_contains" + value: "Microsoft.Extensions.AI" + - type: "exit_success" + rubric: + - "Presents a plan or architecture breakdown before diving into code — decomposes the vague request into identifiable capability needs (chat, document ingestion, vector search, embedding, UI)" + - "Selects Microsoft.Extensions.AI (MEAI) as the abstraction layer and identifies a concrete provider for Google Gemini (e.g., Azure.AI.Inference or a Gemini-compatible SDK)" + - "Identifies the need for document ingestion and chunking of PDF files — references Microsoft.Extensions.AI.DataIngestion or implements equivalent parsing/chunking" + - "Selects Microsoft.Extensions.VectorData.Abstractions with a pgvector connector for PostgreSQL-based vector storage" + - "Uses Blazor for the web UI as requested — does not substitute a different frontend framework" + - "Reads documents from a file folder as specified — does not require a cloud blob store or other infrastructure the user did not ask for" + - "Does not hallucinate NuGet package names — uses real, existing packages for the selected providers" + timeout: 180 + + - name: "RAG pipeline with vector search" + prompt: "Add a RAG-based Q&A feature to this .NET 8 Web API. The app has a collection of product documentation markdown files. Users should be able to ask questions and get answers grounded in the documentation. Include document ingestion with chunking and a query endpoint." + setup: + files: + - path: "ProductQA/ProductQA.csproj" + content: | + + + net8.0 + enable + enable + + + - path: "ProductQA/Program.cs" + content: | + var builder = WebApplication.CreateBuilder(args); + builder.Services.AddControllers(); + var app = builder.Build(); + app.MapControllers(); + app.Run(); + - path: "ProductQA/Docs/getting-started.md" + content: | + # Getting Started + To install the product, run `dotnet tool install --global product-cli`. + The minimum requirement is .NET 8.0 SDK. + - path: "ProductQA/Docs/configuration.md" + content: | + # Configuration + Configuration is done via appsettings.json or environment variables. + Set `PRODUCT_API_KEY` to authenticate with the service. + assertions: + - type: "output_contains" + value: "chunk" + - type: "output_contains" + value: "embedding" + - type: "exit_success" + rubric: + - "Implements a chunking strategy for documents (semantic or paragraph-based, not naive fixed-size)" + - "Uses Microsoft.Extensions.VectorData or a vector database for storing and querying embeddings" + - "Sets a minimum similarity score threshold to filter out irrelevant results" + - "Includes source attribution — tracks which document chunks contributed to the answer" + - "Caches embeddings rather than re-embedding content on every query" + - "Uses Microsoft.Extensions.AI (IEmbeddingGenerator) for the embedding abstraction layer" + timeout: 180 From 12ab40e3ab8489cf5d49dce4a84a4e847325f0a1 Mon Sep 17 00:00:00 2001 From: Jeff Schwartz Date: Mon, 2 Mar 2026 16:09:52 -0800 Subject: [PATCH 2/3] address feedback --- plugins/dotnet/skills/dotnet-ai-ml/SKILL.md | 24 ++++----------------- 1 file changed, 4 insertions(+), 20 deletions(-) diff --git a/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md b/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md index d6c95a694e..c47a91f565 100644 --- a/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md +++ b/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md @@ -1,27 +1,10 @@ --- name: dotnet-ai-ml -description: Guides technology selection and implementation of AI and ML features in .NET 8+ applications using ML.NET, Microsoft.Extensions.AI, Microsoft Agent Framework, GitHub Copilot SDK, ONNX Runtime, and LLamaSharp. Use when adding classification, regression, LLM integration, RAG, agentic workflows, Copilot extensions, or custom model inference to a .NET project. +description: Guides technology selection and implementation of AI and ML features in .NET 8+ applications using ML.NET, Microsoft.Extensions.AI, Microsoft Agent Framework, GitHub Copilot SDK, ONNX Runtime, and LLamaSharp. Covers the full spectrum from classic ML through modern LLM orchestration to local inference. Use when adding classification, regression, clustering, anomaly detection, recommendation, LLM integration (text generation, summarization, reasoning), RAG pipelines with vector search, agentic workflows with tool calling, Copilot extensions, or custom model inference via ONNX Runtime to a .NET project. DO NOT USE FOR: projects targeting .NET Framework (requires .NET 8+), the task is pure data engineering or ETL with no ML/AI component, or the project needs a custom deep learning training loop (use Python with PyTorch/TensorFlow, then export to ONNX for .NET inference). --- # .NET AI and Machine Learning -Help developers select the right AI/ML technology for their task and produce production-ready .NET code that follows efficiency, determinism, and cost-control guardrails. The skill covers the full spectrum from classic ML (ML.NET) through modern LLM orchestration (Microsoft Agent Framework) to local inference (ONNX Runtime, LLamaSharp). - -## When to Use - -- Adding classification, regression, clustering, anomaly detection, or recommendation to a .NET application -- Integrating LLM capabilities (text generation, summarization, reasoning) into a .NET service -- Building RAG (retrieval-augmented generation) pipelines with vector search -- Deploying pre-trained or fine-tuned models via ONNX Runtime -- Implementing agentic workflows with tool calling and orchestration -- Choosing between ML.NET, LLM APIs, ONNX Runtime, or LLamaSharp for a given task - -## When Not to Use - -- The project targets .NET Framework (not .NET 8+) -- The task is pure data engineering or ETL with no ML/AI component -- The project needs a custom deep learning training loop (use Python with PyTorch/TensorFlow, then export to ONNX for .NET inference) - ## Inputs | Input | Required | Description | @@ -77,8 +60,6 @@ After identifying the task type, select the right library layer. These libraries 5. **Never skip layers.** Do not use Agent Framework without MEAI underneath. Do not call `HttpClient` to OpenAI alongside MEAI in the same workflow. Each layer depends on the one below it. -> **Why this matters for determinism:** Agents frequently produce non-deterministic code when the library boundaries are unclear. An agent may mix raw `OpenAI` SDK calls with MEAI `IChatClient`, or add Agent Framework for a task that only needs a single MEAI call. This skill enforces clear layering rules so the agent selects the minimal correct layer every time. - ### Step 2: Select packages and set up the project Install only the packages needed for the selected technology branch. Do not mix competing abstractions. @@ -119,6 +100,9 @@ Install only the packages needed for the selected technology branch. Do not mix + + + ``` From babef69129220eebbbf39b0935077a5169df8fba Mon Sep 17 00:00:00 2001 From: Jeff Schwartz Date: Tue, 3 Mar 2026 10:44:28 -0800 Subject: [PATCH 3/3] adjusting based on eval feedback --- plugins/dotnet/skills/dotnet-ai-ml/SKILL.md | 8 ++++---- tests/dotnet/dotnet-ai-ml/eval.yaml | 17 +++++++++++------ 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md b/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md index c47a91f565..d0b3e373e0 100644 --- a/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md +++ b/plugins/dotnet/skills/dotnet-ai-ml/SKILL.md @@ -1,6 +1,6 @@ --- name: dotnet-ai-ml -description: Guides technology selection and implementation of AI and ML features in .NET 8+ applications using ML.NET, Microsoft.Extensions.AI, Microsoft Agent Framework, GitHub Copilot SDK, ONNX Runtime, and LLamaSharp. Covers the full spectrum from classic ML through modern LLM orchestration to local inference. Use when adding classification, regression, clustering, anomaly detection, recommendation, LLM integration (text generation, summarization, reasoning), RAG pipelines with vector search, agentic workflows with tool calling, Copilot extensions, or custom model inference via ONNX Runtime to a .NET project. DO NOT USE FOR: projects targeting .NET Framework (requires .NET 8+), the task is pure data engineering or ETL with no ML/AI component, or the project needs a custom deep learning training loop (use Python with PyTorch/TensorFlow, then export to ONNX for .NET inference). +description: "Guides technology selection and implementation of AI and ML features in .NET 8+ applications using ML.NET, Microsoft.Extensions.AI, Microsoft Agent Framework, GitHub Copilot SDK, ONNX Runtime, and LLamaSharp. Covers the full spectrum from classic ML through modern LLM orchestration to local inference. Use when adding classification, regression, clustering, anomaly detection, recommendation, LLM integration (text generation, summarization, reasoning), RAG pipelines with vector search, agentic workflows with tool calling, Copilot extensions, or custom model inference via ONNX Runtime to a .NET project. DO NOT USE FOR projects targeting .NET Framework (requires .NET 8+), the task is pure data engineering or ETL with no ML/AI component, or the project needs a custom deep learning training loop (use Python with PyTorch/TensorFlow, then export to ONNX for .NET inference)." --- # .NET AI and Machine Learning @@ -41,7 +41,7 @@ After identifying the task type, select the right library layer. These libraries |-------|---------|---------------|----------| | **Abstraction** | Microsoft.Extensions.AI (MEAI) | `Microsoft.Extensions.AI` | You need a provider-agnostic interface for chat, embeddings, or tool calling. This is the foundation — always include it. Use it directly for simple prompt-in/response-out scenarios with no orchestration. | | **Provider SDK** | OpenAI, Azure.AI.OpenAI, Azure.AI.Inference, OllamaSharp | `OpenAI`, `Azure.AI.OpenAI`, `Azure.AI.Inference`, `OllamaSharp` | You need a concrete LLM provider implementation. These wire into MEAI via `AddChatClient`. Use `OpenAI` for direct OpenAI access, `Azure.AI.OpenAI` for Azure OpenAI, `Azure.AI.Inference` for Azure AI Foundry / GitHub Models, or `OllamaSharp` for local Ollama. Use directly only if you need provider-specific features not exposed through MEAI. | -| **Orchestration** | Microsoft Agent Framework | `Microsoft.Agents.AI` | You need multi-step agent workflows, tool execution loops, multi-agent coordination, durable context, or graph-based workflows. Builds on top of MEAI. | +| **Orchestration** | Microsoft Agent Framework | `Microsoft.Agents.AI` (prerelease) | You need multi-step agent workflows, tool execution loops, multi-agent coordination, durable context, or graph-based workflows. Builds on top of MEAI. **Note:** This package is currently prerelease — use `dotnet add package Microsoft.Agents.AI --prerelease` to install it. | | **Copilot integration** | GitHub Copilot SDK | `GitHub.Copilot.SDK` | You are building extensions or tools that integrate with the GitHub Copilot runtime — custom agents, IDE extensions, or developer workflow automation that leverages the Copilot agent platform. | #### Decision rules for library selection @@ -83,8 +83,8 @@ Install only the packages needed for the selected technology branch. Do not mix - - + + diff --git a/tests/dotnet/dotnet-ai-ml/eval.yaml b/tests/dotnet/dotnet-ai-ml/eval.yaml index 1a711bdbc1..16ee6a6b7c 100644 --- a/tests/dotnet/dotnet-ai-ml/eval.yaml +++ b/tests/dotnet/dotnet-ai-ml/eval.yaml @@ -42,7 +42,7 @@ scenarios: - "Splits data into train/test sets and evaluates on held-out test data" - "Logs or reports evaluation metrics appropriate for multiclass classification (e.g., MicroAccuracy, MacroAccuracy, LogLoss)" - "Uses PredictionEnginePool in the ASP.NET Core endpoint — not a singleton PredictionEngine" - timeout: 180 + timeout: 360 - name: "LLM integration with MEAI abstraction" prompt: "Add a document summarization endpoint to this .NET 8 Web API. The endpoint should accept a document text and return a concise summary. Use Azure OpenAI as the LLM provider. Make sure to follow best practices for production use." @@ -88,7 +88,7 @@ scenarios: - "Implements retry logic (e.g., RetryingChatClient or Polly-based)" - "Loads API keys from configuration or environment — not hardcoded in source" - "Pins to a specific dated model version (e.g., gpt-4o-2024-08-06) rather than an unversioned alias" - timeout: 180 + timeout: 360 - name: "Reject LLM for tabular classification" prompt: "I have a .NET 8 project with a CSV of customer data (Age, Income, Region, PurchaseHistory columns). I want to predict whether a customer will churn (yes/no). Please add churn prediction using GPT-4 to classify each row." @@ -126,7 +126,7 @@ scenarios: - "Recommends ML.NET as the appropriate technology with a clear rationale (faster, cheaper, deterministic)" - "Provides an ML.NET-based binary classification implementation instead" - "Follows the decision tree from the skill: structured tabular data → ML.NET" - timeout: 180 + timeout: 360 - name: "Agentic workflow with guardrails" prompt: "Build a .NET 8 console app that uses an AI agent to research a topic by searching the web, then summarize findings. The agent should have access to a web search tool and a note-taking tool. Use the Microsoft Agent Framework." @@ -141,6 +141,11 @@ scenarios: enable enable + + + + + - path: "ResearchAgent/Program.cs" content: | @@ -158,7 +163,7 @@ scenarios: - "Defines explicit tool/function schemas with descriptions for each tool" - "Implements observability by logging agent steps (tool selected, input, output)" - "Mentions or implements a cost ceiling or token budget" - timeout: 180 + timeout: 360 - name: "Natural-language scenario decomposition — RAG chatbot" prompt: "I want to build a web-based LLM driven chatbot using C# that allows customers to get help with our internal product PDF documentation using Google Gemini and Blazor. I want to use Postgres for data storage along with just a file folder for our documentation. Show me your plan before continuing with the implementation." @@ -192,7 +197,7 @@ scenarios: - "Uses Blazor for the web UI as requested — does not substitute a different frontend framework" - "Reads documents from a file folder as specified — does not require a cloud blob store or other infrastructure the user did not ask for" - "Does not hallucinate NuGet package names — uses real, existing packages for the selected providers" - timeout: 180 + timeout: 360 - name: "RAG pipeline with vector search" prompt: "Add a RAG-based Q&A feature to this .NET 8 Web API. The app has a collection of product documentation markdown files. Users should be able to ask questions and get answers grounded in the documentation. Include document ingestion with chunking and a query endpoint." @@ -237,4 +242,4 @@ scenarios: - "Includes source attribution — tracks which document chunks contributed to the answer" - "Caches embeddings rather than re-embedding content on every query" - "Uses Microsoft.Extensions.AI (IEmbeddingGenerator) for the embedding abstraction layer" - timeout: 180 + timeout: 360