dotnet ai ml skill by jeffschwMSFT · Pull Request #27 · dotnet/skills

jeffschwMSFT · 2026-02-15T22:05:40Z

while exploring .NET application types, the model found a high level of non-determinism when considering classic and agentic AI SDKs.

Copilot

Pull request overview

This PR introduces a comprehensive new skill for .NET AI and Machine Learning development that addresses non-determinism in AI/ML code generation by providing clear technology selection guidance and implementation guardrails.

Changes:

Adds dotnet-ai-ml skill with detailed decision trees for selecting between ML.NET, Microsoft.Extensions.AI, Microsoft Agent Framework, GitHub Copilot SDK, ONNX Runtime, and LLamaSharp
Implements library layering rules to prevent mixing abstraction layers (a major source of non-deterministic agent behavior)
Provides comprehensive evaluation scenarios covering ML.NET classification, LLM integration, technology rejection patterns, agentic workflows with guardrails, and RAG pipelines

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/dotnet/skills/dotnet-ai-ml/SKILL.md	Comprehensive 357-line skill document defining technology selection decision trees, library layering rules, guardrails for classic ML/LLM/agent/RAG implementations, and anti-patterns to reject
src/dotnet/tests/dotnet-ai-ml/eval.yaml	Five evaluation scenarios with 180-second timeouts testing both positive patterns (correct technology usage) and negative patterns (rejecting inappropriate LLM usage for tabular data)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jeffschwMSFT · 2026-02-21T18:58:59Z

Skill	Scenario	Baseline	With Skill	Δ	Verdict
dotnet-ai-ml	ML.NET classification on tabular data	2.0/5	5.0/5	+3.0	✅
dotnet-ai-ml	LLM integration with MEAI abstraction	2.0/5	4.0/5	+2.0	✅
dotnet-ai-ml	Reject LLM for tabular classification	1.0/5	5.0/5	+4.0	✅
dotnet-ai-ml	Agentic workflow with guardrails	2.0/5	3.0/5	+1.0	⚠️
dotnet-ai-ml	RAG pipeline with vector search	2.0/5	4.0/5	+2.0	❌

Note: RAG pipeline with vector search is marked ❌ because task completion regressed (✓→✗) — the with-skill agent timed out at the 180s limit during final verification, despite producing higher-quality code (2.0→4.0). The baseline completed within the time limit but with lower quality. Increasing the agent timeout would likely resolve this regression.

artl93

Some suggestions for your consideration.

jeffschwMSFT · 2026-02-25T21:21:33Z

thank you for the feedback, it should now be addressed. here is the updated evaluation

Rubric criterion	Baseline	With Skill
Plan decomposition	5/5	5/5
Selects MEAI + Gemini provider	1/5	5/5
PDF ingestion/chunking	3/5	4/5
MEVD + pgvector	1/5	3/5
Blazor UI	5/5	5/5
File folder docs	5/5	5/5
No hallucinated packages	5/5	5/5

Add SKILL.md with guidance for .NET AI/ML library selection and eval.yaml with validation scenarios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

danmoseley · 2026-03-02T22:21:30Z

+
+Evaluate the developer's task against this decision tree and select the appropriate technology. State which branch applies and why.
+
+| Task type | Technology | Rationale |


No mention of semantic kernel, whether to recommend use, or not.

it is left out as this is a list of recommended choices

should we explicitly reject SK as an option?

will do. in rereading it fits well with the example of Accord.NET as well. also asked co-pilot and it agreed that it would be good to explicitly add

danmoseley · 2026-03-02T22:22:32Z

+| Ingesting, chunking, and loading documents into a vector store | **Microsoft.Extensions.AI.DataIngestion** (preview) + MEVD | Handles document parsing, text chunking, embedding generation, and upserting into a vector database; pairs with Microsoft.Extensions.VectorData.Abstractions |
+| Both structured ML predictions AND natural language reasoning | **Hybrid**: ML.NET for predictions + LLM for reasoning layer | Keep loosely coupled; ML.NET handles deterministic scoring, LLM adds explanation |
+
+**Critical rule:** Do NOT use an LLM for tasks that ML.NET handles well (classification on tabular data, regression, clustering). LLMs are slower, more expensive, and non-deterministic for these tasks.


is there also a rule that when using an LLM, extract anything that can be a script (eg., tabulation, math, stats) into code/script as it's both cheaper, more accurate and faster than LLM reasoning. maybe that's out of scope?

interesting, this was geared towards augmenting code and the SDK choice as opposed to a workflow

I'm curious if @luisquintanilla has an opinio here. LLMs have been promoted (nont by us) for one-shot classification. I was really torn on this one.

would definitely be good to get Luis to review before merge.

What is the goal for this skill? This skill appears to provide guidance with various technologies that may have different patterns, practices, use-cases. While they are intended to compose together to build solutions, having them all in one place might confuse the coding assistant. It might be better to split up into separate skills and this skill can serve as a meta-skill that references to those respective skills.

maybe. experimentally this has added a lot of value as it is highly opinionated and helps to lead to more determinism when choosing between all our options. behind the choice of what SDK to choose, the models seems very effective. so beyond choosing a good starting point, I am not sure what the other split skills would do.

danmoseley · 2026-03-02T22:25:00Z

+   }
+   ```
+
+4. **Tool schemas**: Define explicit tool/function schemas with descriptions. Never rely on implicit tool discovery.


and skills? don't know how these are surfaced

I think tool is synonymous with skill in this context. from the agent pov, skills are tools

danmoseley · 2026-03-02T22:30:40Z

rerun eval, maybe with increased timeout for the RAG one that's failing?

jeffschwMSFT · 2026-03-03T00:07:16Z

thanks for the feedback. I am rerunning the eval with the updates

jeffschwMSFT · 2026-03-03T18:56:40Z

thanks for the feedback, updated

Skill	Scenario	Baseline	With Skill	Δ	Skills Loaded	Overfit	Verdict
dotnet-ai-ml	ML.NET classification on tabular data	2.3/5	4.3/5	+2.0	✅ dotnet-ai-ml; tools: skill	🟡 0.25	✅
dotnet-ai-ml	LLM integration with MEAI abstraction	2.0/5	3.0/5	+1.0	✅ dotnet-ai-ml; tools: skill, read_bash	🟡 0.25	✅
dotnet-ai-ml	Reject LLM for tabular classification	1.0/5	5.0/5	+4.0	✅ dotnet-ai-ml; tools: skill, read_bash	🟡 0.25	✅
dotnet-ai-ml	Agentic workflow with guardrails	2.3/5	3.0/5	+0.7	✅ dotnet-ai-ml; tools: skill	🟡 0.25	✅
dotnet-ai-ml	Natural-language scenario decomposition — RAG chatbot	3.0/5	3.3/5	+0.3	✅ dotnet-ai-ml; tools: skill, edit	🟡 0.25	✅
dotnet-ai-ml	RAG pipeline with vector search	2.7/5	3.0/5	+0.3	✅ dotnet-ai-ml; tools: skill	🟡 0.25	✅

artl93 · 2026-03-03T19:06:28Z

+
+Evaluate the developer's task against this decision tree and select the appropriate technology. State which branch applies and why.
+
+| Task type | Technology | Rationale |


should we explicitly reject SK as an option?

artl93 · 2026-03-03T19:07:50Z

+| Ingesting, chunking, and loading documents into a vector store | **Microsoft.Extensions.AI.DataIngestion** (preview) + MEVD | Handles document parsing, text chunking, embedding generation, and upserting into a vector database; pairs with Microsoft.Extensions.VectorData.Abstractions |
+| Both structured ML predictions AND natural language reasoning | **Hybrid**: ML.NET for predictions + LLM for reasoning layer | Keep loosely coupled; ML.NET handles deterministic scoring, LLM adds explanation |
+
+**Critical rule:** Do NOT use an LLM for tasks that ML.NET handles well (classification on tabular data, regression, clustering). LLMs are slower, more expensive, and non-deterministic for these tasks.


I'm curious if @luisquintanilla has an opinio here. LLMs have been promoted (nont by us) for one-shot classification. I was really torn on this one.

danmoseley · 2026-03-03T19:51:57Z

@@ -0,0 +1,341 @@
+---
+name: dotnet-ai-ml
+description: "Guides technology selection and implementation of AI and ML features in .NET 8+ applications using ML.NET, Microsoft.Extensions.AI, Microsoft Agent Framework, GitHub Copilot SDK, ONNX Runtime, and LLamaSharp. Covers the full spectrum from classic ML through modern LLM orchestration to local inference. Use when adding classification, regression, clustering, anomaly detection, recommendation, LLM integration (text generation, summarization, reasoning), RAG pipelines with vector search, agentic workflows with tool calling, Copilot extensions, or custom model inference via ONNX Runtime to a .NET project. DO NOT USE FOR projects targeting .NET Framework (requires .NET 8+), the task is pure data engineering or ETL with no ML/AI component, or the project needs a custom deep learning training loop (use Python with PyTorch/TensorFlow, then export to ONNX for .NET inference)."


Suggested change

description: "Guides technology selection and implementation of AI and ML features in .NET 8+ applications using ML.NET, Microsoft.Extensions.AI, Microsoft Agent Framework, GitHub Copilot SDK, ONNX Runtime, and LLamaSharp. Covers the full spectrum from classic ML through modern LLM orchestration to local inference. Use when adding classification, regression, clustering, anomaly detection, recommendation, LLM integration (text generation, summarization, reasoning), RAG pipelines with vector search, agentic workflows with tool calling, Copilot extensions, or custom model inference via ONNX Runtime to a .NET project. DO NOT USE FOR projects targeting .NET Framework (requires .NET 8+), the task is pure data engineering or ETL with no ML/AI component, or the project needs a custom deep learning training loop (use Python with PyTorch/TensorFlow, then export to ONNX for .NET inference)."

description: "Guides technology selection and implementation of AI and ML features in .NET 8+ applications using ML.NET, Microsoft.Extensions.AI (MEAI), Microsoft Agent Framework, GitHub Copilot SDK, ONNX Runtime, and LLamaSharp. Covers the full spectrum from classic ML through modern LLM orchestration to local inference. Use when adding classification, regression, clustering, anomaly detection, recommendation, LLM integration (text generation, summarization, reasoning), RAG pipelines with vector search, agentic workflows with tool calling, Copilot extensions, or custom model inference via ONNX Runtime to a .NET project. DO NOT USE FOR projects targeting .NET Framework (requires .NET 8+), the task is pure data engineering or ETL with no ML/AI component, or the project needs a custom deep learning training loop (use Python with PyTorch/TensorFlow, then export to ONNX for .NET inference)."

users might say MEAI (idk)

danmoseley · 2026-03-03T19:53:25Z

+       .Use(new RetryingChatClient(maxRetries: 3)));
+   ```
+
+4. **Cost control**: Always estimate and log token usage. Choose the smallest model tier that meets quality requirements (e.g., `gpt-4o-mini` before `gpt-4o`).


do we need to suggest how to log token usage? AFAIK our libraries already fire the right events, not sure the app needs to do anything but IDK

seems like MaxOutputTokens may be the key thing - mentioned already above

I've spoken with @stephentoub about this (not recently), and while this was a big ask 2 years ago, it's not so much a thing people focus on now.

I'm not sure what "this" is here. People still definitely compute token counts, in order to ensure they're not hitting token limits or hurting model performance or if they want to estimate or limit costs, but they have most of what they need to do that, and I see folks being ok with approximations more than absolute-to-the-token correct answers.

timheuer · 2026-03-03T22:56:58Z

should we wrap this in a dotnet-ai plugin? MCP template one incoming too

luisquintanilla · 2026-03-06T16:20:16Z

Tracking in #225

Copilot AI review requested due to automatic review settings February 20, 2026 18:37

Copilot started reviewing on behalf of jeffschwMSFT February 20, 2026 18:38 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

Comment thread plugins/dotnet/skills/dotnet-ai-ml/SKILL.md

Comment thread plugins/dotnet/skills/dotnet-ai-ml/SKILL.md

jeffschwMSFT requested review from artl93 and jeffhandley February 23, 2026 17:01

artl93 reviewed Feb 25, 2026

View reviewed changes

jeffschwMSFT requested a review from artl93 February 27, 2026 00:53

jeffschwMSFT requested review from JanKrivanek, ViktorHofer, dbreshears and timheuer as code owners February 28, 2026 00:55

Add dotnet-ai-ml skill

157db6c

Add SKILL.md with guidance for .NET AI/ML library selection and eval.yaml with validation scenarios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jeffschwMSFT requested a review from danmoseley March 2, 2026 16:00

danmoseley reviewed Mar 2, 2026

View reviewed changes

Comment thread plugins/dotnet/skills/dotnet-ai-ml/SKILL.md Outdated

danmoseley reviewed Mar 2, 2026

View reviewed changes

Comment thread plugins/dotnet/skills/dotnet-ai-ml/SKILL.md Outdated

danmoseley reviewed Mar 2, 2026

View reviewed changes

Comment thread plugins/dotnet/skills/dotnet-ai-ml/SKILL.md

Merge remote-tracking branch 'origin/main' into skill/dotnet-ai-ml

29ff01f

jeffschwMSFT added 2 commits March 2, 2026 16:09

address feedback

12ab40e

adjusting based on eval feedback

babef69

jeffschwMSFT requested a review from danmoseley March 3, 2026 19:00

artl93 approved these changes Mar 3, 2026

View reviewed changes

danmoseley reviewed Mar 3, 2026

View reviewed changes

danmoseley approved these changes Mar 3, 2026

View reviewed changes

jeffschwMSFT closed this by deleting the head repository Mar 4, 2026

github-actions Bot mentioned this pull request Jun 3, 2026

fix(dotnet): add Claude Code-shaped .lsp.json so LSP loads in Claude Code #598

Merged

3 tasks


		Evaluate the developer's task against this decision tree and select the appropriate technology. State which branch applies and why.

		\| Task type \| Technology \| Rationale \|

Uh oh!

Conversation

jeffschwMSFT commented Feb 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

jeffschwMSFT commented Feb 21, 2026

Uh oh!

artl93 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeffschwMSFT commented Feb 25, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danmoseley commented Mar 2, 2026

Uh oh!

jeffschwMSFT commented Mar 3, 2026

Uh oh!

jeffschwMSFT commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timheuer commented Mar 3, 2026

Uh oh!

luisquintanilla commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

jeffschwMSFT commented Mar 3, 2026 •

edited

Loading