[dotnet-ai] RAG pipeline

[luisquintanilla]

Parent Issue

Part of the dotnet-ai plugin proposal: #225

Summary

Add the rag-pipeline skill to the dotnet-ai plugin.

This skill provides a self-contained guide for building end-to-end retrieval-augmented generation (RAG) pipelines in .NET. It composes four foundation skills into a coherent workflow: ingest documents → generate embeddings → store in vectors → retrieve and generate grounded answers.

What This Skill Covers

A RAG pipeline has six stages, all covered:

Stage	What Happens	Underlying Skill
1. Document Ingestion	Load and chunk source documents	data-ingestion-pipeline
2. Embedding Generation	Convert chunks to vector embeddings	meai-embeddings
3. Vector Storage	Store embeddings with metadata in a vector store	vector-data-search
4. Retrieval	Query the vector store with the user's question	vector-data-search
5. Prompt Assembly	Combine retrieved context with the user's question	(this skill)
6. Grounded Generation	Generate an answer grounded in retrieved context	meai-chat-integration

Additional coverage:

• Source attribution — include provenance (document name, page, chunk ID) in the response
• Out-of-scope handling — detect when retrieved context doesn't contain the answer and respond honestly
• Relevance filtering — similarity threshold to avoid injecting irrelevant chunks
• Chunk overlap — overlapping windows during ingestion to preserve context across boundaries

What This Skill Does NOT Cover

• Classical ML on structured data (use mlnet)
• Agentic workflows beyond simple RAG (use agentic-workflow)
• Building Copilot extensions (use copilot-sdk-integration)

Files

plugins/dotnet-ai/skills/rag-pipeline/SKILL.md
tests/dotnet-ai/rag-pipeline/eval.yaml

Eval Scenarios

1. Build end-to-end RAG pipeline — prompt asks to build a system where users can ask questions about internal documentation; expects all 6 RAG stages, source attribution, and out-of-scope handling

Key Packages

• Microsoft.Extensions.AI (chat + embeddings)
• Microsoft.Extensions.VectorData (vector storage)
• Microsoft.Extensions.DataIngestion (document loading + chunking)
• Provider packages for LLM and embedding model

Dependencies

Requires the plugin scaffold (sub-issue #1) to be merged first. Ideally the foundation skills (meai-chat-integration, meai-embeddings, vector-data-search, data-ingestion-pipeline) are merged first since this skill composes them and references them by name.

Acceptance Criteria

• All 6 RAG stages are covered with code examples
• Source attribution guidance is present
• Out-of-scope handling (honest "I don't know") is documented
• Relevance threshold filtering is covered
• Skill explicitly references the 4 foundation skills it composes
• eval.yaml validates the end-to-end scenario