diff --git a/plugins/dotnet-ai/skills/data-ingestion-pipeline/SKILL.md b/plugins/dotnet-ai/skills/data-ingestion-pipeline/SKILL.md new file mode 100644 index 0000000000..8122e25ed7 --- /dev/null +++ b/plugins/dotnet-ai/skills/data-ingestion-pipeline/SKILL.md @@ -0,0 +1,161 @@ +--- +name: data-ingestion-pipeline +description: > + Guides building a document ingestion pipeline for RAG using + Microsoft.Extensions.DataIngestion. + USE FOR: Reading documents (Markdown, PDF, Word), chunking text, enriching + chunks (summaries, keywords, sentiment), writing to vector stores. Building + the ingestion half of a RAG system. + DO NOT USE FOR: Vector search/storage setup (use vector-data-search), LLM chat + integration (use meai-chat-integration), end-to-end RAG (use rag-pipeline), + tabular data processing (use mlnet). +--- + +# Data Ingestion Pipeline + +This skill guides an agent through building a document ingestion pipeline for retrieval-augmented generation (RAG) using `Microsoft.Extensions.DataIngestion`. The pipeline reads documents (Markdown, PDF, Word), splits them into chunks, optionally enriches chunks with summaries or keywords, generates embeddings, and writes the results to a vector store. + +## When to Use + +- Building the ingestion half of a RAG system that reads documents into a vector store +- Reading Markdown, PDF, Word, PowerPoint, or HTML documents into a processing pipeline +- Chunking documents for embedding and retrieval +- Enriching chunks with summaries, keywords, sentiment, or classifications before storage +- Writing embedded chunks to a vector store for later search + +## When Not to Use + +- **Vector search/storage setup only** — use the `vector-data-search` skill instead +- **LLM chat integration** — use the `meai-chat-integration` skill instead +- **End-to-end RAG** (ingestion + retrieval + chat) — use the `rag-pipeline` skill instead +- **Tabular or structured data processing** — use the `mlnet` skill instead + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Document source | Yes | Path or location of documents to ingest (Markdown, PDF, Word, etc.) | +| Document format | Yes | File type(s) to read — determines which reader package to install | +| Embedding model | Yes | The embedding model to use for vectorizing chunks (e.g., `text-embedding-3-small`) | +| Vector store | Yes | Target vector store and connector (e.g., Azure AI Search, Qdrant, in-memory) | +| Chunking strategy | Recommended | How to split documents — by headers, sections, or semantic similarity | +| Enrichments | Optional | Which enrichers to apply: summary, keywords, sentiment, classification | + +## Workflow + +> **Commit strategy:** Commit after each step produces a working pipeline stage. This keeps the pipeline buildable and testable incrementally. + +### Step 1: Install packages + +Install the core pipeline package and the reader for your document format: + +``` +dotnet add package Microsoft.Extensions.DataIngestion +``` + +Choose a reader based on document format: + +- **IF Markdown** → `dotnet add package Microsoft.Extensions.DataIngestion.Markdig` +- **IF PDF, Word, PowerPoint, HTML, or other formats** → `dotnet add package Microsoft.Extensions.DataIngestion.MarkItDown` + +For embedding enrichment, add the AI abstractions and your provider package: + +``` +dotnet add package Microsoft.Extensions.AI +``` + +For writing to a vector store, add the vector data abstractions and your connector: + +``` +dotnet add package Microsoft.Extensions.VectorData.Abstractions +``` + +### Step 2: Configure the document reader + +Set up the reader that converts raw files into `IngestionDocument` objects: + +- **Markdig reader**: Reads `.md` files, preserving markdown structure for downstream chunking. +- **MarkItDown reader**: Reads PDF, DOCX, PPTX, HTML, and other formats by converting them to markdown first, then producing `IngestionDocument` objects. + +### Step 3: Choose a chunking strategy + +Select a chunker based on the structure of your documents: + +- **IF documents have clear headers/sections** → `HeaderChunker` — splits on markdown headers, keeping each section as a chunk. +- **IF documents have page or section boundaries** → `SectionChunker` — splits on explicit boundaries. +- **IF you need semantic coherence within chunks** → `SemanticChunker` — groups semantically related paragraphs together. + +Configure token limits and overlap: + +- Set `MaxTokensPerChunk` to fit your embedding model's context window. For most embedding models, **256–512 tokens per chunk** works well. +- Use `Microsoft.ML.Tokenizers` to count tokens accurately: + ```csharp + var tokenizer = TiktokenTokenizer.CreateForModel("text-embedding-3-small"); + var tokenCount = tokenizer.CountTokens(chunkText); + ``` +- Configure **10–20% overlap** between chunks to preserve context at boundaries. + +### Step 4: Add processors/enrichers (optional) + +Add one or more enrichers to augment chunks before storage: + +- **SummaryEnricher** — generates a short summary per chunk using `IChatClient`. +- **KeywordEnricher** — extracts keywords from each chunk. +- **SentimentEnricher** — scores each chunk's sentiment. +- **ClassificationEnricher** — classifies chunks into predefined categories. + +Each enricher runs as a pipeline processor and attaches metadata to the chunk. + +### Step 5: Configure the writer + +Set up `VectorStoreWriter` to write chunks with embeddings to a vector store collection: + +- The writer needs an `IEmbeddingGenerator` to generate embeddings for each chunk. +- The writer needs a `VectorStoreCollection` configured for your target vector store. + +### Step 6: Build and run the pipeline + +Assemble the pipeline using the builder, then execute it: + +```csharp +var pipeline = new IngestionPipelineBuilder() + .AddReader(reader) + .AddChunker(chunker) + .AddProcessor(enricher) // optional, repeat for multiple enrichers + .AddWriter(writer) + .Build(); + +var result = await pipeline.RunAsync(documents); +``` + +### Step 7: Handle errors + +`IngestionResult` may indicate partial success. Check for failures and handle them: + +- Inspect `result.FailedDocuments` for documents that could not be processed. +- Log failures with enough detail to diagnose the issue (file path, stage where failure occurred). +- Retry transient failures (e.g., embedding API rate limits) with exponential backoff. + +## Validation + +- [ ] Documents are read successfully by the configured reader +- [ ] Chunks have the expected size (within `MaxTokensPerChunk` limit) +- [ ] Enrichments are populated on chunks (summaries, keywords, etc., if configured) +- [ ] Chunks are written to the vector store with embeddings +- [ ] A search query against the vector store returns relevant chunks +- [ ] `result.FailedDocuments` is empty or failures are handled + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Chunks too large — exceed embedding model context window | Set `MaxTokensPerChunk` to stay within the model's limit (typically 256–512 tokens). Use `Microsoft.ML.Tokenizers` to validate token counts. | +| Chunks too small — lose context and produce poor retrieval results | Increase `MaxTokensPerChunk` or switch to `SemanticChunker` to keep related content together. | +| No overlap between chunks — information lost at chunk boundaries | Configure 10–20% overlap so context is preserved across chunk boundaries. | +| Using character count instead of token count for chunk sizing | Always use token-based sizing. Character counts do not map reliably to tokens. Use `TiktokenTokenizer.CreateForModel()` to count tokens accurately. | +| Not embedding chunks before writing to vector store | Ensure the `VectorStoreWriter` is configured with an `IEmbeddingGenerator`. Without embeddings, chunks cannot be searched by similarity. | +| Not handling partial pipeline failures | Always check `result.FailedDocuments` after `RunAsync`. Partial failures are silent if not inspected. Retry transient errors and log permanent failures. | + +## More Info + +- [Data ingestion in .NET AI](https://learn.microsoft.com/dotnet/ai/conceptual/data-ingestion) — conceptual overview of the ingestion pipeline diff --git a/plugins/dotnet-ai/skills/vector-data-search/SKILL.md b/plugins/dotnet-ai/skills/vector-data-search/SKILL.md new file mode 100644 index 0000000000..7fc3a8f0b1 --- /dev/null +++ b/plugins/dotnet-ai/skills/vector-data-search/SKILL.md @@ -0,0 +1,100 @@ +--- +name: vector-data-search +description: | + USE FOR: Storing vector embeddings, semantic similarity search, filtered vector search, managing vector store collections, choosing a vector database connector + DO NOT USE FOR: Generating embeddings (use meai-embeddings), document ingestion/chunking (use data-ingestion-pipeline), end-to-end RAG (use rag-pipeline), classical ML on structured data (use mlnet) +--- + +# Vector Data Storage & Semantic Search + +Add vector storage and semantic search to a .NET application using `Microsoft.Extensions.VectorData`. + +## Workflow + +### Step 1 · Install Packages + +Always install the abstractions package: + +``` +dotnet add package Microsoft.Extensions.VectorData.Abstractions +``` + +Then choose a connector based on your scenario: + +| Scenario | Package | +|---|---| +| Prototyping / dev | `Microsoft.SemanticKernel.Connectors.InMemory` | +| Local persistence | `Microsoft.SemanticKernel.Connectors.SqliteVec` (+ `Microsoft.Data.Sqlite`) | +| Azure cloud | `Microsoft.SemanticKernel.Connectors.AzureAISearch` or `Microsoft.SemanticKernel.Connectors.CosmosNoSql` | +| Self-hosted Postgres | `Microsoft.SemanticKernel.Connectors.Postgres` (requires pgvector extension) | +| Qdrant | `Microsoft.SemanticKernel.Connectors.Qdrant` | +| Redis | `Microsoft.SemanticKernel.Connectors.Redis` | + +> **Provenance note:** Connector packages use the `Microsoft.SemanticKernel.Connectors.*` namespace but they depend ONLY on `Microsoft.Extensions.VectorData.Abstractions`. There is no dependency on Semantic Kernel itself. This is a packaging artifact from the migration. + +### Step 2 · Define a Data Model + +```csharp +public class DocumentChunk +{ + [VectorStoreKey] + public string Id { get; set; } + + [VectorStoreData(IsFilterable = true)] + public string Source { get; set; } + + [VectorStoreData] + public string Content { get; set; } + + [VectorStoreVector(Dimensions: 1536, DistanceFunction.CosineSimilarity)] + public ReadOnlyMemory Embedding { get; set; } +} +``` + +### Step 3 · Create a Collection and Upsert Records + +```csharp +// Create a vector store (swap InMemoryVectorStore for the connector of your choice) +var store = new InMemoryVectorStore(); + +// Get a typed collection handle +var collection = store.GetCollection("documents"); + +// Ensure the backing collection/table exists +await collection.EnsureCollectionExistsAsync(); + +// Upsert a record +await collection.UpsertAsync(record); +``` + +### Step 4 · Perform a Vector Search + +```csharp +var results = await collection.SearchAsync(queryEmbedding, top: 5); +``` + +- Use filter expressions with properties marked `IsFilterable = true` to narrow results. +- Set a minimum score threshold to discard irrelevant matches. + +### Step 5 · Register in Dependency Injection + +Register the vector store and collection in `IServiceCollection` so they are available throughout the application. + +## Validation + +- Collection is created successfully. +- Records are upserted without error. +- Search returns semantically relevant results. +- Filtered search correctly narrows the result set. + +## Pitfalls + +- **Dimension mismatch** – The `Dimensions` value in `[VectorStoreVector]` must match the embedding model's output size. +- **Missing distance function** – Not setting `DistanceFunction` explicitly; defaults vary by connector. +- **Unfilterable properties** – Forgetting `IsFilterable = true` on properties used in filter expressions. +- **Collection creation on every request** – Call `EnsureCollectionExistsAsync` at startup, not per-request. +- **Unhandled exceptions** – Catch `VectorStoreOperationException` for transient and configuration errors. + +## More Information + + diff --git a/tests/dotnet-ai/data-ingestion-pipeline/eval.yaml b/tests/dotnet-ai/data-ingestion-pipeline/eval.yaml new file mode 100644 index 0000000000..f261519cca --- /dev/null +++ b/tests/dotnet-ai/data-ingestion-pipeline/eval.yaml @@ -0,0 +1,34 @@ +scenarios: + - name: "Ingest markdown documents for RAG" + prompt: "Build a document ingestion pipeline that reads markdown files from a directory, chunks them by headers, and writes the chunks to an in-memory vector store with embeddings." + setup: + files: + - path: "Ingestor/Ingestor.csproj" + content: | + + + Exe + net10.0 + + + - path: "Ingestor/Program.cs" + content: | + Console.WriteLine("TODO: Build ingestion pipeline"); + - path: "Ingestor/docs/getting-started.md" + content: | + # Getting Started + Install the SDK with `dotnet tool install`. + ## Prerequisites + You need .NET 10 or later. + assertions: + - type: "output_contains" + value: "chunk" + - type: "output_contains" + value: "DataIngestion" + - type: "exit_success" + rubric: + - "Uses Microsoft.Extensions.DataIngestion for the pipeline" + - "Configures a Markdown reader (Markdig)" + - "Uses header-based chunking as specified" + - "Generates embeddings for chunks before writing to vector store" + timeout: 360 diff --git a/tests/dotnet-ai/vector-data-search/eval.yaml b/tests/dotnet-ai/vector-data-search/eval.yaml new file mode 100644 index 0000000000..7f17001ddc --- /dev/null +++ b/tests/dotnet-ai/vector-data-search/eval.yaml @@ -0,0 +1,29 @@ +scenarios: + - name: "Add vector search to web API" + prompt: "Add vector search to this .NET 10 web API. I want to store document chunks with embeddings and search them by semantic similarity. Use InMemory store for development." + setup: + files: + - path: "SearchApi/SearchApi.csproj" + content: | + + + net10.0 + + + - path: "SearchApi/Program.cs" + content: | + var builder = WebApplication.CreateBuilder(args); + var app = builder.Build(); + app.Run(); + assertions: + - type: "output_contains" + value: "VectorStoreKey" + - type: "output_contains" + value: "SearchAsync" + - type: "exit_success" + rubric: + - "Defines a data model with VectorStoreKey, VectorStoreData, and VectorStoreVector attributes" + - "Uses InMemoryVectorStore as requested for development" + - "Creates a collection and implements search endpoint" + - "Sets appropriate vector dimensions matching the embedding model" + timeout: 360