Skip to content

MCP Issue Labeler GitHub Triage Bot - PART I #13552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
anannya03 merged 25 commits into from
Jan 29, 2026
Merged

MCP Issue Labeler GitHub Triage Bot - PART I #13552

anannya03 merged 25 commits into from
Jan 29, 2026

Conversation

@anannya03
Copy link
Member

@anannya03 anannya03 commented Jan 13, 2026 •
edited
Loading

MCP Issue Labeler GitHub Triage Bot

This PR adds an issue labeling system for the microsoft/mcp repository using Azure AI Search, Azure OpenAI, and RAG (Retrieval-Augmented Generation). It also updates the InitialIssueTriage rule for the microsoft/mcp repository to ensure consistent triage behavior across newly created issues.

image

Changes

MCP Labeler (McpOpenAiLabeler)

  • Automatically predicts Server and Tool labels for new MCP issues
  • Uses RAG to retrieve 15 similar labeled issues for context
  • Applies rule-based reasoning with confidence thresholding (0.9)
  • Prevents data leakage via excludeIssueId filter
  • Achieves ~90% accuracy on evaluation dataset
image

MCP-Specific Infrastructure (McpTriageRag)

  • Query preprocessing to remove tool name examples (e.g., "like Storage")
  • Filters for labeled issues only (Server ne null)

Evaluation Tool (Mcp.Evaluator)

  • Console app for testing labeler accuracy against ground truth
  • Single issue prediction, batch evaluation, and test dataset extraction modes
  • Exports results to CSV with detailed metrics

TASK: microsoft/mcp#127

@anannya03 anannya03 changed the title [DRAFT- DO NOT MERGE] Mcp gh issue labeler [DRAFT] MCP Issue Labeler GitHub Triage Bot Jan 15, 2026
@anannya03 anannya03 marked this pull request as ready for review January 16, 2026 06:06
Copilot AI review requested due to automatic review settings January 16, 2026 06:06
@anannya03 anannya03 changed the title [DRAFT] MCP Issue Labeler GitHub Triage Bot MCP Issue Labeler GitHub Triage Bot Jan 16, 2026
Copilot AI reviewed Jan 16, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment •
edited by anannya03
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements an AI-powered issue labeling system for the microsoft/mcp repository using Azure AI Search, Azure OpenAI, and RAG (Retrieval-Augmented Generation). The system automatically predicts Server and Tool labels for new issues based on similarity to historical labeled issues.

Changes:

  • Adds McpOpenAiLabeler with RAG-based label prediction achieving ~90% accuracy
  • Implements MCP-specific infrastructure (McpTriageRag) with query preprocessing and filtering
  • Creates evaluation tool (Mcp.Evaluator) for testing labeler accuracy against ground truth
  • Updates GitHub event processor to route microsoft/mcp issues to new MCP-specific triage logic

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 29 comments.

Show a summary per file
File Description
tools/issue-labeler/src/SearchIndexCreator/README.md Comprehensive setup documentation for search infrastructure
tools/issue-labeler/src/SearchIndexCreator/Program.cs Adds repository-specific routing and delete agent option
tools/issue-labeler/src/SearchIndexCreator/LabelRetrieval.cs Implements MCP label extraction from GitHub
tools/issue-labeler/src/SearchIndexCreator/IssueTriageContentRetrieval.cs Adds MCP issue retrieval with Server/Tool label support
tools/issue-labeler/src/SearchIndexCreator/IssueTriageContentIndex.cs Configures MCP-optimized skillset and field mappings
tools/issue-labeler/src/SearchIndexCreator/IssueKnowledgeAgent.cs Adds configurable reranker threshold and delete capability
tools/issue-labeler/src/SearchIndexCreator/DocumentTypes.cs Adds Comment document type
tools/issue-labeler/src/Mcp.Evaluator/README.md Documentation for evaluation tool usage
tools/issue-labeler/src/Mcp.Evaluator/McpLabelerTool.cs Main evaluation program with prediction and extraction modes
tools/issue-labeler/src/Mcp.Evaluator/Mcp.Evaluator.csproj Project configuration
tools/issue-labeler/src/Mcp.Evaluator/Evaluation/* Test models and evaluator implementation
tools/issue-labeler/src/IssueLabelerService/TraigeRag.cs Changes filter parameter from Dictionary to string
tools/issue-labeler/src/IssueLabelerService/Program.cs Registers McpTriageRag and adds config override capability
tools/issue-labeler/src/IssueLabelerService/Labelers/Mcp/McpTriageRag.cs MCP-specific RAG with query cleaning and labeled issue filtering
tools/issue-labeler/src/IssueLabelerService/Labelers/Mcp/McpOpenAiLabeler.cs Main labeler with extensive classification prompt and validation
tools/issue-labeler/src/IssueLabelerService/Labelers/LabelerFactory.cs Adds McpOpenAI labeler factory case
tools/issue-labeler/src/IssueLabeler.Shared/IssueTriageContent.cs Adds Server and Tool fields
tools/issue-labeler/IssueLabeler.sln Adds Mcp.Evaluator project with platform configurations
tools/issue-labeler/Directory.Build.props Updates Microsoft.Extensions versions
tools/github-event-processor/*/Program.cs Adds DI container and McpIssueProcessing service
tools/github-event-processor/*/McpIssueProcessing.cs MCP-specific triage logic with code owner assignment
tools/github-event-processor/*/IssueProcessing.cs Routes MCP issues to dedicated processor
tools/github-event-processor/*/ConfigConstants.cs Defines MCP label prefixes
tools/github-event-processor/*/Azure.Sdk.Tools.GitHubEventProcessor.csproj Adds hosting dependencies
tools/github-event-processor/Tests/* Test payloads and McpIssueProcessingTests

jsquire
jsquire reviewed Jan 17, 2026
View reviewed changes
tools/github-event-processor/Azure.Sdk.Tools.GitHubEventProcessor/Constants/ConfigConstants.cs Outdated
Comment on lines 3 to 7
using System.Collections.Specialized;
using System.Diagnostics.Contracts;
using System.Linq;
using System.Reflection.Metadata;
using System.Security.Cryptography.X509Certificates;
Copy link
Member

@jsquire jsquire Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good suggestion; using "Remove and sort usings" would be a good idea here.

jsquire
jsquire requested changes Jan 17, 2026
View reviewed changes
radhgupta
radhgupta requested changes Jan 20, 2026
View reviewed changes
@radhgupta radhgupta self-requested a review January 27, 2026 00:27
@anannya03 anannya03 enabled auto-merge (squash) January 27, 2026 00:35
danieljurek
danieljurek reviewed Jan 28, 2026
View reviewed changes
danieljurek
danieljurek reviewed Jan 28, 2026
View reviewed changes
@danieljurek
Copy link
Member

danieljurek commented Jan 28, 2026 •
edited
Loading

Major differences

  • No AzureSdkOwners... using ServiceOwners for that
  • Labels, no colors
    • server-<server> -- (e.g. server-Azure.Mcp)
    • tools-<tool> -- (e.g. tools-Bicep) -- Lives in CODEOWNERS file

Likelyhood of regression is minimal given dependency injection and changes.

Workflow

  1. Issue opened
  2. GitHub Event Processor Action fires, sends request to Functions app and ends
  3. Functions app predicts label and applies it to
  4. GitHub Event Processor Action fires, takes action on new lables

Server labels

Crated as part of onboarding. Not referenced from CODEOWNERS.

https://github.com/microsoft/mcp/issues/labels?q=server-

image

Deployment

How is this deployed? How do we roll back/patch forward if there's a regression?

@danieljurek danieljurek mentioned this pull request Jan 28, 2026
Open
danieljurek
danieljurek approved these changes Jan 28, 2026
View reviewed changes
Copy link
Member

@danieljurek danieljurek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's get this deployed and iterating.

There's some stuff to document here but that knowledge should probably all go in a wiki.

This was referenced Jan 28, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
anannya03 added a commit to Azure/azure-sdk-for-rust that referenced this pull request Jan 29, 2026
@azure-sdk @anannya03
Sync .github/workflows directory with azure-sdk-tools for PR 13552 (#…
f601022 
…3634)

Sync .github/workflows directory with azure-sdk-tools for PR
Azure/azure-sdk-tools#13552 See [eng/common
workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow)

---------

Co-authored-by: anannya03 <aaniee09@gmail.com>
anannya03 added a commit to Azure/azure-sdk-for-js that referenced this pull request Jan 29, 2026
@azure-sdk @anannya03
Sync .github/workflows directory with azure-sdk-tools for PR 13552 (#…
972e3d3 
…37185)

Sync .github/workflows directory with azure-sdk-tools for PR
Azure/azure-sdk-tools#13552 See [eng/common
workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow)

---------

Co-authored-by: anannya03 <aaniee09@gmail.com>
@anannya03 anannya03 merged commit 4f68126 into main Jan 29, 2026
13 checks passed
@anannya03 anannya03 deleted the mcp-gh-issue-labeler branch January 29, 2026 20:35
@anannya03 anannya03 mentioned this pull request Jan 29, 2026
Merged
radhgupta
radhgupta reviewed Feb 2, 2026
View reviewed changes
tools/issue-labeler/src/Mcp.Evaluator/McpLabelerTool.cs
var csvOutput = args.FirstOrDefault(a => a.StartsWith("--output="))?.Split('=')[1]
?? "mcp_evaluation_results.csv";

var credential = new DefaultAzureCredential();
Copy link
Member

@radhgupta radhgupta Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Anannya,
Could you please change this, it is causing S360 alertalerts for the CodeQL DefaultAzureCredential scan. I was told the due date is Feb 13th.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@benbp benbp benbp left review comments

Copilot code review Copilot Copilot left review comments

@radhgupta radhgupta radhgupta approved these changes

@jsquire jsquire jsquire approved these changes

@danieljurek danieljurek danieljurek approved these changes

@jeo02 jeo02 jeo02 approved these changes

@ronniegeraghty ronniegeraghty Awaiting requested review from ronniegeraghty ronniegeraghty is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@anannya03 @radhgupta @danieljurek @jsquire @benbp @jeo02