Add script for local tool description eval workflow #13218
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a PowerShell script to automate the evaluation of tool and prompt descriptions using the Tool Description Evaluator from the MCP repository. The script orchestrates building and running a .NET evaluator tool and moves the generated results to a specified location.
- Introduces a new PowerShell script that wraps the .NET Tool Description Evaluator
- Handles building, running, and collecting output from the evaluator with configurable embedding model endpoints
- Moves generated evaluation results (
results.md) to a target output location
|
The following pipelines have been queued for testing: |
smw-ms
changed the title
|
The following pipelines have been queued for testing: |
jeo02
left a comment
jeo02
left a comment
There was a problem hiding this comment.
Looks good overall would just go through the copilot suggestions and see what's useful then I'll give it a once over and approve.
|
The following pipelines have been queued for testing: |
Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#13218 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow) --------- Co-authored-by: Summer Warren <summerwarren@microsoft.com>
Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#13218 See [eng/common workflow](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/README.md#workflow) --------- Co-authored-by: Summer Warren <summerwarren@microsoft.com>
* Add script for tool description eval workflow * Fix param casing * respond to comments
4 participants
Resolves issue: #13148