Skip to content

Conversation

peterwald
Copy link
Member

@peterwald peterwald commented Apr 8, 2025

The filter is OR'd with the tag filter, and is based on a fuzzy search of the scenario name, iteration name, and chat contents.

image

Microsoft Reviewers: Open in CodeFlow

@peterwald peterwald requested a review from a team as a code owner April 8, 2025 21:03
@peterwald peterwald self-assigned this Apr 8, 2025
@peterwald peterwald added the area-ai-eval Microsoft.Extensions.AI.Evaluation and related label Apr 8, 2025
Copy link
Contributor

@shyamnamboodiripad shyamnamboodiripad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The UX looks great. The only other feedback I have is that it would be nice to be able to search for text within metrics, diagnostics and metric metadata etc. This would enable scenarios like "which scenarios contain an evaluation result for metric xyz", "which other scenarios failed because of the same exception" etc.

There are also some more advanced cases that would likely need some form of search language / natural language search support. Examples include "filter down to all scenarios where metric xyz has value < 3", "which scenarios contain code vulnerability metric where code injection attack was detected in the metric metadata". We could wait for more feedback around the more advanced cases.

@peterwald
Copy link
Member Author

The UX looks great. The only other feedback I have is that it would be nice to be able to search for text within metrics, diagnostics and metric metadata etc. This would enable scenarios like "which scenarios contain an evaluation result for metric xyz", "which other scenarios failed because of the same exception" etc.

There are also some more advanced cases that would likely need some form of search language / natural language search support. Examples include "filter down to all scenarios where metric xyz has value < 3", "which scenarios contain code vulnerability metric where code injection attack was detected in the metric metadata". We could wait for more feedback around the more advanced cases.

Agree that functionality would be valuable. This is intended to be an initial first step. We'll probably need to create a more advanced query function to address those more complex scenarios.

@peterwald peterwald merged commit a3baf04 into dotnet:main Apr 9, 2025
6 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators May 10, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-ai-eval Microsoft.Extensions.AI.Evaluation and related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants