-
Notifications
You must be signed in to change notification settings - Fork 841
Add MEAI.Evaluation libraries #5873
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
41 commits
Select commit
Hold shift + click to select a range
cf0266a
Add MEAI.Evaluation libraries
shyamnamboodiripad 39638f8
Adapt to IChatClient renames
stephentoub 71cda8d
Add docs to test configuration script.
peterwald 1444c71
Version and build the Azure DevOps extension
peterwald 8a28d07
Feedback: Use better overload
shyamnamboodiripad 85af81a
Remove Microsoft.Bcl.Memory dependency since its no longer required
shyamnamboodiripad 850acf4
Feedback: Audit and remove redundant cancellationToken.ThrowIfCancell…
shyamnamboodiripad 63a5163
Feedback: Simplify null checks
shyamnamboodiripad c8aa7a7
Feedback: Simplify type declaration
shyamnamboodiripad a7ddf47
Add missing sealed modifier
shyamnamboodiripad e563454
Feedback: Simplify substring search
shyamnamboodiripad 2752ac5
Feedback: Avoid allocating intermediate string
shyamnamboodiripad a71fa1b
Feedback: Simplify initialization
shyamnamboodiripad eaeb44c
Feedback: Simplify empty collection usage
shyamnamboodiripad c6b7401
Feedback: Use Dictionary.TryAdd on .NET Core
shyamnamboodiripad c1f7caa
Feedback: Use Task.WhenEach on .NET 9
shyamnamboodiripad b7c0c9d
Suppress API compat validation
shyamnamboodiripad 59e6721
Fix some typos
shyamnamboodiripad 025a79a
Feedback: Use WithCancellation
shyamnamboodiripad 0a6ef14
Remove SuppressFinalPackageVersion
shyamnamboodiripad 8ccb695
Feedback: Suppress async warning as opposed to adding empty awaits
shyamnamboodiripad 29bd92f
Feedback: Explicitly turn on asynchronous I/O on streams
shyamnamboodiripad 485699c
Feedback: Convert field to property
shyamnamboodiripad 2b3be06
Feedback: Convert readonly fileds to properties
shyamnamboodiripad 9539fb5
Rremove some stale comments
shyamnamboodiripad ea8b7a7
Add some comments to capture follow up tasks for feedback
shyamnamboodiripad a178ff5
Use span for CacheEntry Deserialize
peterwald cce2db6
Merge branch 'shyam-eval' into ado-azdo
peterwald e083f8f
Stamp azdo package version during C# build.
peterwald aa27138
Fix the RTC rating serializer.
peterwald aa5609a
Stamp azdo package version during C# build.
peterwald 091083e
Fix the RTC rating serializer.
peterwald 78e8b76
Update azdo extension build
peterwald a47f94c
Merge branch 'main' into eval
peterwald 5c0a781
Add issue link.
shyamnamboodiripad 4bf28f1
Fix test configuration script.
peterwald 2e4550e
Merge branch 'eval' of https://github.com/shyamnamboodiripad/extensio…
peterwald 4e9b721
Feedback: Avoid deep recursion
shyamnamboodiripad f4a5b84
Avoid infinite loops and add tests
shyamnamboodiripad 53a39f5
Rename tests for added clarity
shyamnamboodiripad 4125bf8
Merge branch 'main' into eval
shyamnamboodiripad File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| { | ||
| "diagnosticMessages": true, | ||
| "longRunningTestSeconds": 300 | ||
| "longRunningTestSeconds": 300, | ||
| "shadowCopy": false | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| #!/usr/bin/env pwsh | ||
|
|
||
| <# | ||
| .SYNOPSIS | ||
| Configures local repo for online evaluation tests, which use external resources. | ||
|
|
||
| .DESCRIPTION | ||
| This script copies appsettings files from a location on the developer's machine to the test | ||
| project directories so that the tests are configured to connect to external resources. The online | ||
| configuration files are gitignore'd and not checked in to the repo. | ||
|
|
||
| .PARAMETER Configure | ||
| Configure this repo for online evaluation tests, by copying appsettings files from the developer's | ||
| machine to this repo. | ||
| .PARAMETER Unconfigure | ||
| Unconfigure this repo for online evaluation tests, by removing the appsettings files from this repo. | ||
| .PARAMETER ConfigRoot | ||
| ConfigRoot specifies where to copy the configuration files from. The default is $HOME/.config/dotnet-extensions. | ||
| #> | ||
|
|
||
| param ( | ||
shyamnamboodiripad marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| [switch]$Configure=$False, | ||
| [switch]$Unconfigure=$False, | ||
| [string]$ConfigRoot=$Null | ||
| ) | ||
|
|
||
| Write-Host "$PSScriptRoot" | ||
|
|
||
| if ($Configure -and $Unconfigure) { | ||
| Write-Error -Message "Cannot specify both -Configure and -Unconfigure" | ||
| Exit 1 | ||
| } | ||
|
|
||
| if (!(Test-Path $ConfigRoot)) { | ||
| $ConfigRoot = "$HOME/.config/dotnet-extensions" | ||
| } | ||
|
|
||
| $ProjectRoot = Resolve-Path "$PSScriptRoot/../test/Libraries" | ||
| $ReportingConfig = "Microsoft.Extensions.AI.Evaluation.Reporting.Tests/appsettings.local.json" | ||
| $IntegrationConfig = "Microsoft.Extensions.AI.Evaluation.Integration.Tests/appsettings.local.json" | ||
|
|
||
| if ($Configure) { | ||
| if (!(Test-Path -Path "$ConfigRoot/$ReportingConfig")) { | ||
| Write-Host "No configuration found at $ConfigRoot/$ReportingConfig" | ||
| Exit 0 | ||
| } | ||
| if (!(Test-Path -Path "$ConfigRoot/$IntegrationConfig")) { | ||
| Write-Host "No configuration found at $ConfigRoot/$IntegrationConfig" | ||
| Exit 0 | ||
| } | ||
|
|
||
| Copy-Item -Path "$ConfigRoot/$ReportingConfig" -Destination "$ProjectRoot/$ReportingConfig" -Force | ||
| Copy-Item -Path "$ConfigRoot/$IntegrationConfig" -Destination "$ProjectRoot/$IntegrationConfig" -Force | ||
|
|
||
| Write-Host "Test configured to use external resources" | ||
| } elseif ($Unconfigure) { | ||
| Remove-Item -Path "$ProjectRoot/$ReportingConfig" -Force | ||
| Remove-Item -Path "$ProjectRoot/$IntegrationConfig" -Force | ||
|
|
||
| Write-Host "Test unconfigured from using external resources" | ||
| } else { | ||
| Write-Error -Message "Must specify either -Configure or -Unconfigure" | ||
| Exit 1 | ||
| } | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
28 changes: 28 additions & 0 deletions
28
src/Libraries/Microsoft.Extensions.AI.Evaluation.Console/Commands/CleanCacheCommand.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. | ||
|
|
||
| using System.IO; | ||
| using System.Threading; | ||
| using System.Threading.Tasks; | ||
| using Microsoft.Extensions.AI.Evaluation.Console.Utilities; | ||
| using Microsoft.Extensions.AI.Evaluation.Reporting.Storage; | ||
| using Microsoft.Extensions.Logging; | ||
|
|
||
| namespace Microsoft.Extensions.AI.Evaluation.Console.Commands; | ||
|
|
||
| internal sealed class CleanCacheCommand(ILogger logger) | ||
| { | ||
| internal async Task<int> InvokeAsync(DirectoryInfo storageRootDir, CancellationToken cancellationToken = default) | ||
| { | ||
| string storageRootPath = storageRootDir.FullName; | ||
| logger.LogInformation("Storage root path: {storageRootPath}", storageRootPath); | ||
| logger.LogInformation("Deleting expired cache entries..."); | ||
stephentoub marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| var cacheProvider = new DiskBasedResponseCacheProvider(storageRootPath); | ||
|
|
||
| await logger.ExecuteWithCatchAsync( | ||
| () => cacheProvider.DeleteExpiredCacheEntriesAsync(cancellationToken)).ConfigureAwait(false); | ||
|
|
||
| return 0; | ||
stephentoub marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| } | ||
63 changes: 63 additions & 0 deletions
63
src/Libraries/Microsoft.Extensions.AI.Evaluation.Console/Commands/CleanResultsCommand.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. | ||
|
|
||
| using System.Collections.Generic; | ||
| using System.IO; | ||
| using System.Threading; | ||
| using System.Threading.Tasks; | ||
| using Microsoft.Extensions.AI.Evaluation.Console.Utilities; | ||
| using Microsoft.Extensions.AI.Evaluation.Reporting.Storage; | ||
| using Microsoft.Extensions.Logging; | ||
|
|
||
| namespace Microsoft.Extensions.AI.Evaluation.Console.Commands; | ||
|
|
||
| internal sealed class CleanResultsCommand(ILogger logger) | ||
| { | ||
| internal async Task<int> InvokeAsync( | ||
| DirectoryInfo storageRootDir, | ||
| int lastN, | ||
| CancellationToken cancellationToken = default) | ||
| { | ||
| string storageRootPath = storageRootDir.FullName; | ||
| logger.LogInformation("Storage root path: {storageRootPath}", storageRootPath); | ||
|
|
||
| var resultStore = new DiskBasedResultStore(storageRootPath); | ||
|
|
||
| await logger.ExecuteWithCatchAsync( | ||
| async ValueTask () => | ||
shyamnamboodiripad marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| { | ||
| if (lastN is 0) | ||
| { | ||
| logger.LogInformation("Deleting all results..."); | ||
|
|
||
| await resultStore.DeleteResultsAsync(cancellationToken: cancellationToken).ConfigureAwait(false); | ||
| } | ||
| else | ||
| { | ||
| logger.LogInformation("Deleting all results except the {lastN} most recent ones...", lastN); | ||
|
|
||
| HashSet<string> toPreserve = []; | ||
|
|
||
| await foreach (string executionName in | ||
| resultStore.GetLatestExecutionNamesAsync(lastN, cancellationToken).ConfigureAwait(false)) | ||
| { | ||
| _ = toPreserve.Add(executionName); | ||
| } | ||
shyamnamboodiripad marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| await foreach (string executionName in | ||
| resultStore.GetLatestExecutionNamesAsync( | ||
shyamnamboodiripad marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| cancellationToken: cancellationToken).ConfigureAwait(false)) | ||
| { | ||
| if (!toPreserve.Contains(executionName)) | ||
| { | ||
| await resultStore.DeleteResultsAsync( | ||
| executionName, | ||
| cancellationToken: cancellationToken).ConfigureAwait(false); | ||
| } | ||
| } | ||
| } | ||
| }).ConfigureAwait(false); | ||
|
|
||
| return 0; | ||
| } | ||
| } | ||
13 changes: 13 additions & 0 deletions
13
src/Libraries/Microsoft.Extensions.AI.Evaluation.Console/Commands/ReportCommand.Format.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. | ||
|
|
||
| namespace Microsoft.Extensions.AI.Evaluation.Console.Commands; | ||
|
|
||
| internal partial class ReportCommand | ||
| { | ||
| internal enum Format | ||
| { | ||
| html, | ||
| json | ||
| } | ||
| } |
63 changes: 63 additions & 0 deletions
63
src/Libraries/Microsoft.Extensions.AI.Evaluation.Console/Commands/ReportCommand.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| // Licensed to the .NET Foundation under one or more agreements. | ||
| // The .NET Foundation licenses this file to you under the MIT license. | ||
|
|
||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.IO; | ||
| using System.Threading; | ||
| using System.Threading.Tasks; | ||
| using Microsoft.Extensions.AI.Evaluation.Reporting; | ||
| using Microsoft.Extensions.AI.Evaluation.Reporting.Formats.Html; | ||
| using Microsoft.Extensions.AI.Evaluation.Reporting.Formats.Json; | ||
| using Microsoft.Extensions.AI.Evaluation.Reporting.Storage; | ||
| using Microsoft.Extensions.Logging; | ||
|
|
||
| namespace Microsoft.Extensions.AI.Evaluation.Console.Commands; | ||
|
|
||
| internal sealed partial class ReportCommand(ILogger logger) | ||
| { | ||
| internal async Task<int> InvokeAsync( | ||
| DirectoryInfo storageRootDir, | ||
| FileInfo outputFile, | ||
| int lastN, | ||
| Format format, | ||
| CancellationToken cancellationToken = default) | ||
| { | ||
| string storageRootPath = storageRootDir.FullName; | ||
| logger.LogInformation("Storage root path: {storageRootPath}", storageRootPath); | ||
|
|
||
| var results = new List<ScenarioRunResult>(); | ||
| var resultStore = new DiskBasedResultStore(storageRootPath); | ||
|
|
||
| await foreach (string executionName in | ||
| resultStore.GetLatestExecutionNamesAsync(lastN, cancellationToken).ConfigureAwait(false)) | ||
| { | ||
| await foreach (ScenarioRunResult result in | ||
| resultStore.ReadResultsAsync( | ||
| executionName, | ||
| cancellationToken: cancellationToken).ConfigureAwait(false)) | ||
| { | ||
| results.Add(result); | ||
| } | ||
| } | ||
|
|
||
| string outputFilePath = outputFile.FullName; | ||
| string? outputPath = Path.GetDirectoryName(outputFilePath); | ||
| if (outputPath is not null && !Directory.Exists(outputPath)) | ||
| { | ||
| _ = Directory.CreateDirectory(outputPath); | ||
| } | ||
|
|
||
| IEvaluationReportWriter reportWriter = format switch | ||
| { | ||
| Format.html => new HtmlReportWriter(outputFilePath), | ||
| Format.json => new JsonReportWriter(outputFilePath), | ||
| _ => throw new NotSupportedException(), | ||
| }; | ||
|
|
||
| await reportWriter.WriteReportAsync(results, cancellationToken).ConfigureAwait(false); | ||
| logger.LogInformation("Report: {outputFilePath} [{format}]", outputFilePath, format); | ||
|
|
||
| return 0; | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.