Sync eng/common directory with azure-sdk-tools for PR 13693 - #40060
Merged
Azure Pipelines / azure-rest-api-specs - copilot - eval
failed
Jan 30, 2026 in 3m 38s
Build #20260130.1 had test failures
Details
- Failed: 2 (1.98%)
- Passed: 99 (98.02%)
- Other: 0 (0.00%)
- Total: 101
Annotations
Check failure on line 135 in Build log
azure-pipelines / azure-rest-api-specs - copilot - eval
Build log #L135
Script failed with exit code: 1
azure-pipelines / azure-rest-api-specs - copilot - eval
Azure.Sdk.Tools.Cli.Evaluations.Scenarios.Scenario.Evaluate_ValidateTypespec
Expected metric.Interpretation!.Failed to be False because Result did not match the expected outcome. Provided LLM Result contained no tool call. Expected to call: azsdk_verify_setup,azsdk_run_typespec_validation, but found True.
Raw output
at AwesomeAssertions.Execution.LateBoundTestFramework.Throw(String message)
at AwesomeAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
at AwesomeAssertions.Execution.AssertionChain.FailWith(Func`1 getFailureReason)
at AwesomeAssertions.Primitives.BooleanAssertions`1.BeFalse(String because, Object[] becauseArgs)
at Azure.Sdk.Tools.Cli.Evaluations.Helpers.EvaluationHelper.ValidateBooleanMetricEvaluator(EvaluationResult result, String metricName) in /mnt/vss/_work/1/s/azure-sdk-tools/tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/Helpers/EvaluationHelper.cs:line 20
at Azure.Sdk.Tools.Cli.Evaluations.Helpers.EvaluationHelper.ValidateToolInputsEvaluator(EvaluationResult result) in /mnt/vss/_work/1/s/azure-sdk-tools/tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/Helpers/EvaluationHelper.cs:line 27
at Azure.Sdk.Tools.Cli.Evaluations.Scenarios.Scenario.Evaluate_ValidateTypespec() in /mnt/vss/_work/1/s/azure-sdk-tools/tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/Scenarios/AzureRestApiSpecs/Evaluate_ValidateTypespec.cs:line 49
at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
azure-pipelines / azure-rest-api-specs - copilot - eval
Azure.Sdk.Tools.Cli.Evaluations.Scenarios.Scenario.Evaluate_AllToolsHaveTestPrompts
Coverage gap: 3 tool(s) have no test prompts in TestPrompts.json. Tool owners must add 2-3 prompt variations for each:
- azsdk_package_generate_samples
- azsdk_package_translate_samples
- azsdk_typespec_generate_authoring_plan
To add prompts, edit: tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/TestData/TestPrompts.json
Raw output
at Azure.Sdk.Tools.Cli.Evaluations.Scenarios.Scenario.Evaluate_AllToolsHaveTestPrompts() in /mnt/vss/_work/1/s/azure-sdk-tools/tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/Scenarios/General/Evaluate_PromptToToolMatch.cs:line 117
at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
1) at Azure.Sdk.Tools.Cli.Evaluations.Scenarios.Scenario.Evaluate_AllToolsHaveTestPrompts() in /mnt/vss/_work/1/s/azure-sdk-tools/tools/azsdk-cli/Azure.Sdk.Tools.Cli.Evaluations/Scenarios/General/Evaluate_PromptToToolMatch.cs:line 117
Loading