diff --git a/docs/core/testing/media/test-azdoreport-failure.png b/docs/core/testing/media/test-azdoreport-failure.png new file mode 100644 index 0000000000000..bc31b5c5a6d20 Binary files /dev/null and b/docs/core/testing/media/test-azdoreport-failure.png differ diff --git a/docs/core/testing/microsoft-testing-platform-extensions-test-reports.md b/docs/core/testing/microsoft-testing-platform-extensions-test-reports.md index d4c5fe6392844..568a815985230 100644 --- a/docs/core/testing/microsoft-testing-platform-extensions-test-reports.md +++ b/docs/core/testing/microsoft-testing-platform-extensions-test-reports.md @@ -27,3 +27,33 @@ The available options as follows: | `--report-trx-filename` | The name of the generated TRX report. The default name matches the following format `__.trx`. | The report is saved inside the default _TestResults_ folder that can be specified through the `--results-directory` command line argument. + +## Azure DevOps reports + +Azure DevOps report plugin enhances test running for developers that host their code on GitHub, but build on Azure DevOps build agents. It adds additional information to failures to show failure directly in GitHub PR. + +![Error annotation in GitHub PR files view](./media/test-azdoreport-failure.png) + +The extension is shipped in [Microsoft.Testing.Extensions.AzureDevOpsReport](https://nuget.org/packages/Microsoft.Testing.Extensions.AzureDevOpsReport) package. + +The available options as follows: + +| Option | Description | +|--|--| +| `--report-azdo` | Enable outputting errors / warnings in CI builds. | +| `--report-azdo-severity` | Severity to use for the reported event. Options are: `error` (default) and `warning`. | + +The extension automatically detects that it is running in continuous integration (CI) environment by checking the `TF_BUILD` environment variable. + +### Determining the line to report + +To highlight the correct line in code where failure occurred, AzureDevOps report plugin searches the error stacktrace for a file that exists in the current repository. + +To determine this it: + +- finds the repository root, this is done by searching the `.git` directory closest to the location from where the test application is started (as determined by `AppContext.BaseDirectory`). +- finds the first line in stack trace that has file location and line (the library needs to have debug symbols). +- excludes all files that end with `Assert.cs` to avoid showing details of your assertion implementations or wrappers. +- excludes all files that don't exist on disk (typically those are lines from external libraries that ship debug symbols e.g. MSTest). + +(These are internal details that serve to aid debugging the behavior and might change in the future.) diff --git a/docs/core/testing/unit-testing-mstest-sdk.md b/docs/core/testing/unit-testing-mstest-sdk.md index df8566a175799..df51a1d4f10a1 100644 --- a/docs/core/testing/unit-testing-mstest-sdk.md +++ b/docs/core/testing/unit-testing-mstest-sdk.md @@ -104,6 +104,7 @@ You can set the profile using the property `TestingExtensionsProfile` with one o * [Hot Reload](./microsoft-testing-platform-extensions-hosting.md#hot-reload) * [Retry](./microsoft-testing-platform-extensions-policy.md#retry) * [Trx Report](./microsoft-testing-platform-extensions-test-reports.md#visual-studio-test-reports) + * [AzureDevOpsReport](./microsoft-testing-platform-extensions-test-reports.md#azure-devops-reports) Here's a full example, using the `None` profile: @@ -118,17 +119,19 @@ Here's a full example, using the `None` profile: ``` -| Extension/Profile | None | Default | AllMicrosoft | -|-------------------------------------------------------------------------------------------|:----:|:------------------:|:--------------------------------------:| -| [Code Coverage](https://www.nuget.org/packages/Microsoft.Testing.Extensions.CodeCoverage) | | :heavy_check_mark: | :heavy_check_mark: | -| [Crash Dump](https://www.nuget.org/packages/Microsoft.Testing.Extensions.CrashDump) | | | :heavy_check_mark: | -| [Fakes](https://www.nuget.org/packages/Microsoft.Testing.Extensions.Fakes) | | | :heavy_check_mark:† | -| [Hang Dump](https://www.nuget.org/packages/Microsoft.Testing.Extensions.HangDump) | | | :heavy_check_mark: | -| [Hot Reload](https://www.nuget.org/packages/Microsoft.Testing.Extensions.HotReload) | | | :heavy_check_mark: | -| [Retry](https://www.nuget.org/packages/Microsoft.Testing.Extensions.Retry) | | | :heavy_check_mark: | -| [Trx](https://www.nuget.org/packages/Microsoft.Testing.Extensions.TrxReport) | | :heavy_check_mark: | :heavy_check_mark: | - -† MSTest.Sdk 3.7.0+ +| Extension/Profile | None | Default | AllMicrosoft | +| ------------------------------------------------------------------------------------------------- | :---: | :----------------: | :-----------------: | +| [Code Coverage](https://www.nuget.org/packages/Microsoft.Testing.Extensions.CodeCoverage) | | :heavy_check_mark: | :heavy_check_mark: | +| [Crash Dump](https://www.nuget.org/packages/Microsoft.Testing.Extensions.CrashDump) | | | :heavy_check_mark: | +| [Fakes](https://www.nuget.org/packages/Microsoft.Testing.Extensions.Fakes) | | | :heavy_check_mark:¹ | +| [Hang Dump](https://www.nuget.org/packages/Microsoft.Testing.Extensions.HangDump) | | | :heavy_check_mark: | +| [Hot Reload](https://www.nuget.org/packages/Microsoft.Testing.Extensions.HotReload) | | | :heavy_check_mark: | +| [Retry](https://www.nuget.org/packages/Microsoft.Testing.Extensions.Retry) | | | :heavy_check_mark: | +| [Trx](https://www.nuget.org/packages/Microsoft.Testing.Extensions.TrxReport) | | :heavy_check_mark: | :heavy_check_mark: | +| [AzureDevOpsReport](./microsoft-testing-platform-extensions-test-reports.md#azure-devops-reports) | | | :heavy_check_mark:² | + +¹ MSTest.Sdk 3.7.0+ +² MSTest.Sdk 3.11.0+ ### Enable or disable extensions