Skip to content

Fail the tests when an analyzer throws - #1349

Merged
meziantou merged 1 commit into
mainfrom
feature/meziantou-analyzer-1331-991325
Aug 27, 2026
Merged

meziantou merged 1 commit into
mainfrom
feature/meziantou-analyzer-1331-991325

Conversation

@meziantou

Copy link
Copy Markdown
Owner

Fixes #1331

What changed

VerifyDiagnosticResults filters the actual diagnostics by the configured default rule id before counting them. Roslyn surfaces an analyzer exception as an AD0001 diagnostic with Location.None: GetSortedDiagnosticsFromDocuments did add it to the list, and this filter then discarded it. In a test that expects zero diagnostics, expected and actual were both 0 and the test was green even though the analyzer threw.

The harness now passes CompilationWithAnalyzersOptions with an onAnalyzerException handler that records the exception, and AssertNoAnalyzerException fails right after GetAnalyzerDiagnosticsAsync — before the DefaultAnalyzerId filter, before sorting, and on the code fix paths too, as all of them go through GetSortedDiagnosticsFromDocuments. The failure message contains the analyzer type and the full exception with its stack trace, which is more actionable than the AD0001 message. A check on the AD0001 diagnostics remains as a safety net in case Roslyn ever reports a failure without invoking the handler.

VerifyDiagnostic_ReportsAnalyzerException covers it: an analyzer that throws, registered with id: so the DefaultAnalyzerId filtering is active, and no expected diagnostic — the exact shape that was silently green before.

Verification

  • dotnet test --max-parallel-test-modules 2: 18789 passed, 0 failed, for the 5 Roslyn versions (4.8, 4.14, 5.0, 5.6, 5.9). The CompilationWithAnalyzersOptions constructor overload works on 4.8.
  • No pre-existing crash surfaced: the two issues filed alongside are already fixed on main (60df5ba, c6054ab).
  • dotnet run --project src/DocumentationGenerator exits 0 and does not change any markdown file, as this is a test-only change.

Not included: the same filter hiding a second rule

The issue also mentions that the DefaultAnalyzerId filter hides a wrong second rule from the multi-descriptor analyzers. Removing the filter breaks 27 tests on roslyn5.9: OptimizeLinqUsageAnalyzer (12), UseStringComparisonAnalyzer (5), UseLangwordInXmlComment (3), ConcurrentDictionaryMustPreventClosureWhenAccessingTheKey (3), and 4 others. Each one needs a decision on whether the second diagnostic is expected or a false positive, so this PR leaves it out rather than blanket-annotating them. Happy to do it in a follow-up.

VerifyDiagnosticResults filters the actual diagnostics by the configured
default rule id before counting them. Roslyn surfaces an analyzer
exception as an AD0001 diagnostic, so that filter discarded it and a test
expecting no diagnostic stayed green even though the analyzer crashed.

Pass CompilationWithAnalyzersOptions with an onAnalyzerException handler
that records the exception, and fail right after
GetAnalyzerDiagnosticsAsync, before any filtering and on the code fix
paths too. The message contains the analyzer type and the full exception,
which is more actionable than the AD0001 message. Keep a check on the
AD0001 diagnostics as a safety net.
@meziantou
meziantou merged commit 3b731e6 into main Aug 27, 2026
13 checks passed
@meziantou
meziantou deleted the feature/meziantou-analyzer-1331-991325 branch August 27, 2026 00:13
This was referenced Aug 27, 2026
IhateTrains pushed a commit to ParadoxGameConverters/ImperatorToCK3 that referenced this pull request Aug 27, 2026
Updated
[Meziantou.Analyzer](https://github.com/meziantou/Meziantou.Analyzer)
from 3.0.177 to 3.0.184.

<details>
<summary>Release notes</summary>

_Sourced from [Meziantou.Analyzer's
releases](https://github.com/meziantou/Meziantou.Analyzer/releases)._

## 3.0.184

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.184>

## What's Changed
* Compute the package version from the git tags instead of nuget.org by
@​meziantou in meziantou/Meziantou.Analyzer#1350


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.183...3.0.184

## 3.0.183

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.183>

## What's Changed
* Fix MA0026 crash on an unterminated block comment (#​1328) by
@​meziantou in meziantou/Meziantou.Analyzer#1346
* Fix MA0050 code fixer producing an extension local function (CS1106)
by @​meziantou in
meziantou/Meziantou.Analyzer#1348
* Fix MA0028 code fixer producing uncompilable code (#​1327) by
@​meziantou in meziantou/Meziantou.Analyzer#1347
* Fail the tests when an analyzer throws by @​meziantou in
meziantou/Meziantou.Analyzer#1349


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.182...3.0.183

## 3.0.182

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.182>

## What's Changed
* Fix MA0073 code fixer producing uncompilable code for low-precedence
operands by @​meziantou in
meziantou/Meziantou.Analyzer#1344
* Do not crash on an invalid regex in the .editorconfig (MA0003, MA0104)
by @​meziantou in
meziantou/Meziantou.Analyzer#1345


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.181...3.0.182

## 3.0.181

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.181>

## What's Changed
* Add a language attribute code fix and MA0218/MA0219 for XML comments
by @​meziantou in
meziantou/Meziantou.Analyzer#1320
* Compile the text produced by the fixer in the test harness by
@​meziantou in meziantou/Meziantou.Analyzer#1342
* Fix StackOverflow crash on cyclic local initializers
(MA0091/MA0092/MA0093) by @​meziantou in
meziantou/Meziantou.Analyzer#1341
* Avoid the Sqlite data-flow analysis on every MA0042 invocation by
@​meziantou in meziantou/Meziantou.Analyzer#1343


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.180...3.0.181

## 3.0.180

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.180>

## What's Changed
* Fix MAS0004 stopping at the first unrelated CA1507 diagnostic by
@​meziantou in meziantou/Meziantou.Analyzer#1340


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.179...3.0.180

## 3.0.179

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.179>

## What's Changed
* Fix bare `#pragma warning disable` in AvoidUsingRedundantElseAnalyzer
by @​meziantou in
meziantou/Meziantou.Analyzer#1339


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.178...3.0.179

## 3.0.178

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.178>

## What's Changed
* Fix MA0075 false positive on FormattableString.Invariant by
@​meziantou in meziantou/Meziantou.Analyzer#1322


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.177...3.0.178

Commits viewable in [compare
view](meziantou/Meziantou.Analyzer@3.0.177...3.0.184).
</details>

[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Meziantou.Analyzer&package-manager=nuget&previous-version=3.0.177&new-version=3.0.184)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test harness: analyzer crashes (AD0001) are filtered out of diagnostic assertions

1 participant