Add comprehensive async method tests and documentation for issue #1843#1864
Merged
Bertk merged 9 commits intocoverlet-coverage:masterfrom Mar 26, 2026
Merged
Add comprehensive async method tests and documentation for issue #1843#1864Bertk merged 9 commits intocoverlet-coverage:masterfrom
Bertk merged 9 commits intocoverlet-coverage:masterfrom
Conversation
…rage#1843 - Added Issue_1843_ComprehensiveAsync test class with 13 async method patterns - Covers Task, ValueTask, ConfigureAwait, branching, exception handling, LINQ, parallel calls, and async enumerables - Added 4 test methods to validate instrumentation completeness: 1. AsyncAwait_Issue_1843_ComprehensiveInstrumentation: Verifies all methods are instrumented and executed 2. AsyncAwait_Issue_1843_VerifyAllMethodsDiscovered: Ensures method discovery is complete before execution 3. AsyncAwait_Issue_1843_MultipleAwaitPoints: Tests varying complexity of async state machines 4. AsyncAwait_Issue_1843_VerifyMetricsNotRegressed: Validates coverage ratio and sequence points These tests address the core problem in issue coverlet-coverage#1843 where MTP reported only 35% of methods compared to MSBuild, catching scenarios where async methods might be silently skipped during instrumentation.
…#1843 tests - Analyzed why AsyncAwait_Issue_1843_VerifyAllMethodsDiscovered fails on .NET 10 - Root cause: .NET 10 compiler optimizations for async methods differ from .NET 8/9 - Identified 4 main failure scenarios: method count mismatch, state machine pattern changes, instrumentation filters, external process execution - Proposed 4-phase solution: diagnostic investigation, test infrastructure updates, code adjustments, validation - Recommended immediate actions: add diagnostic logging, run isolated tests, determine root cause - Provided 3 options for test infrastructure: run on all frameworks, dedicated .NET 10 job, or add framework parameter - Included code examples for framework-specific test adjustments - Risk assessment and success criteria defined
- Corrected test command examples to use MTP syntax instead of VSTest - Added comparison table showing VSTest vs MTP filter differences - Clarified that --filter-method expects method name WITHOUT 'FullyQualifiedName=' prefix - Added wildcard example for running all Issue coverlet-coverage#1843 tests - Updated both implementation and failure analysis documents Key fix: --filter-method 'Namespace.Class.Method' (NOT 'FullyQualifiedName=...') This addresses why 'Zero tests ran' when trying to run Issue coverlet-coverage#1843 tests on .NET 10
…d module file existen ce checkThe AsyncAwait_Issue_1843_VerifyAllMethodsDiscovered test was failing on all frameworks because it checked if the instrumented module file exists on disk. This check is unnecessary because: 1. Instrumented modules may be cleaned up immediately after use 2. .NET 10+ might use in-memory assemblies 3. The critical validation is method discovery, not file persistence Changes: - Removed File.Exists check for instrumentedResult.Module - Added comment explaining why file existence is not validated - Test now focuses on what matters: verifying that all 13 async methods are discovered during instrumentation Test results: - ✅ All 3 frameworks pass (net8.0, net9.0, net10.0) - ✅ Specifically tested on .NET 10.0 with correct MTP filter syntax This fix allows the test to properly validate the core issue coverlet-coverage#1843 symptom (missing method discovery) without being blocked by unrelated file system timing issues.
Bertk
force-pushed
the
coverlet-issue-1843
branch
from
bc1d31a to
46a0f2f
Compare
…overage Added comprehensive validation tests to catch the Issue coverlet-coverage#1843 bug where coverage becomes completely empty when multiple SUT classes all have async methods. New validation tests in coverlet.MTP.validation.tests: 1. Issue1843_AsyncMethods_CoverageDataNotEmpty - Tests that async methods in SUT classes generate proper coverage data - Creates test project with MULTIPLE classes that ALL have async methods - Verifies coverage XML contains classes and line data - CRITICAL: This catches the exact symptom where <classes /> becomes empty 2. Issue1843_BothClassesAsync_CoverageNotEmpty - Exact reproduction of Issue coverlet-coverage#1843 scenario - Creates StringLengthCalculator and IntegerFormatter both with async methods - Verifies both classes appear in coverage - This reproduces the 'both methods are async' case that caused empty coverage Helper methods added: - CreateAsyncTestProject: Creates test project with multiple async SUT classes - CreateIssue1843ExactReproProject: Creates exact repro of Issue coverlet-coverage#1843 - CreateAsyncSutLibraryProject: Generates SUT with AsyncCalculator, AsyncStringProcessor, AsyncDataFetcher - CreateAsyncTestProjectFiles: Generates async test cases - CreateIssue1843SutProject: Generates exact repro SUT (StringLengthCalculator + IntegerFormatter) - CreateIssue1843TestProject: Generates exact repro tests Root cause hypothesis: System.Text.Json migration (commit coverlet-coverage#1733) may have issues with: - DictionaryKeyPolicy = JsonNamingPolicy.CamelCase affecting async state machine class names - Async state machine names containing '<' and '>' characters being incorrectly serialized - Key mismatches in coverage data dictionaries when deserializing These tests will fail if the Issue coverlet-coverage#1843 regression recurs, providing automatic protection against future async coverage bugs.
- Do not exclude async/iterator state machine types from coverage (fixes coverlet-coverage#1843) - Remove DictionaryKeyPolicy.CamelCase to preserve dictionary keys in JSON output and merging - Add environment variable for coverage enabled flag in test host provider - Add comprehensive tests for JSON serialization/deserialization, async class names, and branch data - Update test code snippets with copyright/license headers - Prevent regressions in coverage reporting and merging for async scenarios
Contributor
There was a problem hiding this comment.
Pull request overview
This PR targets the coverage regression described in issue #1843 by adjusting how compiler-generated async/iterator state machine types are handled during instrumentation, and by adding multiple layers of tests (core coverage + MTP validation) plus supporting documentation.
Changes:
- Adjust coverlet.core instrumentation and MergeWith deserialization settings to better preserve async state machine coverage and JSON key fidelity.
- Add comprehensive async-await sample methods and new coverage/validation tests to catch regressions like “empty coverage” in MTP scenarios.
- Add documentation describing the test implementation and a separate .NET 10 failure analysis write-up.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/coverlet.core.tests/Reporters/JsonSerializationTests.cs | Adds JSON serialization/deserialization tests around async state machine naming and key-policy behavior. |
| test/coverlet.core.coverage.tests/Samples/Instrumentation.AsyncAwait.cs | Adds a new Issue_1843_ComprehensiveAsync sample class with many async patterns for instrumentation validation. |
| test/coverlet.core.coverage.tests/Coverage/CoverageTests.AsyncAwait.cs | Adds multiple coverage tests asserting async method discovery/instrumentation completeness for issue #1843. |
| test/coverlet.MTP.validation.tests/CollectCoverageTests.cs | Adds MTP end-to-end validation tests that ensure async-heavy SUTs still produce non-empty coverage output. |
| src/coverlet.core/Instrumentation/Instrumenter.cs | Changes type-exclusion behavior to avoid losing async/iterator state machine coverage due to exclude-attribute configuration. |
| src/coverlet.core/Coverage.cs | Removes DictionaryKeyPolicy from the JSON options used for MergeWith deserialization. |
| src/coverlet.MTP/EnvironmentVariables/CoverletExtensionEnvironmentVariableProvider.cs | Propagates a “coverage enabled” env var to the test host when --coverlet is provided. |
| Documentation/Issues/issue-1843-test-implementation.md | Adds a test implementation summary and execution guidance. |
| Documentation/Issues/issue-1843-net10-failure-analysis.md | Adds a detailed analysis document about a .NET 10-specific failure scenario. |
…1843) Improve instrumentation to ensure async/iterator state machines are not excluded solely due to [CompilerGenerated] or [GeneratedCode] attributes, but still honor user-configured exclusions. Add helper for attribute checks and expand interface detection. Update tests and sample code to streamline async method calls and clarify test approach. Add documentation note on end-to-end test methodology.
Bertk
added
bug
This was referenced Apr 17, 2026
This was referenced Apr 18, 2026
Open
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces comprehensive tests to address the coverage regression described in issue #1843, specifically targeting async method instrumentation in the Coverlet project. The changes add a new test class with 13 diverse async method patterns and four new unit tests to ensure that async methods are properly discovered, instrumented, and reported by the coverage tool. These tests are designed to catch regressions similar to those reported in the issue and to ensure future robustness for async code coverage.
New async method samples and test coverage for issue #1843:
Issue_1843_ComprehensiveAsyncclass inInstrumentation.AsyncAwait.cscontaining 13 async method patterns, including variations withTask,ValueTask,ConfigureAwait, branching, exception handling, LINQ, parallelism, multiple await points, switch expressions, null coalescing, andIAsyncEnumerable.CoverageTests.AsyncAwait.csto:Documentation and test execution guidance:
issue-1843-test-implementation.md) documenting the problem, root cause analysis, test implementation, expected outcomes, and instructions for building and running the new tests.