Repository navigation
Fix TUnit0014 on interface implementations, Aspire exit-code culture and [AssertionFrom] expectation messages - #7135
Conversation
ReferencesHelper builds the compilation's references from the assemblies already loaded in the test process. System.Net.Primitives (home of HttpStatusCode on .NET) is not loaded by a plain run of the test exe, but is loaded by server-mode hosts (`dotnet test`, Rider, VS). The .NET snapshots were captured with HttpStatusCode unresolved (bare type name, spurious `value is null` checks), so the test failed whenever it ran from an IDE or `dotnet test`. Reference the assembly explicitly and update the net8/9/10 snapshots; they now match the existing Net4_7 snapshot.
TUnit.Engine.Tests launches TUnit.TestProject, its F#/VB.NET siblings and TUnit.TestProject.HtmlReportDefaults from hard-coded bin/Release folders. CI builds everything in Release, but an IDE builds Debug and does not build TestProject/F#/VB at all (they are not in TUnit.Dev.slnx), so locally the suite failed with missing executables or ran stale Release binaries. Add build-only project references for the launched apps and resolve their output folder from this assembly's own configuration. CI is unchanged (Release/Release); a Debug run from Rider/VS/dotnet test now builds what it launches and passes without manual Release builds.
The project was an xUnit v2 VSTest project without a runner adapter, while
global.json makes `dotnet test` Microsoft.Testing.Platform-only. `dotnet
test` therefore refused it ("All projects must use that test runner"),
which also aborted solution-wide runs of TUnit.slnx, and its tests only ran
under an IDE's bundled xUnit runner.
Move it to xunit.v3 with the MTP runner (already pinned centrally). No test
code changes; all 64 tests pass both via `dotnet test` and as a plain
executable.
RpcTests and the Engine F#/VB/HTML-report tests start `dotnet build` / `dotnet run` via a bare "dotnet", i.e. whatever is first on PATH. An IDE does not necessarily share that PATH with the installation it builds and runs tests with (notably macOS apps started from the Dock), and that dotnet can lack an SDK satisfying global.json: the child then exits with 155 (SdkResolveFailure) and the tests fail with no explanation. Resolve the muxer from DOTNET_HOST_PATH, else from the runtime running the test process, falling back to PATH. Include the build output in the RpcTests pre-build failure message.
These tests compare against invariant spellings ("1.1", "1.5", "-1",
"[-5]") of values formatted with the current culture, so they failed on
machines with e.g. sv-SE (decimal comma, U+2212 MINUS SIGN). In
TUnit.TestProject this also failed TUnit.Engine.Tests' ExpectedStateTests,
which requires every EngineTest=Pass test to pass.
Mark them [InvariantCulture]; the product's culture-aware formatting is
unchanged.
Exit codes are identifiers people search for (e.g. -532462766 / 0xE0434352, the .NET unhandled exception code). Under cultures such as sv-SE the diagnostics printed negative codes with U+2212 MINUS SIGN, which also failed FormatExitCode_KnownCode_IncludesNumberAndMeaning.
An earlier commit in this branch marked GenericArgumentsTest [InvariantCulture], but TUnit.Core.SourceGenerator.Tests snapshots the generator output for this file and the generator emits the test's attributes (and line numbers), so the snapshot no longer matched. Format the value invariantly in the method body instead, which leaves the generated metadata unchanged.
Both projects are TUnit-only (no xUnit tests or usings) but referenced xunit and xunit.runner.visualstudio, which also put a VSTest adapter into these Microsoft.Testing.Platform test apps.
…ning Replace the [InvariantCulture] pins added earlier in this branch, which only hid the tests' implicit en-US assumption, with expectations that hold in any culture: - CollectionAssertionTests: TUnit formats values in failure messages with the current culture, so build the expected "[-5]" the same way and keep testing what a user in that culture sees. - AsyncTests: compare against the message formatted like the throwing factory formats it. - WrapMultiParam fixture: culture is incidental, format invariantly. Verified under en-US, sv-SE, de-DE, tr-TR and ar-SA.
The last non-TUnit test project in the repo. It now uses the shared eng/TestProject.props/.targets setup like every other test project, so it runs on the in-repo engine and analyzers. Attributes were converted with TUnit's own TUXU0001 code fixer; the assertions (which the fixer left as Xunit.Assert in these synchronous methods) were converted to awaited TUnit assertions, with the test methods and assertion helpers made async. Built with warnings as errors to rule out un-awaited helpers; a deliberately broken shared helper fails 8 tests, so assertions propagate. All 64 tests pass via `dotnet test` and as a plain executable. Drops an unused helper.
The project's 64 generator-caching tests were in TUnit.slnx only, so CI neither built nor ran them. Add it to TUnit.CI.slnx and a pipeline module that runs it like the other test projects (Release, net10.0).
AssertionMethodGenerator rendered a custom ExpectationMessage as
`{(negated ? "not " : "")} {message}`, so every [AssertionFrom]
assertion with a custom message (e.g. string.IsNullOrEmpty, char
IsUpper, Task IsCompleted; their messages omit the leading "to" by
convention) failed with "Expected be null or empty" (double space, no
"to"), and negated ones with "Expected not be ...".
Emit `to {(negated ? "not " : "")}{message}` instead, matching the
"to not ..." wording used elsewhere. Snapshots updated (only that line
changes); Net4_7 snapshots received the identical substitution as they
cannot be regenerated off Windows. Adds a message regression test.
- RangeAssertionTests: HasEndFromBeginning and IsFullRange (successor of the obsolete IsAll) had no active tests, only commented-out ones saying the generator did not produce them yet. - AssertionGroupTests: the And/Or grouping tests over IsNullOrEmpty were commented out as "not available"; reviving them exposed the message bug fixed in the previous commit. - Old/StringAssertionTests: drop commented-out IsNullOrEmpty / IsNullOrWhiteSpace tests that StringNullabilityAssertionTests already covers, and their stale "not available" note.
…lation DesignTimeProjectReferenceTests started `dotnet build` via a bare "dotnet" from PATH, like the RpcTests/Engine tests fixed earlier in this branch. Use the shared DotNetCli resolver so it uses the same installation.
TUnit0014 ("public method missing [Test]; add the attribute or make it
private") special-cased Dispose/DisposeAsync/InitializeAsync by name but
not interface implementations in general, so a test class implementing
e.g. ITestStartEventReceiver got a warning on OnTestStart that it could
not follow without breaking the interface (seen in
TUnit.UnitTests/PlaywrightCompositionLifecycleTests).
Skip methods that implement an interface member of the containing type.
A same-named public method that does not implement the member is still
reported.
…hots - TUnit.Assertions.SourceGenerator.Tests (70 snapshot tests per TFM) had no pipeline module, so CI never ran it; that is how a wrong HttpStatusCode snapshot and the "Expected be ..." message bug went unnoticed. Add a module mirroring RunSourceGeneratorTestsModule. - RunSourceGeneratorTestsModule says to stay aligned with the project's TargetFrameworks but skipped net9.0, leaving the DotNet9_0 snapshots unchecked. Add it.
Five generator test projects built their in-memory compilations from AppDomain.CurrentDomain.GetAssemblies(), i.e. whatever the process had loaded so far. That set depends on how the tests are launched (IDE and `dotnet test` server-mode hosts load assemblies a plain run does not), which is how HttpStatusCode resolved in one mode and not the other (see the HttpStatusCode snapshot fix earlier in this branch). The Should tests' comment even assumed it reflected TRUSTED_PLATFORM_ASSEMBLIES; it does not. Add a shared TestHostReferences helper: on .NET it returns the TRUSTED_PLATFORM_ASSEMBLIES list (framework + deps.json, fixed by the build); .NET Framework has no such list and keeps the loaded assemblies, so the Windows-only Net4_7 snapshots are unaffected. Drops the now redundant explicit HttpStatusCode reference. No snapshot changes; all five projects pass on net8/9/10 via `dotnet test` and as plain runs.
Most test modules ran net10.0 only (TestBaseModule's default), Mocks.Http, Mocks.Logging and AspNetCore.Tests ran net10.0+net8.0, and the five analyzer modules ran a single hard-coded TFM via `dotnet test` (net8.0, or net9.0 for AspNetCore analyzers), although their projects build net8.0/net9.0/net10.0 and some contain TFM-specific tests (e.g. TUnit.Assertions.Tests has a different test set on net8.0). - TestBaseModule now defaults to net10.0, net9.0 and net8.0; frameworks a project did not build are already skipped, so net10.0-only projects still run once. - Drop the net10.0+net8.0 overrides. - Move the analyzer modules onto TestBaseModule, keeping their hang dump options with a per-framework dump file name.
…p dead files - tests/TUnit.Core.Tests and tests/TUnit.IntegrationTests held test files but no project, so they were never compiled or run (the former used NUnit's Assert.That(x, Is.EqualTo(...)) and an invalid `var` tuple with a null element). ArgumentFormatter had no other tests: port its 9 tests to TUnit.UnitTests, with expectations matching current output (bools render as ToString(), "True"; numbers in the current culture). Drop CleanArchitectureTests, whose tests waited 100 ms for another test to have run. - tests/Snapshots/TUnit._.verified held two BOM-only std-stream files that no test produces.
…ject setup Both hand-rolled a subset of eng/TestProject.props/.targets (engine props, engine/assertions/generator/analyzer references) and so lacked the assertion analyzers every other test project gets, e.g. the one that flags an un-awaited Assert.That, which silently checks nothing. Import the shared props/targets like the other test projects. No new diagnostics; all tests pass via `dotnet test` and as plain runs.
The project's RootNamespace and namespaces were TUnit.Assertions.SourceGenerator.IncrementalTests although it covers the core generators too (hooks, infrastructure, dynamic tests, property injection, ...), which made it show up under a misleading node in test explorers. Use the project name.
TUnit.Dev.slnx says it leaves out only the Roslyn version variants, but it also omitted the Should library and its tests, the Mocks InternalsAccess task and tests, SourceGenerator.IncrementalTests, AspNetCore.Tests and Templates.Tests, so "run all tests" from the dev solution never ran them. Add them with the projects they need (each runs in seconds without external services). Aspire.Tests (needs Docker, >1 min) and the template's Playwright content project stay out.
PublicizeAssemblyReferencesTaskTests uses Mono.Cecil directly but only received it transitively through the task project reference. An IDE build whose restore did not carry the transitive package (seen in Rider on macOS) failed with CS0103 "The name 'Mono' does not exist". Declare the dependency the code actually has.
…very OS The trusted-platform-assemblies list has no defined order and differs between Linux and macOS hosts. Generators emit referenced assemblies in reference order, so DuplicateTypeNameAcrossAssembliesTests' snapshot listed assemblies in a different order on macOS. Sort the paths by file name to make compilations identical on every host.
The scrubber only handled CI's deterministic /_/ source paths. A local checkout path longer than 80 characters is wrapped by the API generator into "..." + continuation lines, so OpenTelemetry_Library_Has_No_API_Changes kept a fragment of the path on macOS. Replace whole path literals, including continuations, for Unix and Windows paths.
…on and path scrubbing - AssertionMethodGenerator: accept [AssertionFrom] expectation messages that already start with "to " (the documented style) so they no longer render as "to to ...", and leave a spelled-out "to not ..." alone instead of negating it twice. Built-in messages, which omit the "to", are unchanged. - DotNetCli: only prefer the running installation's muxer when it has an SDK; a runtime-only installation cannot run `dotnet build`, so fall back to `dotnet` on PATH. - PublicAPI: only scrub rooted ".cs" string literals followed by the caller line number argument, so public values such as "/v1" are kept, and cover the scrubber with focused tests (short CI paths, wrapped Unix and Windows paths, unrelated strings).
…essages
A custom ExpectationMessage was emitted verbatim into GetExpectation()'s
interpolated string, but the generated class stores method parameters
as `_{name}` fields, so the documented `"to start with {value}"` failed
to compile with CS0103. Rewrite `{name}` to `{_name}` for each parameter,
as [GenerateAssertion] already does.
Quote the placeholders in the StartsWith/EndsWith examples so the shown
failure messages ('he') match what the examples produce.
Placeholders were substituted with one string.Replace per parameter, so
the escaped "{{value}}" also became "{_value}" and printed the generated
field name. Tokenize the message once instead: "{name}" for a parameter
becomes the "{_name}" field, and everything else stays literal text,
including escaped braces, placeholders that name no parameter, stray
braces, quotes and backslashes, which previously produced code that did
not compile.
Decide the "to"/"to not" wording in one helper for custom and default
messages; a negated default now reads "to not satisfy X" rather than
"not to satisfy X", matching the custom messages.
Document how ExpectationMessage is rendered on both attributes and in the
docs, and guard every built-in [AssertionFrom] declaration, including the
files without a snapshot test, against "to to" and "not not".
Condition classes that shared a name were numbered in the order they were emitted, via in-band markers patched with string.Replace. Adding or removing an attribute could change which class got a number, which changes public generated type names. Overloads that map to the same name are now told apart when the attribute is rendered: the first by parameter count and signature keeps the plain name, and the others get a stable hash of their signature. Attributes on one method with different sources, such as different expectation messages, keep the plain name for the source that sorts first and a hash of their source for the rest, renamed token by token. Names now depend only on the declarations, not on order.
Agreed, the order dependence was the real problem. 8d7b295 removes the markers,
TUnit's own output and the PublicAPI snapshots are unchanged. |
The generated source is built with AppendLine, so it carries the host's line endings, and the hashed names of attributes that share a method differed between Windows and Linux. Normalize to \n before sorting and hashing, and pin the hashed names in a test that runs on every host.
Review of #7135I read the production changes. The most important part is What looks good
Suggestions / risks
No blocking issues found. Overall this is a solid fix, with good test coverage listed in the diff (naming, message rendering, incremental, and snapshot tests). |
Pins that the two naming steps compose: overloads are told apart when the attribute is rendered, and attributes with different messages for each overload at emit time. The two steps now point at each other.
|
Review of PR 7135 I reviewed the diff directly, because the code-review skill failed to run. I didn't build the project or run any tests. Overall: The fixes are well-reasoned, and the tests and snapshots cover each one. The changes are TUnit0014 skipping interface implementations, Aspire formatting exit codes with the invariant culture, Concerns
Positives
No blocking issues from my read. Item 3 is the one I'd resolve or pin with a test before merge. |
…xpectation-messages # Conflicts: # src/TUnit.Assertions.SourceGenerator/Generators/AssertionMethodGenerator.cs # src/TUnit.Assertions/Conditions/SpanAssertionExtensions.cs # tests/TUnit.PublicAPI/Tests.Assertions_Library_Has_No_API_Changes.DotNet10_0.verified.txt # tests/TUnit.PublicAPI/Tests.Assertions_Library_Has_No_API_Changes.DotNet8_0.verified.txt # tests/TUnit.PublicAPI/Tests.Assertions_Library_Has_No_API_Changes.DotNet9_0.verified.txt
Review: PR 7135 (analyzer, Aspire and
|
… argument count A renamed generic condition class also renamed references to a same-named class of another non-zero arity, e.g. BoxCheckAssertion<T, U> when only BoxCheckAssertion<T> collided, so the generated code did not compile.
Review of #7135I read the diff for the analyzer, Aspire and What looks good
Suggestions (non-blocking)
Overall this is a clean split-out with solid test coverage. Approving in spirit. |
Review of #7135I read the generator changes ( What looks good
Concerns and suggestions (non-blocking)
Overall this is a solid fix with good tests. My concerns are mostly about long-term maintainability of the naming machinery. 🤖 Generated with Claude Code |
|
Thanks! |
…72 expectation guard - Generated data source factories and property metadata getters now escape member names that are C# keywords (e.g. @event), which previously produced uncompilable code. - BuiltInExpectationMessageTests guard threshold lowered: net472 renders 95 expectations because #if NET declarations are not generated, failing Windows CI since #7135.
net472 parses the [AssertionFrom] sources without the NET symbols, so their #if NET declarations drop out and only 95 expectations render, failing the > 100 guard on the Windows leg since #7135.
The net472 test compilation has no NET preprocessor symbols and older BCL references, so only 95 [AssertionFrom] expectations render there versus 121 on net10.0. The > 100 guard from #7135 therefore failed every Windows modularpipeline job.
… into their own workflow (#7226) * ci: publish the AOT test app and run the AOT engine tests in their own workflow Inside the .NET matrix the AOT test-app publish competed with the test modules for CPUs and was the critical path on every OS (12-25 minutes), with the Reflection-mode engine tests held back until it finished. Co-Authored-By: Claude <noreply@anthropic.com> * ci: gate NuGet publishing on the aot.yml run for the commit UploadToNuGetModule now waits for an aot.yml run on the same SHA to pass before pushing packages, and fails when none passed or none finishes within 60 minutes. The release job gains actions: read for the lookup. The AOT workflow's concurrency group keys PR runs by PR number, so fork PRs that share a branch name no longer cancel each other. * test: lower the expectation-count guard below the net472 count net472 parses the [AssertionFrom] sources without the NET symbols, so their #if NET declarations drop out and only 95 expectations render, failing the > 100 guard on the Windows leg since #7135. * ci: move the NugetTester AOT publishes into the AOT workflow They are three more native AOT publishes, and in the matrix they still competed with the engine tests for CPUs. The AOT workflow now packs too. Co-Authored-By: Claude <noreply@anthropic.com> * ci: keep polling for an unlisted AOT run, lift the upload module timeout and pass GitVersion outputs through env UploadToNuGetModule waits up to 60 minutes for aot.yml, but kept the default 30-minute module timeout, so a slow AOT run aborted the release before the wait's own deadline. A run that is not listed yet now counts as pending, and a missing GITHUB_REPOSITORY fails with a clear message. aot.yml passes the GitVersion outputs to the export script through env, so a branch name in InformationalVersion cannot reach the shell source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: stop the AOT version env vars colliding with GitVersion's on Windows Windows environment names are case-insensitive, so the step's GITVERSION_SEMVER merged with the GitVersion action's GitVersion_SemVer and bash saw only the action's spelling. Version was exported empty, TUnit.Assertions.Should's "$(Version)-beta" became invalid and restore failed on windows-latest. Co-Authored-By: Claude <noreply@anthropic.com> * ci: retry the AOT release gate's run listing on GitHub API errors Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
This PR holds the production fixes split out of #7129. It is stacked on #7129, so until that PR merges this one also shows its commits; the change itself is the last commit,
fix: reapply the analyzer, Aspire and [AssertionFrom] fixes split out of #7129. Once #7129 merges I'll mergemainin so only that commit remains.Before: TUnit0014 flagged public methods that implement an interface member. Aspire printed out-of-memory exit codes with the current culture, so sv-SE showed a U+2212 minus sign.
[AssertionFrom]customExpectationMessagevalues rendered without "to" (Expected be null or empty),{param}placeholders were not substituted, attributes that shared a method lost all but the first message, and overloads that mapped to the same generated class name all used the first overload's class.After: TUnit0014 skips interface implementations. Exit codes are formatted with the invariant culture. Custom expectation messages read "to …" / "to not …" with
NegateLogic,{param}placeholders (including non-ASCII names) are substituted,{{/}}print literal braces, each attribute keeps its own message, and each overload gets its own condition class.[AssertionFrom]diagnostics are reported at the attribute.How:
AssertionMethodGeneratorrenders message text in one tokenizer pass, decides the "to"/"not" wording in one place, and resolves generated class names at emit time through per-overload references, so colliding names get numbered classes instead of being dropped. The attribute location is only computed when a diagnostic is reported, which keeps incremental caching intact.Release note:
[AssertionFrom]ExpectationMessageis now written without the leading "to"; the generator adds "to", and "not" withNegateLogic. Messages that already start with "to " or "to not " render as before.Related Issue
Split out of #7129.
Type of Change
Checklist
Required
TUnit-Specific Requirements
TUnit.Assertions.SourceGenerator.TestsandTUnit.PublicAPItests.received.txtfiles and accepted them as.verified.txt.verified.txtfilesTesting
Additional Notes
The review points from the github-actions reviews on #7129 about the generator are addressed here: tests for the message forms (
ExpectationMessageRenderingTests,BuiltInExpectationMessageTests), a guard that no class-name marker reaches the output, numbered classes for colliding overloads (TU0002 is gone), and the lazy attribute location.Summary by CodeRabbit
Enhancements
Bug Fixes
Documentation