Add TUnit-focused eval scenarios for dotnet-test-frameworks - #699
Conversation
Adds 5 new evaluation scenarios and tightens 2 existing scenarios to exercise the expanded TUnit coverage in dotnet-test-frameworks/SKILL.md (see #677). New scenarios: - Convert cross-framework assertions to TUnit syntax (await / IsEqualTo / IsTrue / IsNull / IsAssignableTo / Contains / Throws<T>) - Diagnose silently-passing TUnit test with missing await (debugging mystery framed without naming the pitfall) - Refactor TUnit try/catch to native exception assertion (Throws<T>() / ThrowsExactly<T>() / WithMessage) - TUnit lifecycle hooks at test / class / assembly / session scope - TUnit skip mechanisms — attribute, assembly-wide [assembly: Skip], and dynamic Skip.Test(...) Updated scenarios: - Identify TUnit framework: rubric now distinguishes [ClassDataSource] as a fixture/data source rather than a class marker (TUnit classes are convention-based, like xUnit) - Identify integration tests: adds a TUnit Project D using live SqlConnection and requires recommending [Category("Integration")] for TUnit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR expands the dotnet-test-frameworks evaluation coverage with TUnit-focused scenarios, aligning the eval suite with the broader TUnit documentation improvements planned in #677.
Changes:
- Updates existing TUnit identification and integration-test scenarios with corrected TUnit expectations.
- Adds new scenarios for TUnit assertions, missing
await, exception assertions, lifecycle hooks, and skip mechanisms. - Strengthens rubrics to guard against fallback to MSTest/xUnit/NUnit syntax.
Show a summary per file
| File | Description |
|---|---|
tests/dotnet-test/dotnet-test-frameworks/eval.yaml |
Adds and updates skill-validator eval scenarios covering TUnit-specific framework behavior. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 1/1 changed files
- Comments generated: 1
Skill Coverage Report
Uncovered:
|
|
/evaluate |
Apply the same updated and new scenarios as in eval.yaml to the parallel eval.vally.yaml so both pipelines exercise the new TUnit-focused checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
|
/evaluate |
Skill Validation Results
[1] (Plugin) Quality unchanged but weighted score is -2.6% due to: tokens (12928 → 17579), time (12.0s → 15.8s) Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Adds evaluation scenarios that demonstrate the value of the expanded TUnit coverage in #677.
New scenarios (5)
" + �wait Assert.That(x) + "" + + + "" + IsEqualTo + "" + / + "" + IsTrue + "" + / + "" + IsNull + "" + / + "" + IsAssignableTo + "" + / + "" + Contains + "" + / + "" + Throws<T> + "" + )" + �wait + "" +" + �wait + "" + causing tests to pass silently" + .Throws<T>().WithMessage(...) + "" + / + "" + .ThrowsExactly<T>() + "" + )" + [Before(Test|Class|Assembly|TestSession)] + "" + )" + [Skip] + "" + , + "" + [assembly: Skip] + "" + , + "" + Skip.Test() + "" + )Updated scenarios (2)
" + [ClassDataSource<T>] + "" + as a fixture/data source rather than a class marker (TUnit is convention-based, like xUnit), aligning with the test-class-marker correction in TUnit Fixes & Improvements #677." + SqlConnection + "" + and requires recommending + "" + [Category(\"Integration\")] + "" + for TUnit (the new entry in the integration-markers section).Validation
Ran + "
" + skill-validator evaluate --runs 2 + "" + against the SKILL.md from #677 with model + "" + claude-opus-4.6 + "" + :Verdict: + "
" + 1/1 skills passed validation + "" + . The judge specifically caught the baseline hallucinating non-existent TUnit APIs (e.g. + "" + Assert.Skip(...) + "" + ) where the skill content correctly produces + "" + Skip.Test(...) + "" + ." + skill-validator check --plugin ./plugins/dotnet-test + "" + also passes.Notes
" + [Before(Assembly)] + "" + rows, and + "" + Skip.Test() + "" + references the rubric checks for only exist once TUnit Fixes & Improvements #677 is in." + OverfittingDetection.md + "" + guidance); negative rubric items guard against the LLM defaulting to NUnit/xUnit syntax in TUnit answers.