dotnet-test: fix eval timeouts + reduce plugin skill-menu pressure for activation

plugins/dotnet-test/skills/find-untested-sources/SKILL.md

@@ -10,6 +10,11 @@ description: >
   DO NOT USE FOR: line/branch coverage or CRAP risk (use coverage-analysis);
   whether existing tests are strong (use test-gap-analysis or assertion-quality).
 license: MIT
+# Agent-orchestrated helper (invoked by name from code-testing-researcher and the
+# code-testing pipeline); kept out of the model-facing skill menu so it does not
+# consume the plugin's 15,000-char skill-menu budget or add routing noise that
+# suppresses activation of the user-facing test skills. Still invocable by name.
+disable-model-invocation: true
 ---
 
 # Find Untested Sources

tests/dotnet-test/coverage-analysis/eval.yaml

@@ -29,7 +29,7 @@ scenarios:
       - "Identifies CalculateGpa as the highest-risk method (high complexity, low coverage)"
       - "Provides actionable recommendations for the riskiest methods"
       - "Generates or displays a coverage analysis report"
-    timeout: 300
+    timeout: 480
 
   - name: "Run coverage from scratch without existing data"
     prompt: "Analyze code coverage and risk hotspots for this project. I haven't run any coverage yet."
@@ -88,4 +88,4 @@ scenarios:
       - "Quantifies the impact — covering CalculateGpa would raise overall coverage significantly"
       - "Uses CRAP scores to distinguish the risky gap (CalculateGpa) from trivial ones"
       - "Provides specific test recommendations for the uncovered method"
-    timeout: 300
+    timeout: 480

tests/dotnet-test/run-tests/eval.yaml

@@ -73,7 +73,7 @@ scenarios:
     timeout: 420
 
   - name: "Run tests on a specific TFM with TRX in a multi-TFM MTP project (SDK 9)"
-    prompt: "This MSTest project uses Microsoft.Testing.Platform and targets both net8.0 and net9.0. I'm on .NET SDK 9. I want to run only the net9.0 tests and produce a TRX report. Show me the exact command."
+    prompt: "This MSTest project uses Microsoft.Testing.Platform and targets both net8.0 and net9.0. I'm on .NET SDK 9. I want to run only the net9.0 tests and produce a TRX report saved as net9-results.trx. Show me the exact command."
     setup:
       files:
         - path: "TestProject.csproj"
@@ -85,12 +85,14 @@ scenarios:
     assertions:
       - type: "output_matches"
         pattern: "dotnet test[^\\r\\n]*--framework\\s+net9\\.0[^\\r\\n]*--\\s+--report-trx"
+      - type: "output_matches"
+        pattern: "--report-trx-filename\\s+\"?net9-results\\.trx"
       - type: "exit_success"
     rubric:
       - "Runs only the net9.0 target framework rather than both TFMs"
       - "Identifies that the project uses MTP (TestingPlatformDotnetTestSupport) with SDK 9 argument-separator behavior"
       - "Keeps '--framework net9.0' before the '--' separator (built-in dotnet test flag)"
-      - "Passes '--report-trx' after the '--' separator (MTP extension flag, required on SDK 9)"
+      - "Passes BOTH MTP extension flags after the single '--' separator: '--report-trx' and '--report-trx-filename net9-results.trx' — it does not split them across the separator or drop the '--report-trx-filename' extension"
       - "Avoids VSTest-style TRX syntax such as '--logger trx'"
     expect_tools: ["bash"]
     timeout: 480
@@ -100,7 +102,7 @@ scenarios:
   # ============================================================================
 
   - name: "Filter MSTest tests by category on VSTest"
-    prompt: "I only want to run the integration tests using dotnet test. How do I filter by the Integration category?"
+    prompt: "I only want to run the Integration tests using dotnet test, but I need to exclude the ones in the PaymentServiceTests class. Show me the exact command."
     setup:
       additional_required_skills: ["filter-syntax"]
       files:
@@ -113,12 +115,16 @@ scenarios:
         pattern: "--filter"
       - type: "output_matches"
         pattern: "TestCategory=Integration"
+      - type: "output_contains"
+        value: "&"
+      - type: "output_matches"
+        pattern: "(?:FullyQualifiedName|ClassName|Name)\\s*(?:!~|!=)[^\"&|]*PaymentService"
       - type: "exit_success"
     rubric:
-      - "Uses the VSTest --filter flag with the TestCategory property"
-      - "Uses the correct syntax: --filter \"TestCategory=Integration\""
-      - "Does not use MTP-style filter flags like --filter-trait"
-      - "Does not suggest xUnit-style Trait filtering for an MSTest project"
+      - "Uses the VSTest --filter flag and keeps TestCategory=Integration to select the integration tests"
+      - "Combines the category filter with a class exclusion using the '&' (AND) combinator"
+      - "Excludes PaymentServiceTests with a negation operator — '!~' (does-not-contain) or '!=' (not-equals) against FullyQualifiedName/ClassName — rather than a positive match"
+      - "Does not run all tests and post-filter, and does not use MTP-only flags like --filter-not-class for this VSTest project"
     expect_tools: ["bash"]
     timeout: 120
 
@@ -193,7 +199,7 @@ scenarios:
       - "xUnit v3 on MTP requires framework-specific filter flags, not VSTest expression syntax"
       - "Correctly handles the '--' separator for SDK 9"
     expect_tools: ["bash"]
-    timeout: 120
+    timeout: 240
 
   - name: "Filter xUnit v3 tests by class pattern and trait using query filter language"
     prompt: "I have an xUnit v3 project with several test classes. I need to run only the Smoke tests from classes whose name contains 'Integration' — I want a single combined filter expression for dotnet test rather than separate flags."
@@ -219,7 +225,7 @@ scenarios:
       - "The expression correctly targets only Smoke tests from integration test classes, not all Smoke tests or all integration tests"
       - "Used the path-segment syntax with wildcard segments and a trait qualifier like [Category=Smoke]"
     expect_tools: ["bash"]
-    timeout: 120
+    timeout: 240
 
   - name: "Filter TUnit tests by class using treenode-filter"
     prompt: "I want to run only the SmsNotificationTests from this TUnit project using dotnet test. What's the command?"

tests/dotnet-test/test-gap-analysis/eval.yaml

@@ -103,7 +103,7 @@ scenarios:
       - "Acknowledged that both zero and negative quantities are tested for guard clauses"
       - "Acknowledged that RemoveStock verifies both the return value and the resulting stock level"
       - "If any gaps were found, they were presented as minor improvements rather than critical weaknesses"
-    timeout: 300
+    timeout: 420
 
   # ==========================================================================
   # Scenario 4: Non-activation — user wants to write new tests
@@ -140,7 +140,7 @@ scenarios:
     rubric:
       - "Wrote test methods for the ShoppingCart class"
       - "Covered the AddItem and GetTotal methods"
-    timeout: 120
+    timeout: 300
 
   # ==========================================================================
   # Scenario 5: Report quality — exclude trivial/generated code, trace call

tests/dotnet-test/writing-mstest-tests/eval.yaml

@@ -24,7 +24,7 @@ scenarios:
       - "Tests both success and failure paths (e.g., null input, empty items)"
       - "Uses Assert.ThrowsExactly or Assert.Throws for exception testing instead of [ExpectedException]"
       - "Uses Assert.AreEqual with expected as first argument and actual as second"
-    timeout: 180
+    timeout: 360
 
   - name: "Write data-driven tests for a calculator"
     prompt: "I have a PriceCalculator class with CalculateTotal and ApplyDiscount methods. Help me write data-driven MSTest tests using DataRow and DynamicData that cover multiple input combinations following modern MSTest best practices."
@@ -44,7 +44,7 @@ scenarios:
       - "Seals the test class"
       - "Uses Assert.ThrowsExactly or Assert.Throws for testing invalid argument scenarios"
       - "Does not use manual foreach loops to iterate test data — uses proper data-driven attributes instead"
-    timeout: 180
+    timeout: 360
 
   - name: "Write async tests with cancellation"
     prompt: |
@@ -90,7 +90,7 @@ scenarios:
 
   - name: "Fix swapped Assert.AreEqual arguments"
     prompt: |
-      Something seems off with my tests — I think I need to fix swapped Assert.AreEqual arguments. Can you review and fix the MSTest assertions?
+      When these MSTest tests fail, the failure messages look backwards — the value the code actually produced is reported as "Expected", and the literal I wrote is reported as "Actual". Review these assertions and fix whatever is causing that. Don't touch anything that is already correct.
 
       ```csharp
       [TestClass]
@@ -109,18 +109,28 @@ scenarios:
               var result = Calculator.Multiply(42, 0);
               Assert.AreEqual(result, 0);
           }
+
+          [TestMethod]
+          public void Subtract_Numbers_ReturnsDifference()
+          {
+              var result = Calculator.Subtract(10, 4);
+              Assert.AreEqual(6, result);
+          }
       }
       ```
     reject_tools: ["bash", "edit", "create"]
     assertions:
       - type: output_matches
-        pattern: "Assert\\.AreEqual\\(5"
+        pattern: "Assert\\.AreEqual\\(5,\\s*result"
       - type: output_matches
-        pattern: "Assert\\.AreEqual\\(0"
+        pattern: "Assert\\.AreEqual\\(0,\\s*result"
+      - type: output_not_matches
+        pattern: "Assert\\.AreEqual\\(result,\\s*6\\)"
     rubric:
-      - "Identifies the swapped arguments in Assert.AreEqual — expected should be first, actual second"
-      - "Fixes to Assert.AreEqual(5, result) and Assert.AreEqual(0, result)"
-      - "Explains why argument order matters (failure messages show 'Expected: X, Actual: Y')"
+      - "Diagnoses the root cause unaided (the prompt does not name it): the first two assertions pass arguments in the wrong order — MSTest's signature is `Assert.AreEqual(expected, actual)`, so the literal must come first"
+      - "Fixes the two swapped assertions to `Assert.AreEqual(5, result)` and `Assert.AreEqual(0, result)`"
+      - "Leaves the already-correct `Assert.AreEqual(6, result)` in Subtract_Numbers_ReturnsDifference unchanged — does not swap a correct assertion"
+      - "Explains that argument order drives the 'Expected: X, Actual: Y' failure message, and may reference the MSTEST0017 analyzer that flags this"
     timeout: 120
 
   - name: "Modernize legacy test patterns"
@@ -205,9 +215,9 @@ scenarios:
       - type: output_matches
         pattern: "ContainsSingle|Assert\\.IsNotNull"
     rubric:
-      - "The non-empty validation no longer relies on a generic Assert.IsTrue boolean check — it uses a dedicated assertion that clearly reports 'collection was empty' on failure"
-      - "The exact-count validation uses a count-specific or equality assertion rather than wrapping the comparison in Assert.IsTrue"
-      - "The single-element lookup uses an assertion that produces a meaningful failure message when zero or multiple elements match, rather than throwing an unrelated exception"
+      - "The non-empty validation no longer relies on a generic Assert.IsTrue boolean check — it uses a dedicated assertion such as the real MSTest 3.8+ `Assert.IsNotEmpty(collection)` (or `Assert.AreNotEqual(0, x.Count)`) that clearly reports 'collection was empty' on failure. Grading note: `Assert.IsNotEmpty`, `Assert.IsEmpty`, `Assert.HasCount`, `Assert.ContainsSingle`, `Assert.Contains`, and `Assert.DoesNotContain` are real assertion methods introduced in MSTest 3.8+ (Microsoft.Testing) — do not penalize them as non-existent or fabricated."
+      - "The exact-count validation uses a count-specific assertion such as the real MSTest 3.8+ `Assert.HasCount(3, collection)` (or `Assert.AreEqual(3, x.Count)`) rather than wrapping the comparison in Assert.IsTrue"
+      - "The single-element lookup uses an assertion that produces a meaningful failure message when zero or multiple elements match — such as the real MSTest 3.8+ `Assert.ContainsSingle(...)` — rather than `Single()` followed by Assert.IsTrue, which throws an unrelated exception"
       - "The null check uses a null-specific assertion rather than Assert.IsTrue with a boolean null comparison"
       - "Explains why specialized assertions produce better failure diagnostics than generic Assert.IsTrue"
     timeout: 120
@@ -348,7 +358,7 @@ scenarios:
       - "Tests BuildGreeting validates the output starts with 'Dear' and ends with a comma"
       - "Tests GenerateSlug validates the slug pattern contains only lowercase letters, digits, and hyphens"
       - "Tests exception handling for null or invalid inputs"
-    timeout: 180
+    timeout: 360
 
   # ============================================================================
   # Goal 6: Comparison assertions
@@ -400,7 +410,7 @@ scenarios:
       - "Tests boundary conditions at grade thresholds (e.g., exactly 90 vs 89)"
       - "Tests that NormalizeScore caps the result at 100 when rawScore exceeds maxRaw"
       - "Seals the test class"
-    timeout: 120
+    timeout: 240
 
   # ============================================================================
   # Goal 7: Collection, null, and reference assertions