diff --git a/eng/eval-quality/underpowered-allowlist.txt b/eng/eval-quality/underpowered-allowlist.txt index e0cd22701e..04b98b74cb 100644 --- a/eng/eval-quality/underpowered-allowlist.txt +++ b/eng/eval-quality/underpowered-allowlist.txt @@ -71,14 +71,8 @@ tests/dotnet11/system-text-json-net11/eval.yaml tests/dotnet-blazor/use-js-interop/eval.yaml tests/dotnet-experimental/exp-test-maintainability/eval.yaml tests/dotnet-maui/maui-app-lifecycle/eval.yaml -tests/dotnet-maui/maui-collectionview/eval.yaml tests/dotnet-maui/maui-data-binding/eval.yaml -tests/dotnet-maui/maui-dependency-injection/eval.yaml tests/dotnet-maui/maui-safe-area/eval.yaml -tests/dotnet-maui/maui-shell-navigation/eval.yaml -tests/dotnet-maui/maui-theming/eval.yaml tests/dotnet-msbuild/msbuild-antipatterns/eval.yaml tests/dotnet-template-engine/template-smart-defaults/eval.yaml tests/dotnet-test/find-untested-sources/eval.yaml -tests/dotnet-test/generate-testability-wrappers/eval.yaml -tests/dotnet-test/grade-tests/eval.yaml diff --git a/tests/dotnet-test/filter-syntax/eval.yaml b/tests/dotnet-test/filter-syntax/eval.yaml new file mode 100644 index 0000000000..16ea8630cb --- /dev/null +++ b/tests/dotnet-test/filter-syntax/eval.yaml @@ -0,0 +1,171 @@ +name: filter-syntax +description: Evaluates the dotnet-test/filter-syntax skill +type: capability +config: + timeout: 4m +stimuli: + - name: Select one category and exclude another on a VSTest project + prompt: > + Our CI job for Contoso.Billing.Tests currently runs the whole suite and it + takes far too long. I want one command that runs only the integration + tests but leaves out the ones we marked as slow. I'd rather not touch the + test code or the project file to make this happen. + environment: + files: + - src: ./fixtures/vstest-mstest + dest: . + graders: + - type: output-matches + config: + pattern: --filter + - type: output-matches + config: + pattern: TestCategory + - type: output-matches + config: + pattern: '!=|!~' + - type: exit-success + - type: prompt + rubric: + - Gave a single command that both selects the integration tests and excludes the slow ones + - Used a negated operator for the exclusion rather than proposing two separate runs + - Combined the two conditions into one expression with the AND combinator + - Did not propose editing the test source or the project file to achieve the selection + constraints: + reject_tools: + - edit + - create + + - name: Pass a filter to a Microsoft.Testing.Platform project on the .NET 9 SDK + prompt: > + I ran `dotnet test --filter "TestCategory=Smoke"` on Contoso.Shipping.Tests and + the SDK rejected the option instead of running my smoke tests. The project + builds fine and the tests all run when I don't filter. What command should + I actually be running here, and will it change when we move to a newer SDK? + environment: + files: + - src: ./fixtures/mtp-nunit-sdk9 + dest: . + graders: + - type: output-matches + config: + pattern: --\s+--filter + - type: output-matches + config: + pattern: Category + - type: exit-success + - type: prompt + rubric: + - Gave a command that forwards the filter to the test application rather than to the SDK command line itself + - Attributed the rejected option to how this project is run, not to a malformed filter expression + - Did not tell the user to abandon the property-and-operator expression for a different filter syntax, because this + framework accepts it unchanged on this platform + - Stated whether the extra separator is still required on a newer SDK + constraints: + reject_tools: + - edit + - create + + - name: Filter xUnit v3 tests that do not accept the generic filter expression + prompt: > + Contoso.Catalog.Tests is on xUnit v3 and `--filter "FullyQualifiedName~SearchIntegrationTests"` + does nothing — every test still runs. I need three things: run just the + SearchIntegrationTests class, run everything tagged Smoke, and finally run + only the Smoke-tagged tests under the Integration namespace. That last one + has a constraint: our pipeline template only ever passes a single filter + option and its value, so it cannot pass two options. What do I use? + environment: + files: + - src: ./fixtures/mtp-xunit-v3 + dest: . + graders: + - type: output-matches + config: + pattern: --filter-class + - type: output-matches + config: + pattern: --filter-trait + - type: output-matches + config: + pattern: --filter-query + - type: exit-success + - type: prompt + rubric: + - Explained that this runner does not honour the generic property-expression filter, which is why nothing was excluded + - Gave a working command for the single-class selection using the runner's own option + - Gave a working command for the tag-based selection using the runner's own option + - Expressed the namespace-and-tag selection with a single filter option, appending the tag selector to the method + segment as `[traitName=traitValue]` + - Commands account for the separator the installed SDK requires + constraints: + reject_tools: + - edit + - create + + - name: Filter a TUnit suite down to one class and one property value + prompt: > + Contoso.Portal.Tests is a TUnit project. I want to know how to run just the + LoginTests class, how to run a single test by name no matter which class it + is in, how to run everything tagged with the Smoke category, and how to skip + everything tagged Slow. Give me the exact commands. + environment: + files: + - src: ./fixtures/tunit + dest: . + graders: + - type: output-matches + config: + pattern: --treenode-filter + - type: output-matches + config: + pattern: /\*/ + - type: exit-success + - type: prompt + rubric: + - Used the path-shaped selector this framework provides rather than a property-expression filter + - Used wildcards for the path segments the request does not constrain + - Showed both the include-by-property and the exclude-by-property forms + - Explained what each position in the path stands for, so the user can build further selections + constraints: + reject_tools: + - edit + - create + + - name: Translate CI filter expressions after moving to xUnit v3 + prompt: | + We are moving Contoso.Catalog.Tests from the old runner to xUnit v3 and our + pipeline has these four filter arguments baked into different jobs: + + 1. `FullyQualifiedName~SearchIntegrationTests` + 2. `FullyQualifiedName=Contoso.Catalog.Tests.Integration.PricingIntegrationTests.Price_WithTax_IncludesVat` + 3. `Category=Smoke` + 4. `FullyQualifiedName~PricingIntegrationTests&Category=Smoke` + + Give me the replacement for each job. Do not change any files. + environment: + files: + - src: ./fixtures/mtp-xunit-v3 + dest: . + graders: + - type: output-matches + config: + pattern: --filter-class + - type: output-matches + config: + pattern: --filter-method + - type: output-matches + config: + pattern: --filter-trait + - type: exit-success + - type: prompt + rubric: + - Stated that the four expressions do not keep working unchanged on this runner + - Turned the substring match on the class into a class selection with wildcards on both sides + - Turned the exact fully-qualified name into an exact method selection + - Turned the category condition into a tag selection carrying a name and a value + - Translated the fourth expression, which combines a class condition and a tag condition, into a form this runner + actually accepts + constraints: + reject_tools: + - edit + - create diff --git a/tests/dotnet-test/filter-syntax/fixtures/mtp-nunit-sdk9/Contoso.Shipping.Tests.csproj b/tests/dotnet-test/filter-syntax/fixtures/mtp-nunit-sdk9/Contoso.Shipping.Tests.csproj new file mode 100644 index 0000000000..30d7ab9f57 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/mtp-nunit-sdk9/Contoso.Shipping.Tests.csproj @@ -0,0 +1,15 @@ + + + + net9.0 + false + true + + + + + + + + + diff --git a/tests/dotnet-test/filter-syntax/fixtures/mtp-nunit-sdk9/ShippingTests.cs b/tests/dotnet-test/filter-syntax/fixtures/mtp-nunit-sdk9/ShippingTests.cs new file mode 100644 index 0000000000..b5308c5e95 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/mtp-nunit-sdk9/ShippingTests.cs @@ -0,0 +1,27 @@ +using NUnit.Framework; + +namespace Contoso.Shipping.Tests; + +[TestFixture] +public class RateCalculatorTests +{ + [Test] + [Category("Unit")] + public void Rate_DomesticParcel_UsesFlatFee() { Assert.Pass(); } + + [Test] + [Category("Smoke")] + public void Rate_InternationalParcel_AddsSurcharge() { Assert.Pass(); } +} + +[TestFixture] +public class LabelPrinterTests +{ + [Test] + [Category("Smoke")] + public void Print_ValidLabel_ReturnsPdfBytes() { Assert.Pass(); } + + [Test] + [Category("Slow")] + public void Print_TenThousandLabels_CompletesWithinBudget() { Assert.Pass(); } +} diff --git a/tests/dotnet-test/filter-syntax/fixtures/mtp-nunit-sdk9/global.json b/tests/dotnet-test/filter-syntax/fixtures/mtp-nunit-sdk9/global.json new file mode 100644 index 0000000000..a02bac5a26 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/mtp-nunit-sdk9/global.json @@ -0,0 +1,6 @@ +{ + "sdk": { + "version": "9.0.200", + "rollForward": "latestFeature" + } +} diff --git a/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/Contoso.Catalog.Tests.csproj b/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/Contoso.Catalog.Tests.csproj new file mode 100644 index 0000000000..36211c7949 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/Contoso.Catalog.Tests.csproj @@ -0,0 +1,16 @@ + + + + net9.0 + Exe + false + true + + + + + + + + + diff --git a/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/IntegrationTests.cs b/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/IntegrationTests.cs new file mode 100644 index 0000000000..3edec0868c --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/IntegrationTests.cs @@ -0,0 +1,25 @@ +using Xunit; + +namespace Contoso.Catalog.Tests.Integration; + +public class SearchIntegrationTests +{ + [Fact] + [Trait("Category", "Smoke")] + public void Search_KnownTerm_ReturnsHits() { Assert.True(true); } + + [Fact] + [Trait("Category", "Nightly")] + public void Search_FullReindex_Completes() { Assert.True(true); } +} + +public class PricingIntegrationTests +{ + [Fact] + [Trait("Category", "Smoke")] + public void Price_WithTax_IncludesVat() { Assert.True(true); } + + [Fact] + [Trait("Category", "Nightly")] + public void Price_UnknownSku_Throws() { Assert.True(true); } +} diff --git a/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/UnitTests.cs b/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/UnitTests.cs new file mode 100644 index 0000000000..e45c426563 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/UnitTests.cs @@ -0,0 +1,21 @@ +using Xunit; + +namespace Contoso.Catalog.Tests.Unit; + +public class SearchQueryParserTests +{ + [Fact] + [Trait("Category", "Smoke")] + public void Parse_SingleTerm_ReturnsTerm() { Assert.True(true); } + + [Fact] + [Trait("Category", "Regression")] + public void Parse_UnbalancedQuotes_Throws() { Assert.True(true); } +} + +public class PriceFormatterTests +{ + [Fact] + [Trait("Category", "Smoke")] + public void Format_WholeAmount_HasTwoDecimals() { Assert.True(true); } +} diff --git a/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/global.json b/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/global.json new file mode 100644 index 0000000000..a02bac5a26 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/mtp-xunit-v3/global.json @@ -0,0 +1,6 @@ +{ + "sdk": { + "version": "9.0.200", + "rollForward": "latestFeature" + } +} diff --git a/tests/dotnet-test/filter-syntax/fixtures/tunit/Contoso.Portal.Tests.csproj b/tests/dotnet-test/filter-syntax/fixtures/tunit/Contoso.Portal.Tests.csproj new file mode 100644 index 0000000000..785fd4b979 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/tunit/Contoso.Portal.Tests.csproj @@ -0,0 +1,13 @@ + + + + net10.0 + Exe + false + + + + + + + diff --git a/tests/dotnet-test/filter-syntax/fixtures/tunit/PortalTests.cs b/tests/dotnet-test/filter-syntax/fixtures/tunit/PortalTests.cs new file mode 100644 index 0000000000..63d74870d3 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/tunit/PortalTests.cs @@ -0,0 +1,26 @@ +using System.Threading.Tasks; +using TUnit.Core; + +namespace Contoso.Portal.Tests.Api; + +public class LoginTests +{ + [Test] + [Category("Smoke")] + public async Task AcceptCookiesTest() { await Task.CompletedTask; } + + [Test] + [Category("Slow")] + public async Task LoginWithExpiredPasswordTest() { await Task.CompletedTask; } +} + +public class SignupTests +{ + [Test] + [Category("Smoke")] + public async Task SignupWithValidEmailTest() { await Task.CompletedTask; } + + [Test] + [Category("Slow")] + public async Task SignupRejectsDuplicateEmailTest() { await Task.CompletedTask; } +} diff --git a/tests/dotnet-test/filter-syntax/fixtures/vstest-mstest/BillingTests.cs b/tests/dotnet-test/filter-syntax/fixtures/vstest-mstest/BillingTests.cs new file mode 100644 index 0000000000..7427163188 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/vstest-mstest/BillingTests.cs @@ -0,0 +1,33 @@ +using Microsoft.VisualStudio.TestTools.UnitTesting; + +namespace Contoso.Billing.Tests; + +[TestClass] +public class InvoiceTests +{ + [TestMethod] + [TestCategory("Unit")] + public void CreateInvoice_ValidLines_ComputesTotal() { Assert.IsTrue(true); } + + [TestMethod] + [TestCategory("Integration")] + public void CreateInvoice_PersistsToDatabase() { Assert.IsTrue(true); } + + [TestMethod] + [TestCategory("Integration")] + [TestCategory("Slow")] + public void ReconcileLedger_FullMonth_Balances() { Assert.IsTrue(true); } +} + +[TestClass] +public class PaymentTests +{ + [TestMethod] + [TestCategory("Unit")] + public void Charge_NegativeAmount_Throws() { Assert.IsTrue(true); } + + [TestMethod] + [TestCategory("Integration")] + [TestCategory("Slow")] + public void Charge_RealGateway_Succeeds() { Assert.IsTrue(true); } +} diff --git a/tests/dotnet-test/filter-syntax/fixtures/vstest-mstest/Contoso.Billing.Tests.csproj b/tests/dotnet-test/filter-syntax/fixtures/vstest-mstest/Contoso.Billing.Tests.csproj new file mode 100644 index 0000000000..dc80072d54 --- /dev/null +++ b/tests/dotnet-test/filter-syntax/fixtures/vstest-mstest/Contoso.Billing.Tests.csproj @@ -0,0 +1,14 @@ + + + + net8.0 + false + + + + + + + + + diff --git a/tests/dotnet-test/generate-testability-wrappers/eval.yaml b/tests/dotnet-test/generate-testability-wrappers/eval.yaml index 976716f1a5..675774fcea 100644 --- a/tests/dotnet-test/generate-testability-wrappers/eval.yaml +++ b/tests/dotnet-test/generate-testability-wrappers/eval.yaml @@ -23,6 +23,9 @@ stimuli: - type: output-matches config: pattern: (AddSingleton|Register|services\.Add) + - type: output-matches + config: + pattern: Assert\.(True|IsTrue|AreEqual|Equal) - type: exit-success - type: prompt rubric: @@ -30,6 +33,8 @@ stimuli: - Showed how to register TimeProvider.System in the DI container - Showed how to inject TimeProvider into the ReportGenerator class - Showed how to use FakeTimeProvider in tests to pin time + - The sample test actually asserts on a time-dependent outcome after advancing the pinned clock, rather than only + constructing the fake - Mentioned the Microsoft.Extensions.TimeProvider.Testing NuGet package for testing - name: Generate custom Environment wrapper @@ -48,12 +53,20 @@ stimuli: - type: output-matches config: pattern: (GetEnvironmentVariable|MachineName) + - type: output-matches + config: + pattern: \bsealed\b + - type: output-matches + config: + pattern: AddSingleton - type: exit-success - type: prompt rubric: - Generated an interface with methods for GetEnvironmentVariable and MachineName - Generated a default implementation that delegates to the real Environment static class - Provided DI registration code for the wrapper + - Chose a singleton lifetime (`AddSingleton`) because the wrapper holds no state, and said what would change for a + stateful wrapper that needs `AddTransient` or `AddScoped` - Only wrapped the two members actually used in the code, not every Environment member - name: Recommend System.IO.Abstractions for file system calls @@ -71,14 +84,50 @@ stimuli: - type: output-matches config: pattern: (MockFileSystem|mock.*file) + - type: output-matches + config: + pattern: Assert\.(Equal|AreEqual) - type: exit-success - type: prompt rubric: - Recommended System.IO.Abstractions NuGet package instead of writing a custom wrapper - Showed how to register IFileSystem in DI - Showed how to use MockFileSystem in tests + - The sample test seeds the in-memory file system and then asserts on what the code under test read back from it - Did not generate a custom file system wrapper when the established package exists + - name: Make time controllable in a library that has no DI container + prompt: > + Contoso.Retention is a NuGet library we ship. Its whole public API is + static, so there is nothing for a caller to construct and no service + container I can register anything into, and I'm not allowed to change the + released signatures. Every method reads DateTime.UtcNow directly, so I + can't test the expiry rules. Our test suite runs in parallel. How do I make + the current time controllable from tests here? + environment: + files: + - src: ./fixtures/no-di-library + dest: Contoso.Retention + graders: + - type: output-matches + config: + pattern: AsyncLocal + - type: output-matches + config: + pattern: \breadonly\b + - type: output-matches + config: + pattern: (IDisposable|Dispose) + - type: exit-success + - type: prompt + rubric: + - Offered a way to substitute the clock that works without a service container and without changing the public API + - Kept the substituted value flow-local across `async`/`await` so tests running in parallel cannot observe each + other's substitution + - Scoped the substitution to something the test disposes, so the real clock is restored when the test ends + - Explained the trade-offs of this ambient-context approach, including its per-call cost on the production path + - Did not tell the user to introduce a DI container or change the released signatures after they ruled both out + - name: Decline wrapper generation for already-abstracted code prompt: > My code already uses IFileSystem from System.IO.Abstractions everywhere. diff --git a/tests/dotnet-test/generate-testability-wrappers/fixtures/no-di-library/Contoso.Retention.csproj b/tests/dotnet-test/generate-testability-wrappers/fixtures/no-di-library/Contoso.Retention.csproj new file mode 100644 index 0000000000..5d6e8277cb --- /dev/null +++ b/tests/dotnet-test/generate-testability-wrappers/fixtures/no-di-library/Contoso.Retention.csproj @@ -0,0 +1,9 @@ + + + + net8.0 + enable + enable + + + diff --git a/tests/dotnet-test/generate-testability-wrappers/fixtures/no-di-library/RetentionPolicy.cs b/tests/dotnet-test/generate-testability-wrappers/fixtures/no-di-library/RetentionPolicy.cs new file mode 100644 index 0000000000..eaab9392aa --- /dev/null +++ b/tests/dotnet-test/generate-testability-wrappers/fixtures/no-di-library/RetentionPolicy.cs @@ -0,0 +1,20 @@ +namespace Contoso.Retention; + +/// +/// Shipped as a NuGet library. Every member below is part of the released, +/// static public API surface — there is nothing here for a caller to construct +/// and nothing for a container to resolve. +/// +public static class RetentionPolicy +{ + public static bool IsExpired(DateTimeOffset createdAt, TimeSpan retention) + => DateTime.UtcNow - createdAt > retention; + + public static DateTimeOffset NextSweep(DateTimeOffset lastSweep) + => lastSweep.AddDays(1) < DateTime.UtcNow + ? DateTime.UtcNow + : lastSweep.AddDays(1); + + public static string StampFileName(string prefix) + => $"{prefix}_{DateTime.UtcNow:yyyyMMddHHmmss}.log"; +} diff --git a/tests/dotnet-test/grade-tests/eval.yaml b/tests/dotnet-test/grade-tests/eval.yaml index 5097d7e647..fdf8895229 100644 --- a/tests/dotnet-test/grade-tests/eval.yaml +++ b/tests/dotnet-test/grade-tests/eval.yaml @@ -135,6 +135,61 @@ stimuli: reject_tools: - edit - create + - name: Keep a 62-test grading report readable as a PR comment + prompt: | + Please grade every test declared in `Catalog.Tests/ProductCatalogTests.cs` + individually for test quality — one row per test — and give me a short + summary. The code under test is in `Catalog.Core/ProductCatalog.cs`. + Apply this repository's established per-test grading policy and do not + modify any files. + + This goes straight into a PR comment, so it has to stay readable when a + reviewer opens the thread: I don't want to scroll past dozens of rows of + passing tests before I reach anything I need to act on, and I don't want + the same problem restated several times in different words. + environment: + files: + - src: fixtures/large-suite/Catalog.Core/Catalog.Core.csproj + dest: Catalog.Core/Catalog.Core.csproj + - src: fixtures/large-suite/Catalog.Core/ProductCatalog.cs + dest: Catalog.Core/ProductCatalog.cs + - src: fixtures/large-suite/Catalog.Tests/Catalog.Tests.csproj + dest: Catalog.Tests/Catalog.Tests.csproj + - src: fixtures/large-suite/Catalog.Tests/ProductCatalogTests.cs + dest: Catalog.Tests/ProductCatalogTests.cs + graders: + - type: output-matches + config: + pattern: \|\s*Test\s*\|\s*Grade\s*\|\s*Band\s*\|\s*Notes\s*\| + - type: output-matches + config: + pattern:
+ - type: output-matches + config: + pattern: (TotalValue_Runs.*\|\s*F\s*\|) + - type: output-matches + config: + pattern: (Find_KnownSku_ReturnsSomething.*\|\s*C\s*\|) + - type: exit-success + - type: prompt + rubric: + - Kept the rendered table bounded and collapsed the overflow rows into a `
` block instead of spilling + every graded test into the PR comment + - Put the tests that need attention above the ones that do not, so the first rows a reviewer sees are the + lowest-graded ones + - The summary opens with the single most consequential observation and what to do about it, rather than being a + recap of the table + - Each distinct issue is reported once under the category that fits it best, rather than the same finding + appearing again under several categories + - Graded the tests whose body ends without any assertion as F, and the ones whose only assertion is a null check + as C + - Did not silently drop tests from the report — every test in the file is accounted for, whether in the visible + rows or the collapsed section + constraints: + reject_tools: + - edit + - create + - name: Grade tests when the production code under test is unavailable prompt: | Please grade each of the following test methods individually for test diff --git a/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Core/Catalog.Core.csproj b/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Core/Catalog.Core.csproj new file mode 100644 index 0000000000..5d6e8277cb --- /dev/null +++ b/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Core/Catalog.Core.csproj @@ -0,0 +1,9 @@ + + + + net8.0 + enable + enable + + + diff --git a/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Core/ProductCatalog.cs b/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Core/ProductCatalog.cs new file mode 100644 index 0000000000..ab605e3fd0 --- /dev/null +++ b/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Core/ProductCatalog.cs @@ -0,0 +1,64 @@ +namespace Catalog.Core; + +public sealed record Product(string Sku, string Name, decimal Price, int Quantity); + +public sealed class ProductCatalog +{ + private readonly Dictionary _products = new(StringComparer.OrdinalIgnoreCase); + + public int Count => _products.Count; + + public void Add(Product product) + { + ArgumentNullException.ThrowIfNull(product); + if (_products.ContainsKey(product.Sku)) + throw new ArgumentException($"Duplicate SKU '{product.Sku}'.", nameof(product)); + _products[product.Sku] = product; + } + + public Product? Find(string sku) + { + ArgumentException.ThrowIfNullOrWhiteSpace(sku); + return _products.TryGetValue(sku, out var product) ? product : null; + } + + public bool Remove(string sku) + { + ArgumentException.ThrowIfNullOrWhiteSpace(sku); + return _products.Remove(sku); + } + + public decimal TotalValue() => _products.Values.Sum(p => p.Price * p.Quantity); + + public IReadOnlyList Search(string term) + { + ArgumentException.ThrowIfNullOrWhiteSpace(term); + return [.. _products.Values.Where(p => + p.Name.Contains(term, StringComparison.OrdinalIgnoreCase) || + p.Sku.Contains(term, StringComparison.OrdinalIgnoreCase))]; + } + + public void Restock(string sku, int quantity) + { + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(quantity); + if (!_products.TryGetValue(sku, out var product)) + throw new KeyNotFoundException($"Unknown SKU '{sku}'."); + _products[sku] = product with { Quantity = product.Quantity + quantity }; + } + + public void ApplyDiscount(decimal percent) + { + ArgumentOutOfRangeException.ThrowIfNegative(percent); + ArgumentOutOfRangeException.ThrowIfGreaterThan(percent, 100m); + foreach (var sku in _products.Keys.ToList()) + { + var product = _products[sku]; + var discounted = Math.Round(product.Price * (100m - percent) / 100m, 2); + _products[sku] = product with { Price = discounted }; + } + } + + public void Clear() => _products.Clear(); + + public bool IsInStock(string sku) => Find(sku) is { Quantity: > 0 }; +} diff --git a/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Tests/Catalog.Tests.csproj b/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Tests/Catalog.Tests.csproj new file mode 100644 index 0000000000..b85256f641 --- /dev/null +++ b/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Tests/Catalog.Tests.csproj @@ -0,0 +1,19 @@ + + + + net8.0 + enable + enable + false + + + + + + + + + + + + diff --git a/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Tests/ProductCatalogTests.cs b/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Tests/ProductCatalogTests.cs new file mode 100644 index 0000000000..e2079e155f --- /dev/null +++ b/tests/dotnet-test/grade-tests/fixtures/large-suite/Catalog.Tests/ProductCatalogTests.cs @@ -0,0 +1,615 @@ +using Catalog.Core; +using Microsoft.VisualStudio.TestTools.UnitTesting; + +namespace Catalog.Tests; + +[TestClass] +public sealed class ProductCatalogTests +{ + private static ProductCatalog Seeded() + { + var catalog = new ProductCatalog(); + catalog.Add(new Product("WID-1", "Widget", 10.00m, 3)); + catalog.Add(new Product("GAD-2", "Gadget", 25.50m, 2)); + catalog.Add(new Product("DOO-3", "Doohickey", 4.25m, 0)); + return catalog; + } + + // ---------------------------------------------------------------- Add + + [TestMethod] + public void Add_NewProduct_IncreasesCount() + { + var catalog = new ProductCatalog(); + + catalog.Add(new Product("WID-1", "Widget", 10.00m, 1)); + + Assert.AreEqual(1, catalog.Count); + } + + [TestMethod] + public void Add_NewProduct_IsFindableBySku() + { + var catalog = new ProductCatalog(); + var product = new Product("WID-1", "Widget", 10.00m, 1); + + catalog.Add(product); + + Assert.AreEqual(product, catalog.Find("WID-1")); + } + + [TestMethod] + public void Add_NullProduct_ThrowsArgumentNullException() + { + var catalog = new ProductCatalog(); + + Assert.ThrowsExactly(() => catalog.Add(null!)); + } + + [TestMethod] + public void Add_DuplicateSku_ThrowsArgumentException() + { + var catalog = Seeded(); + + var ex = Assert.ThrowsExactly( + () => catalog.Add(new Product("WID-1", "Widget Clone", 1.00m, 1))); + Assert.Contains("WID-1", ex.Message); + } + + [TestMethod] + public void Add_ThreeProducts_CountIsThree() + { + var catalog = Seeded(); + + Assert.AreEqual(3, catalog.Count); + } + + [TestMethod] + public void Add_ProductWithZeroPrice_IsAccepted() + { + var catalog = new ProductCatalog(); + + catalog.Add(new Product("FREE-0", "Sample", 0m, 5)); + + Assert.AreEqual(0m, catalog.Find("FREE-0")!.Price); + } + + [TestMethod] + public void Count_AfterDuplicateAddAttempt_IsUnchanged() + { + var catalog = Seeded(); + + try + { + catalog.Add(new Product("WID-1", "Widget Clone", 1.00m, 1)); + } + catch (ArgumentException) + { + // expected + } + + Assert.AreEqual(3, catalog.Count); + } + + // --------------------------------------------------------------- Find + + [TestMethod] + public void Find_ExistingSku_ReturnsMatchingProduct() + { + var catalog = Seeded(); + + var product = catalog.Find("GAD-2"); + + Assert.AreEqual("Gadget", product!.Name); + } + + [TestMethod] + public void Find_UnknownSku_ReturnsNull() + { + var catalog = Seeded(); + + Assert.IsNull(catalog.Find("NOPE-9")); + } + + [TestMethod] + public void Find_EmptySku_ThrowsArgumentException() + { + var catalog = Seeded(); + + Assert.ThrowsExactly(() => catalog.Find(" ")); + } + + [TestMethod] + public void Find_AfterRemove_ReturnsNull() + { + var catalog = Seeded(); + + catalog.Remove("GAD-2"); + + Assert.IsNull(catalog.Find("GAD-2")); + } + + // ------------------------------------------------------------- Remove + + [TestMethod] + public void Remove_ExistingSku_ReturnsTrue() + { + var catalog = Seeded(); + + Assert.IsTrue(catalog.Remove("WID-1")); + } + + [TestMethod] + public void Remove_ExistingSku_DecreasesCount() + { + var catalog = Seeded(); + + catalog.Remove("WID-1"); + + Assert.AreEqual(2, catalog.Count); + } + + [TestMethod] + public void Remove_UnknownSku_ReturnsFalse() + { + var catalog = Seeded(); + + Assert.IsFalse(catalog.Remove("NOPE-9")); + } + + [TestMethod] + public void Remove_UnknownSku_LeavesCountUnchanged() + { + var catalog = Seeded(); + + catalog.Remove("NOPE-9"); + + Assert.AreEqual(3, catalog.Count); + } + + // -------------------------------------------------------------- Count + + [TestMethod] + public void Count_EmptyCatalog_IsZero() + { + Assert.AreEqual(0, new ProductCatalog().Count); + } + + [TestMethod] + public void Count_AfterClear_IsZero() + { + var catalog = Seeded(); + + catalog.Clear(); + + Assert.AreEqual(0, catalog.Count); + } + + // --------------------------------------------------------- TotalValue + + [TestMethod] + public void TotalValue_EmptyCatalog_IsZero() + { + Assert.AreEqual(0m, new ProductCatalog().TotalValue()); + } + + [TestMethod] + public void TotalValue_SingleProduct_IsPriceTimesQuantity() + { + var catalog = new ProductCatalog(); + catalog.Add(new Product("WID-1", "Widget", 10.00m, 3)); + + Assert.AreEqual(30.00m, catalog.TotalValue()); + } + + [TestMethod] + public void TotalValue_MultipleProducts_SumsAllLines() + { + var catalog = Seeded(); + + Assert.AreEqual(81.00m, catalog.TotalValue()); + } + + [TestMethod] + public void TotalValue_AfterRemove_ExcludesRemovedLine() + { + var catalog = Seeded(); + + catalog.Remove("GAD-2"); + + Assert.AreEqual(30.00m, catalog.TotalValue()); + } + + [TestMethod] + public void TotalValue_AfterDiscount_ReflectsNewPrices() + { + var catalog = Seeded(); + + catalog.ApplyDiscount(50m); + + Assert.AreEqual(40.50m, catalog.TotalValue()); + } + + // ------------------------------------------------------------- Search + + [TestMethod] + public void Search_MatchingTerm_ReturnsMatches() + { + var catalog = Seeded(); + + var results = catalog.Search("Widget"); + + Assert.HasCount(1, results); + } + + [TestMethod] + public void Search_NoMatch_ReturnsEmptyList() + { + var catalog = Seeded(); + + Assert.IsEmpty(catalog.Search("Sprocket")); + } + + [TestMethod] + public void Search_IsCaseInsensitive() + { + var catalog = Seeded(); + + var results = catalog.Search("gadget"); + + Assert.AreEqual("GAD-2", results[0].Sku); + } + + [TestMethod] + public void Search_NullTerm_ThrowsArgumentNullException() + { + var catalog = Seeded(); + + Assert.ThrowsExactly(() => catalog.Search(null!)); + } + + [TestMethod] + public void Search_WhitespaceTerm_ThrowsArgumentException() + { + var catalog = Seeded(); + + Assert.ThrowsExactly(() => catalog.Search(" ")); + } + + [TestMethod] + public void Search_PartialSkuMatch_ReturnsProduct() + { + var catalog = Seeded(); + + var results = catalog.Search("DOO"); + + Assert.AreEqual("Doohickey", results[0].Name); + } + + // ------------------------------------------------------------ Restock + + [TestMethod] + public void Restock_KnownSku_IncreasesQuantity() + { + var catalog = Seeded(); + + catalog.Restock("WID-1", 4); + + Assert.AreEqual(7, catalog.Find("WID-1")!.Quantity); + } + + [TestMethod] + public void Restock_UnknownSku_ThrowsKeyNotFound() + { + var catalog = Seeded(); + + Assert.ThrowsExactly(() => catalog.Restock("NOPE-9", 1)); + } + + [TestMethod] + public void Restock_NegativeQuantity_ThrowsArgumentOutOfRange() + { + var catalog = Seeded(); + + Assert.ThrowsExactly(() => catalog.Restock("WID-1", -1)); + } + + [TestMethod] + public void Restock_ZeroQuantity_ThrowsArgumentOutOfRange() + { + var catalog = Seeded(); + + Assert.ThrowsExactly(() => catalog.Restock("WID-1", 0)); + } + + [TestMethod] + public void Restock_TwiceInARow_AccumulatesQuantity() + { + var catalog = Seeded(); + + catalog.Restock("WID-1", 2); + catalog.Restock("WID-1", 5); + + Assert.AreEqual(10, catalog.Find("WID-1")!.Quantity); + } + + // ------------------------------------------------------ ApplyDiscount + + [TestMethod] + public void ApplyDiscount_TenPercent_ReducesEveryPrice() + { + var catalog = Seeded(); + + catalog.ApplyDiscount(10m); + + Assert.AreEqual(9.00m, catalog.Find("WID-1")!.Price); + } + + [TestMethod] + public void ApplyDiscount_Zero_LeavesPricesUnchanged() + { + var catalog = Seeded(); + + catalog.ApplyDiscount(0m); + + Assert.AreEqual(10.00m, catalog.Find("WID-1")!.Price); + } + + [TestMethod] + public void ApplyDiscount_Negative_ThrowsArgumentOutOfRange() + { + var catalog = Seeded(); + + Assert.ThrowsExactly(() => catalog.ApplyDiscount(-5m)); + } + + [TestMethod] + public void ApplyDiscount_OverOneHundred_ThrowsArgumentOutOfRange() + { + var catalog = Seeded(); + + Assert.ThrowsExactly(() => catalog.ApplyDiscount(120m)); + } + + [TestMethod] + public void ApplyDiscount_RoundsToTwoDecimals() + { + var catalog = Seeded(); + + catalog.ApplyDiscount(33m); + + Assert.AreEqual(2.85m, catalog.Find("DOO-3")!.Price); + } + + // -------------------------------------------------------------- Clear + + [TestMethod] + public void Clear_PopulatedCatalog_RemovesEveryProduct() + { + var catalog = Seeded(); + + catalog.Clear(); + + Assert.IsEmpty(catalog.Search("Widget")); + } + + [TestMethod] + public void Clear_EmptyCatalog_DoesNotThrow() + { + var catalog = new ProductCatalog(); + + catalog.Clear(); + + Assert.AreEqual(0, catalog.Count); + } + + // ---------------------------------------------------------- IsInStock + + [TestMethod] + public void IsInStock_PositiveQuantity_ReturnsTrue() + { + var catalog = Seeded(); + + Assert.IsTrue(catalog.IsInStock("WID-1")); + } + + [TestMethod] + public void IsInStock_ZeroQuantity_ReturnsFalse() + { + var catalog = Seeded(); + + Assert.IsFalse(catalog.IsInStock("DOO-3")); + } + + [TestMethod] + public void IsInStock_UnknownSku_ReturnsFalse() + { + var catalog = Seeded(); + + Assert.IsFalse(catalog.IsInStock("NOPE-9")); + } + + [TestMethod] + public void IsInStock_AfterRestockFromZero_ReturnsTrue() + { + var catalog = Seeded(); + + catalog.Restock("DOO-3", 6); + + Assert.IsTrue(catalog.IsInStock("DOO-3")); + } + + // -------------------------------------------- null-check-only checks + + [TestMethod] + public void Find_KnownSku_ReturnsSomething() + { + var catalog = Seeded(); + + var product = catalog.Find("WID-1"); + + Assert.IsNotNull(product); + } + + [TestMethod] + public void Search_KnownTerm_ReturnsSomething() + { + var catalog = Seeded(); + + var results = catalog.Search("Widget"); + + Assert.IsNotNull(results); + } + + [TestMethod] + public void Search_EmptyResultSet_ReturnsList() + { + var catalog = Seeded(); + + var results = catalog.Search("Sprocket"); + + Assert.IsNotNull(results); + } + + [TestMethod] + public void Add_ThenFind_ReturnsSomething() + { + var catalog = new ProductCatalog(); + catalog.Add(new Product("NEW-1", "Newbie", 1.00m, 1)); + + var product = catalog.Find("NEW-1"); + + Assert.IsNotNull(product); + } + + [TestMethod] + public void Restock_ThenFind_ReturnsSomething() + { + var catalog = Seeded(); + catalog.Restock("WID-1", 1); + + var product = catalog.Find("WID-1"); + + Assert.IsNotNull(product); + } + + [TestMethod] + public void ApplyDiscount_ThenFind_ReturnsSomething() + { + var catalog = Seeded(); + catalog.ApplyDiscount(5m); + + var product = catalog.Find("GAD-2"); + + Assert.IsNotNull(product); + } + + [TestMethod] + public void Clear_ThenSearch_ReturnsList() + { + var catalog = Seeded(); + catalog.Clear(); + + var results = catalog.Search("Widget"); + + Assert.IsNotNull(results); + } + + [TestMethod] + public void Remove_ThenSearch_ReturnsList() + { + var catalog = Seeded(); + catalog.Remove("WID-1"); + + var results = catalog.Search("Gadget"); + + Assert.IsNotNull(results); + } + + [TestMethod] + public void TotalValue_ThenFind_ReturnsSomething() + { + var catalog = Seeded(); + _ = catalog.TotalValue(); + + var product = catalog.Find("DOO-3"); + + Assert.IsNotNull(product); + } + + [TestMethod] + public void Find_AfterMultipleAdds_ReturnsSomething() + { + var catalog = Seeded(); + catalog.Add(new Product("EXT-4", "Extra", 2.00m, 1)); + + var product = catalog.Find("EXT-4"); + + Assert.IsNotNull(product); + } + + [TestMethod] + public void Search_AfterRestock_ReturnsList() + { + var catalog = Seeded(); + catalog.Restock("GAD-2", 3); + + var results = catalog.Search("Gadget"); + + Assert.IsNotNull(results); + } + + // ------------------------------------------------ assertion-free runs + + [TestMethod] + public void Add_ManyProducts_Runs() + { + var catalog = new ProductCatalog(); + for (var i = 0; i < 20; i++) + { + catalog.Add(new Product($"BULK-{i}", $"Bulk {i}", i, i)); + } + } + + [TestMethod] + public void Clear_Runs() + { + var catalog = Seeded(); + catalog.Clear(); + } + + [TestMethod] + public void Restock_Runs() + { + var catalog = Seeded(); + catalog.Restock("WID-1", 3); + } + + [TestMethod] + public void ApplyDiscount_Runs() + { + var catalog = Seeded(); + catalog.ApplyDiscount(15m); + } + + [TestMethod] + public void Remove_Runs() + { + var catalog = Seeded(); + catalog.Remove("GAD-2"); + } + + [TestMethod] + public void Search_Runs() + { + var catalog = Seeded(); + _ = catalog.Search("Widget"); + } + + [TestMethod] + public void TotalValue_Runs() + { + var catalog = Seeded(); + _ = catalog.TotalValue(); + } +} diff --git a/tests/dotnet-test/writing-mstest-tests/eval.yaml b/tests/dotnet-test/writing-mstest-tests/eval.yaml index 4dbf383981..0719265c4b 100644 --- a/tests/dotnet-test/writing-mstest-tests/eval.yaml +++ b/tests/dotnet-test/writing-mstest-tests/eval.yaml @@ -524,6 +524,55 @@ stimuli: - bash - edit - create + - name: Replace generic IsTrue checks for null, identity, emptiness, and absence + prompt: | + Every check in this MSTest test is written with Assert.IsTrue. Rewrite it so each check uses the most specific MSTest assertion available for what it is actually verifying, following MSTest best practices. + + ```csharp + [TestMethod] + public void SessionStore_StartTouchEndPurge_BehavesCorrectly() + { + var store = new SessionStore(); + var session = store.Start("alice"); + + Assert.IsTrue(ReferenceEquals(session, store.Get("alice"))); + Assert.IsTrue(store.Get("nobody") == null); + + store.End("alice"); + Assert.IsTrue(!store.ActiveSessions().Contains(session)); + + store.Purge(); + Assert.IsTrue(store.ActiveSessions().Count == 0); + } + ``` + graders: + - type: output-matches + config: + pattern: Assert\.AreSame + - type: output-matches + config: + pattern: Assert\.IsNull + - type: output-matches + config: + pattern: Assert\.DoesNotContain + - type: output-matches + config: + pattern: Assert\.IsEmpty + - type: prompt + rubric: + - Replaced the ReferenceEquals check with an assertion dedicated to verifying that two references are the same + object, not merely equal values + - Replaced the '== null' comparison with a null-specific assertion + - Replaced the negated Contains check with an assertion for an item being absent from a collection + - Replaced the 'Count == 0' check with a dedicated empty-collection assertion + - Explained that the specific assertions report what actually differed on failure, whereas the generic boolean form + can only report that a condition was false + constraints: + reject_tools: + - bash + - edit + - create + - name: Configure test parallelization and MSTest.Sdk project prompt: | I'm setting up a new MSTest.Sdk project from scratch. I want test parallelization: